# Onebox user agent
**URL:** https://meta.discourse.org/t/onebox-user-agent/61498
**Category:** Feature
**Created:** [April 24, 2017, 2:35pm UTC](https://meta.discourse.org/t/onebox-user-agent/61498 "2017-04-24T14:35:12Z")
**Posts on this page:** 19
**Page:** 1
### Author:  [@kevinranks](https://meta.discourse.org/u/kevinranks)
#### Post date: [April 24, 2017, 2:35pm UTC](https://meta.discourse.org/t/onebox-user-agent/61498/1 "2017-04-24T14:35:12Z")
It appears that when onebox fetches a url to grab opengraph data, it sends a user agent string of “Ruby”.
Can this be updated to provide a more useful user agent that identifies the crawler properly?
---
### Author:  [@eviltrout](https://meta.discourse.org/u/eviltrout)
#### Post date: [April 24, 2017, 3:26pm UTC](https://meta.discourse.org/t/onebox-user-agent/61498/2 "2017-04-24T15:26:57Z")
What problem are you trying to solve here? Is there a site that is not working because of `Ruby` or is there another reason why you’d like to change it.
If you want a pointer to where in the code this happens, it’s in our [onebox gem](https://github.com/discourse/onebox/blob/67a165a14b4bde573691138d383a15b74ab60d36/lib/onebox/helpers.rb#L36). You’d have to add a custom header there if one does not exist.
---
### Author:  [@kevinranks](https://meta.discourse.org/u/kevinranks)
#### Post date: [April 24, 2017, 3:42pm UTC](https://meta.discourse.org/t/onebox-user-agent/61498/3 "2017-04-24T15:42:48Z")
Hey @eviltrout,
Besides being good practice to identify the crawler properly, one specific example would be prerendering:
> **[Seamless JavaScript Rendering for SEO, AEO, and LLM Visibility](https://prerender.io/)**
>
> Helping businesses show up on every organic search platform: Google, ChatGPT, Reddit, and other LLMs. Trusted by 100,000+ brands worldwide. Start for free.
For sites that use Angular, it’s important to be able to identify crawlers so that the page can be prerendered and served to the crawler. Without it, the crawler may end up extracting things like:
``.
---
### Author:  [@eviltrout](https://meta.discourse.org/u/eviltrout)
#### Post date: [April 24, 2017, 4:03pm UTC](https://meta.discourse.org/t/onebox-user-agent/61498/4 "2017-04-24T16:03:31Z")
That example is pretty odd – in what case would it ever make sense for the prerenderer to **not** include the og:description value in the initial HTML payload? It seems to me like any time the document is rendered on the server side it should fill in that value, regardless of user agent.
Do you have an example site that uses this pattern that Onebox doesn’t work with?
I’m not against adding a User Agent (although I’m not sure what it would be), but I am curious about a definitive example of where we are currently broken.
---
### Author:  [@kevinranks](https://meta.discourse.org/u/kevinranks)
#### Post date: [April 24, 2017, 4:24pm UTC](https://meta.discourse.org/t/onebox-user-agent/61498/5 "2017-04-24T16:24:54Z")
Pages are not prerendered by default. The default is to render the page in the browser (which your crawler cannot do). Therefore it’s neccessary to identify the user agents of crawlers that require the page to be prerendered.
Check out this official gist by [prerender.io](http://prerender.io) to see the nginx configuration to identify crawlers and serve a prendered page: [Official prerender.io nginx.conf for nginx · GitHub](https://gist.github.com/thoop/8165802)
The standard format for cawling bots can be seen here: [User agent - Wikipedia](https://en.wikipedia.org/wiki/User_agent#Format_for_automated_agents_.28bots.29)
> Googlebot/2.1 (+http://www.google.com/bot.html)
It’s recommended to include a url in the user agent string that provides information about what the bot does and how to block it if needed. Does Onebox obey robots.txt rules?
Anyways, I see no harm in at least updating the user agent to something like:
> Onebox/1.6.7 ([GitHub - discourse/onebox: (DEPRECATED) A gem for turning URLs into website previews · GitHub](https://github.com/discourse/onebox))
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [April 24, 2017, 4:37pm UTC](https://meta.discourse.org/t/onebox-user-agent/61498/6 "2017-04-24T16:37:20Z")
Sorry we don’t want this. Some sites block opengraph retrieval for any unknown user agents. So we want a user agent that is a common / popular device, which won’t be blocked.
---
### Author:  [@kevinranks](https://meta.discourse.org/u/kevinranks)
#### Post date: [April 24, 2017, 4:41pm UTC](https://meta.discourse.org/t/onebox-user-agent/61498/7 "2017-04-24T16:41:32Z")
Your user agent is currently “Ruby”. How is that common / popular?
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [April 24, 2017, 4:42pm UTC](https://meta.discourse.org/t/onebox-user-agent/61498/8 "2017-04-24T16:42:13Z")
If so then that’s a bug. Amazon for example bars it. Can you confirm on this @techapj?
---
### Author:  [@kevinranks](https://meta.discourse.org/u/kevinranks)
#### Post date: [April 24, 2017, 4:52pm UTC](https://meta.discourse.org/t/onebox-user-agent/61498/9 "2017-04-24T16:52:56Z")
Another option could be to add “Prerender” to the user agent. I’m guessing you won’t want to do this either, but I’m really just looking for a solution so I can support _your_ crawler.
It seems odd that you guys wouldn’t want to give developers with single page applications a way to support your crawler. I can understand why you would want to make sure your crawler doesn’t get blanket blocked, but there’s gotta be a compromising solution here…
---
### Author:  [@techAPJ](https://meta.discourse.org/u/techAPJ)
#### Post date: [April 24, 2017, 5:05pm UTC](https://meta.discourse.org/t/onebox-user-agent/61498/10 "2017-04-24T17:05:06Z")
Currently we are not specifying “User-Agent” header while making an HTTP request to retrieve response, so `Net::HTTP` library sends “Ruby” as default user agent.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [April 24, 2017, 5:17pm UTC](https://meta.discourse.org/t/onebox-user-agent/61498/11 "2017-04-24T17:17:33Z")
I don’t see much harm in adding an extra http header we send with our requests, I also don’t see much harm in allowing site owners to override user agent via a hidden site setting
But I don’t think the team will work on prototyping or testing either of these
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [April 24, 2017, 5:24pm UTC](https://meta.discourse.org/t/onebox-user-agent/61498/12 "2017-04-24T17:24:30Z")
I think the onebox library should offer this as a customizable string at minimum. Does not need to be exposed in Discourse per se..
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [April 24, 2017, 5:40pm UTC](https://meta.discourse.org/t/onebox-user-agent/61498/13 "2017-04-24T17:40:57Z")
> [@kevinranks](#):
>
> Anyways, I see no harm in at least updating the user agent to something like:
>
> > Onebox/1.6.7 ([GitHub - discourse/onebox: (DEPRECATED) A gem for turning URLs into website previews · GitHub](https://github.com/discourse/onebox))
Just a small info: I used to work with a big Akka Scala app, and the http library from akka freaks out with User Agents that contain `http://`, they say it’s [a spec violation](https://github.com/spray/spray/issues/458#issuecomment-23192180).
So maybe a simple default like `Discourse Onebox 1.6.7` is enough.
---
### Author:  [@kevinranks](https://meta.discourse.org/u/kevinranks)
#### Post date: [April 24, 2017, 5:49pm UTC](https://meta.discourse.org/t/onebox-user-agent/61498/14 "2017-04-24T17:49:02Z")
That works for me!
I didn’t dig into the RFC, but many of Google’s user agents contain urls. They are prefixed with +, if that makes any difference.
> **[Google Crawler (User Agent) Overview | Google Crawling Infrastructure | ...](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers?hl=en&visit_id=639159067748462140-3422325392&rd=1)**
>
> Google crawlers discover and scan websites. This overview will help you understand the common Google crawlers including the Googlebot user agent.
---
### Author:  [@fefrei](https://meta.discourse.org/u/fefrei)
#### Post date: [April 24, 2017, 7:34pm UTC](https://meta.discourse.org/t/onebox-user-agent/61498/15 "2017-04-24T19:34:37Z")
Wait, that bug report ist about `@` – and even that is allowed within `(` parenthesis `)`. It looks like in a `(``)`-delimited comment, `` is allowed 👍
---
### Author:  [@elijah](https://meta.discourse.org/u/elijah)
#### Post date: [April 24, 2017, 8:22pm UTC](https://meta.discourse.org/t/onebox-user-agent/61498/16 "2017-04-24T20:22:32Z")
> [@fefrei](#):
>
> It looks like in a ()-delimited comment,\ is allowed 👍
AND `(` and `)` are allowed in the comment under special circumstances:
1. When backslash escaped
2. When nesting comments
So, for example, my PSP’s built-in browser issues:
`User-Agent: Mozilla/4.0 (PSP (PlayStation Portable); 2.00)`
Which is apparently RFC compliant even though there are parens within parens.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [May 23, 2017, 10:14pm UTC](https://meta.discourse.org/t/onebox-user-agent/61498/17 "2017-05-23T22:14:58Z")
@eviltrout will be adding this feature tomorrow for a few different reasons.
---
### Author:  [@eviltrout](https://meta.discourse.org/u/eviltrout)
#### Post date: [May 24, 2017, 4:21pm UTC](https://meta.discourse.org/t/onebox-user-agent/61498/18 "2017-05-24T16:21:00Z")
This commit adds a custom User-Agent to discourse oneboxes:
[https://github.com/discourse/discourse/commit/8bc93c0b018045bf23769b8ecc4ec5b493368667](https://github.com/discourse/discourse/commit/8bc93c0b018045bf23769b8ecc4ec5b493368667)
The User agent looks like this:
```plaintext
Discourse Forum Onebox v1.8.0.beta13
```
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [August 5, 2017, 4:52am UTC](https://meta.discourse.org/t/onebox-user-agent/61498/21 "2017-08-05T04:52:01Z")