Since a few days (I’d say just after updating to Discourse 2.5 beta 5 - and continues on beta 6 - but it can well be a coincidence) embedding of YouTube videos works itermittently.
After a few hours not working, it just restarts embedding OK.
Since yesterday, it stopped working completely.
I don’t see any specific suspicious error in the forum logs.
Could it be a timeout problem? Is there a way to specifically investigate this?
Thanks in advance!
I found something odd by examining the traffic created by requesting a rebake of a post containing a youtube link.
The request fails with error 404. 
Another little bit of extra detail.
I’m running Discourse 2.5.0.beta6 updated to commit 2d880b42a3 (rebuilt just before sending this post).
When creating a new post with a YouTube link, from the Google Chrome console I have an extra error just before the 404 for the GET to onebox, referring to a Service Worker registration error.
I want to point out that any onebox works but YouTube. 
The browser can’t give you the real reason a onebox is failing. You need to try and make a similar request from your server.
I understand what you are suggesting, but:
if I repeat the simple GET call (passing headers and stuff as expected) to /onebox?url=… i simply get the error HTML 404
if I repeat, on the server, the request that onebox does, copying it from the ruby code of youtube_onebox.rb, that is curl 'curl 'https://www.youtube.com/oembed?format=json&url=https://www.youtube.com/watch?v=Xl-PTTeRsik' for one of the non-ebmedded video, it seems to work perfectly.
The call actually returns:
{"author_name":"AstronautiCAST","version":"1.0","height":270,"author_url":"https:\/\/www.youtube.com\/user\/AstronautiCAST","provider_name":"YouTube","provider_url":"https:\/\/www.youtube.com\/","thumbnail_height":360,"width":480,"html":"\u003ciframe width=\"480\" height=\"270\" src=\"https:\/\/www.youtube.com\/embed\/Xl-PTTeRsik?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen\u003e\u003c\/iframe\u003e","type":"video","thumbnail_width":480,"title":"Loading cargo into HTV-9 Konutori","thumbnail_url":"https:\/\/i.ytimg.com\/vi\/Xl-PTTeRsik\/hqdefault.jpg"}root@fait-2020:/var/discourse#
which contains all that onebox needs for embedding.
I don’t get it
@Falco really sorry to bother you, but the lack of youtube oneboxing is really crippling the user experience of our forum.
Yesterday I tried several things, such as trying to read the source code (to understand at what stage and what kind of condition could trigger the return of a 404 from onebox) and up to the point of cloning the current production forum to a new, separate Digital Ocean droplet.
Reading the code was not really helping, mainly due to my lack of real knowledge of the platform.
One question remains open on this: is the oneboxing process logging somewhere? It would be so useful to understand the error that brings onebox to give up and return 404.
The clone Droplet was a tentative to check whether the IP of the productions server was banned by Youtube and at the same time whether there was something related to the domain name we use.
Well, even though IP address and domain name were different, the YouTube oneboxing was not working.
I really hope somebody can suggest me at least a way to trace what the oneboxing does.
I have the root cause: Youtube banned our IP, but it’s not clear why as at the time this happened we were not doing any massive rebake or similar.
Question for @Falco or @codinghorror: was the install of 2.5.0beta 5 or 6 triggering any rebake, which would have exceeded the request limit?
No we didn’t add any recent rebakes. Sound like a YouTube change, as other users complained too.
Thanks everybody for the replies.
It remains unclear to me though why if I mimic the request that the onebox engine does (or at least I expect this to be the case. Is it @Falco?) then I get a JSON response with a proper answer, and not a 429.
Is there another request done by Onebox that gets a 429, before performing the request as per my screenshot, that is curl 'https://www.youtube.com/oembed?format=json&url=https://www.youtube.com/watch?v=Xl-PTTeRsik' ?
It goes without saying that this requests were made from the very server running Discourse (so same outgoing IP).
One curl statement is unlikely trip a rate limit?
Try a series of them in quick succession? Run it in a repetitive bash script? (But not too much or you might get banned)
Well, simply trying to recook a single post with a single link to youtube gets us a 404 from onebox…
An interesting finding after a weekend of troubleshooting.
This can maybe be something you want to have a look into @Falco or @codinghorror?
At the moment, reading the source code of youtube_onebox.rb , I see it supports 3 URLs from which the video id is extracted.
http://youtu.be/<videoid>https://www.youtube.com/embed/<videoid>https://www.youtube.com/watch?v=<videoid>Every attempt to onebox videos whose links are in 1. and 3. formats fail with onebox returning 404 (which I think is related to our IP being banned)
When I try to embed links having format as 2., it works!
I wonder if and how this could be related to the fact explained in this post.
Some understanding (and logging) of the inner working of onebox (i.e. what exact call(s) is(are) made to youtube) would be super useful…
Back here to give closure to this thread.
Yesterday YouTube removed our ban and we’re back in business.
There are a couple of points that, in my view, would be interesting to understand anyhow:
/embed/ URL never stopped working somehow, even when banned).Thanks all which answered my panicked questions! 
THIS!!! (is a very good idea I’ve mentioned it before)
I guess part of the challenge here is if the target returns a page without og tags, Onebox will not know that this was because of some redirect. However, that could be logged (“ONEBOX: no og tags found”?) and all errors that lead to blank oneboxes should definitely be logged.
tl;dr I’d like to add that we’re experiencing what seems to be the same problem here. If there is a rate-limit issue due to some recent change then I think other users will start to experience this during migration, re-baking of posts or perhaps just down to a really busy forum. The fact that onebox seemingly fails silently means that these issues aren’t visible until users start to complain that YouTube oneboxes are missing.
We’re on 2.6.0.beta 1
Users were getting messages about non-secure content. On investigation, Chrome seemed to be complaining about images linked from HTTP sites. So I configured Discourse to download all images/media and serve them over HTTPS.
Once I’d changed the setting, this meant doing a re-bake of historic posts. Since that rebake, a large chunk of YouTube videos that were previously onebox’d have now changed back to the linked URL
We have one thread of 10,000 posts that consists of solely YouTube video replies and all posts are URLs and not oneboxes.
During the re-bake, all queued jobs processed organically so it’s not jobs stuck in a queue of deleted jobs.
I haven’t seen the same error messages that @marcozambi described, but I believe we are tripping a rate limit too.
In support of this rate-limit theory, a small piece of code I wrote to re-bake posts worked (onebox’d) for the first 80+ YouTube videos in a thread then failed to convert the remaining videos.
At that point, even editing the post, making a small amendment and resaving did not force the URL to be onebox ‘expanded’. At the same time, all queues were empty or had minor jobs being instantly processed as I would expect.
Attempts to re-run that code over a 30 minute period failed to force the oneboxing of the links. I don’t think 80 is a magic number here, just what was available from the quota we had.
@marcozambi mentioned that the /embed/ format YouTube link worked when others failed, so I amended the code to use a regex search-and-replace of YouTube links to turn them into the /embed/ format.
The code worked.
Re-running the code to just rebake the posts again failed to turn into onebox representations.
My plan is to experiment with a task that converts all YouTube links in the large thread to the /embed/ YouTube format. If that fails or we trip a higher rate limit, then I’ll take a look at @merefield’s Onebox Assistant.
I’ll post an update later.
OK, there is certainly something strange going on and it appears to be rate-limit related.
I’m not sure if we’re being rate-limited because I did a massive rebake and we’ve gone on the naughty step, or if we’re tripping limits that others will see.
Oneboxing of YouTube videos seems to have a limit and once that limit is reached the Oneboxing fails silently.
I feel this has to be changed for hopefully obvious reasons, but specifically for anyone that does a migration or rebake that will have no idea that lots of un-expanded or once-expanded Oneboxes are now just vanilla URLs.
@marcozambi mentioned above that the YouTube URL format that features /embed/ before the video ID works when the other formats are failing (presumably due to a rate-limit issue).
Here’s a video that illustrates that phenomenon well.
When this screencast was captured, there were no jobs clogging the queues and the forum was otherwise performing well.
Prior to this video, YouTube links had already started to fail to be expanded by OneBox.
What you will see is the compose window where Onebox fails to expand a YouTube link in the https://youtu.be/<video-id> format.
I then change the format to be in the https://youtube.com/embed/<video-id> format and Onebox expands it.
I then try again with the original format and it fails.
During this video capture, I tracked the browser console and network tabs. I recognise that the issue is surely between our server and YouTube rather than between my browser and our server, but I include them below in case they are useful.
(apologies for the zoomed-out nature of the image - I hope they’re visible when zoomed-in)
And here is the network trace when the Onebox worked.
I’m not convinced that the /embed/ format of link is a panacea here,
I think it seems to be a route that has separate rate limits such that when the https://youtu.be/<video-id> route hits a limit, there is another route with a separate limit on the https://youtube.com/embed/<video-id> route.
Evidence of the fact that both routes are limited comes from a utility I wrote to change the format of the YouTube embeds on a monster 10K post thread featuring 99% YouTube video replies.
At this stage Onebox was already failing to expand the https://youtu.be/<video-id> formatted links.
My utility, which changed the YouTube video URL to the https://youtube.com/embed/<video-id> format ran against the first 3000 posts in the thread.
It worked well for the first 1108 and then, whilst it changed the format for the next ~1900 posts, but they were not expanded by Onebox.
During this time, lots of jobs were generated (my code used post.revise) and all processed without error or retry.
Anecdotally, I noticed that job processing seemed to dramatically accelerate at a certain stage. I guess this might have been because the Onebox code was quickly getting some form of error from YouTube - but I didn’t time it and it could have been a number of things.
I’d be happy to try to supply more detailed evidence here, but not sure what I can do without instrumenting the Onebox gem.
I’m a hacker and not a Ruby expert but I’d gladly try to follow some high-level instructions.
Performing some short repetitive curl scripts from the servers command line with the same user agent might allow you to isolate a rate limit issue.
Agree that the workaround is probably working just because it’s a separate count.
Here are some more results. Note, there are many assumptions in the post below - based on a lack of real knowledge.
I’ll follow this post up with my opinion of what is going on and what should happen.
Thanks for the response, Robert.
Note that Oneboxing of videos using the /watch route were (and still are!) failing when I tried this so I didn’t need a loop to force it to fail.
So one assumption I’ve made is that the user-agent that Onebox is using is Discourse Forum Onebox v2.6.0.beta1 based on this code…
I picked a random video and attempted to use curl to read the headers.
I did this from inside the Docker container on my live site which produced the following response.
command
curl --user-agent "Discourse Forum Onebox v2.6.0.beta1" -sD - -o /dev/null "https://m.youtube.com/watch?v=s0ONj4TG0UA"
response:
curl --user-agent "Discourse Forum Onebox v2.6.0.beta1" -sD - -o /dev/null "https://m.youtube.com/watch?v=s0ONj4TG0UA"
HTTP/2 303
content-length: 0
p3p: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en-GB for more info."
cache-control: no-cache
x-frame-options: SAMEORIGIN
content-type: text/html; charset=utf-8
location: https://www.youtube.com/watch?v=s0ONj4TG0UA&app=desktop
accept-ch-lifetime: 2592000
x-content-type-options: nosniff
accept-ch: DPR
expires: Tue, 27 Apr 1971 19:44:06 GMT
strict-transport-security: max-age=31536000
date: Fri, 07 Aug 2020 11:35:21 GMT
server: YouTube Frontend Proxy
x-xss-protection: 0
set-cookie: VISITOR_INFO1_LIVE=rcVTSJn81Ck; path=/; domain=.youtube.com; secure; expires=Wed, 03-Feb-2021 11:35:20 GMT; httponly; samesite=None
set-cookie: YSC=cFXIPerzT3Y; path=/; domain=.youtube.com; secure; httponly; samesite=None
set-cookie: GPS=1; path=/; domain=.youtube.com; expires=Fri, 07-Aug-2020 12:05:20 GMT
alt-svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
So I was redirected using a 303 response to the URL in the the location header which was https://www.youtube.com/watch?v=s0ONj4TG0UA&app=desktop.
This simply had the effect of appending &app=desktop to the URL.
command
curl --user-agent "Discourse Forum Onebox v2.6.0.beta1" -sD - -o /dev/null "https://www.youtube.com/watch?v=s0ONj4TG0UA&app=desktop"
response
HTTP/2 429
x-content-type-options: nosniff
expires: Tue, 27 Apr 1971 19:44:06 GMT
x-frame-options: SAMEORIGIN
cache-control: no-cache
p3p: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en-GB for more info."
accept-ch-lifetime: 2592000
content-type: text/html; charset=utf-8
accept-ch: DPR
strict-transport-security: max-age=31536000
content-length: 48982
date: Fri, 07 Aug 2020 11:46:00 GMT
server: YouTube Frontend Proxy
x-xss-protection: 0
set-cookie: VISITOR_INFO1_LIVE=VQwNuouhq-s; path=/; domain=.youtube.com; secure; expires=Wed, 03-Feb-2021 11:46:00 GMT; httponly; samesite=None
set-cookie: YSC=8IRfPRFRY6c; path=/; domain=.youtube.com; secure; httponly; samesite=None
set-cookie: GPS=1; path=/; domain=.youtube.com; expires=Fri, 07-Aug-2020 12:16:00 GMT
set-cookie: VISITOR_INFO1_LIVE=VQwNuouhq-s; path=/; domain=.youtube.com; secure; expires=Wed, 03-Feb-2021 11:46:00 GMT; httponly; samesite=None
set-cookie: YSC=8IRfPRFRY6c; path=/; domain=.youtube.com; secure; httponly; samesite=None
set-cookie: GPS=1; path=/; domain=.youtube.com; expires=Fri, 07-Aug-2020 12:16:00 GMT
alt-svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
So I am being sent a 429 “too many requests” response code, but without being sent a retry-after header - cease and desist with no negotiation.
Either way, if this is what Onebox is seeing, it’s ignoring the response or at least I don’t know where to go looking for it if it is being logged.
Whilst this might be a legitimate thing to do for a single 429, seeing many 429 responses in a very short period of time cannot be ignored.
For completeness, I immediately tried to get the same video but this time using the /embed/ route.
command
curl --user-agent "Discourse Forum Onebox v2.6.0.beta1" -sD - -o /dev/null "https://www.youtube.com/embed/s0ONj4TG0UA"
response
HTTP/2 200
accept-ch-lifetime: 2592000
content-type: text/html; charset=utf-8
expires: Tue, 27 Apr 1971 19:44:06 GMT
x-content-type-options: nosniff
cache-control: no-cache
p3p: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en-GB for more info."
strict-transport-security: max-age=31536000
accept-ch: DPR
date: Fri, 07 Aug 2020 11:55:29 GMT
server: YouTube Frontend Proxy
x-xss-protection: 0
set-cookie: VISITOR_INFO1_LIVE=PNE6x6djF00; path=/; domain=.youtube.com; secure; expires=Wed, 03-Feb-2021 11:55:29 GMT; httponly; samesite=None
set-cookie: VISITOR_INFO1_LIVE=PNE6x6djF00; path=/; domain=.youtube.com; secure; expires=Wed, 03-Feb-2021 11:55:29 GMT; httponly; samesite=None
set-cookie: GPS=1; path=/; domain=.youtube.com; expires=Fri, 07-Aug-2020 12:25:29 GMT
set-cookie: YSC=pDW-hdbauK8; path=/; domain=.youtube.com; secure; httponly; samesite=None
alt-svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
accept-ranges: none
vary: Accept-Encoding
200 - Success.
Not sure if this is of any significance at all but…is the embedded plugin lazy-yt enabled by default? I noticed it in my development installation.
It seems to monkey patch the to_html method of the YouTube Oneboxer.
I don’t know if it is significant, but the original Onebox’s to_html method returns the /embed/ URL format…
Whereas the lazy-yt plugin uses the /watch?v= URL format.
Is there anything else I can do to show there is a problem that needs some form of attention? Next post will explain what I think is the root cause.