Youtube videos onebox embedding stopped working

bletch · August 7, 2020, 12:27pm

Here are some more results. Note, there are many assumptions in the post below - based on a lack of real knowledge.

I’ll follow this post up with my opinion of what is going on and what should happen.

Thanks for the response, Robert.

Note that Oneboxing of videos using the /watch route were (and still are!) failing when I tried this so I didn’t need a loop to force it to fail.

So one assumption I’ve made is that the user-agent that Onebox is using is Discourse Forum Onebox v2.6.0.beta1 based on this code…

github.com

discourse/discourse/blob/6a417c308f8abf30b0dec712426a009cabce859e/config/initializers/100-onebox_options.rb#L14


      
            Onebox.options = {
              twitter_client: TwitterApi,
              redirect_limit: 3,
              user_agent: "Discourse Forum Onebox v#{Discourse::VERSION::STRING}",
              allowed_ports: [80, 443, SiteSetting.port.to_i]
            }
          else
            Onebox.options = {
              twitter_client: TwitterApi,
              redirect_limit: 3,
              user_agent: "Discourse Forum Onebox v#{Discourse::VERSION::STRING}"
            }
          end

I picked a random video and attempted to use curl to read the headers.

I did this from inside the Docker container on my live site which produced the following response.

Result of first curl using /watch? route

command

curl --user-agent "Discourse Forum Onebox v2.6.0.beta1" -sD - -o /dev/null "https://m.youtube.com/watch?v=s0ONj4TG0UA"

response:

curl --user-agent "Discourse Forum Onebox v2.6.0.beta1" -sD - -o /dev/null "https://m.youtube.com/watch?v=s0ONj4TG0UA"
HTTP/2 303 
content-length: 0
p3p: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en-GB for more info."
cache-control: no-cache
x-frame-options: SAMEORIGIN
content-type: text/html; charset=utf-8
location: https://www.youtube.com/watch?v=s0ONj4TG0UA&app=desktop
accept-ch-lifetime: 2592000
x-content-type-options: nosniff
accept-ch: DPR
expires: Tue, 27 Apr 1971 19:44:06 GMT
strict-transport-security: max-age=31536000
date: Fri, 07 Aug 2020 11:35:21 GMT
server: YouTube Frontend Proxy
x-xss-protection: 0
set-cookie: VISITOR_INFO1_LIVE=rcVTSJn81Ck; path=/; domain=.youtube.com; secure; expires=Wed, 03-Feb-2021 11:35:20 GMT; httponly; samesite=None
set-cookie: YSC=cFXIPerzT3Y; path=/; domain=.youtube.com; secure; httponly; samesite=None
set-cookie: GPS=1; path=/; domain=.youtube.com; expires=Fri, 07-Aug-2020 12:05:20 GMT
alt-svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

So I was redirected using a 303 response to the URL in the the location header which was https://www.youtube.com/watch?v=s0ONj4TG0UA&app=desktop.

This simply had the effect of appending &app=desktop to the URL.

Result of the second curl to redirected URL - still on the /watch? route

command
curl --user-agent "Discourse Forum Onebox v2.6.0.beta1" -sD - -o /dev/null "https://www.youtube.com/watch?v=s0ONj4TG0UA&app=desktop"

response

HTTP/2 429 
x-content-type-options: nosniff
expires: Tue, 27 Apr 1971 19:44:06 GMT
x-frame-options: SAMEORIGIN
cache-control: no-cache
p3p: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en-GB for more info."
accept-ch-lifetime: 2592000
content-type: text/html; charset=utf-8
accept-ch: DPR
strict-transport-security: max-age=31536000
content-length: 48982
date: Fri, 07 Aug 2020 11:46:00 GMT
server: YouTube Frontend Proxy
x-xss-protection: 0
set-cookie: VISITOR_INFO1_LIVE=VQwNuouhq-s; path=/; domain=.youtube.com; secure; expires=Wed, 03-Feb-2021 11:46:00 GMT; httponly; samesite=None
set-cookie: YSC=8IRfPRFRY6c; path=/; domain=.youtube.com; secure; httponly; samesite=None
set-cookie: GPS=1; path=/; domain=.youtube.com; expires=Fri, 07-Aug-2020 12:16:00 GMT
set-cookie: VISITOR_INFO1_LIVE=VQwNuouhq-s; path=/; domain=.youtube.com; secure; expires=Wed, 03-Feb-2021 11:46:00 GMT; httponly; samesite=None
set-cookie: YSC=8IRfPRFRY6c; path=/; domain=.youtube.com; secure; httponly; samesite=None
set-cookie: GPS=1; path=/; domain=.youtube.com; expires=Fri, 07-Aug-2020 12:16:00 GMT
alt-svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

So I am being sent a 429 “too many requests” response code, but without being sent a retry-after header - cease and desist with no negotiation.

Either way, if this is what Onebox is seeing, it’s ignoring the response or at least I don’t know where to go looking for it if it is being logged.

Whilst this might be a legitimate thing to do for a single 429, seeing many 429 responses in a very short period of time cannot be ignored.

Result of third curl - this time using the /embed/ route

For completeness, I immediately tried to get the same video but this time using the /embed/ route.

command

curl --user-agent "Discourse Forum Onebox v2.6.0.beta1" -sD - -o /dev/null "https://www.youtube.com/embed/s0ONj4TG0UA"

response

HTTP/2 200 
accept-ch-lifetime: 2592000
content-type: text/html; charset=utf-8
expires: Tue, 27 Apr 1971 19:44:06 GMT
x-content-type-options: nosniff
cache-control: no-cache
p3p: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en-GB for more info."
strict-transport-security: max-age=31536000
accept-ch: DPR
date: Fri, 07 Aug 2020 11:55:29 GMT
server: YouTube Frontend Proxy
x-xss-protection: 0
set-cookie: VISITOR_INFO1_LIVE=PNE6x6djF00; path=/; domain=.youtube.com; secure; expires=Wed, 03-Feb-2021 11:55:29 GMT; httponly; samesite=None
set-cookie: VISITOR_INFO1_LIVE=PNE6x6djF00; path=/; domain=.youtube.com; secure; expires=Wed, 03-Feb-2021 11:55:29 GMT; httponly; samesite=None
set-cookie: GPS=1; path=/; domain=.youtube.com; expires=Fri, 07-Aug-2020 12:25:29 GMT
set-cookie: YSC=pDW-hdbauK8; path=/; domain=.youtube.com; secure; httponly; samesite=None
alt-svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
accept-ranges: none
vary: Accept-Encoding

200 - Success.

lazy-yt plugin seems to re-write URLs in /watch format

Not sure if this is of any significance at all but…is the embedded plugin lazy-yt enabled by default? I noticed it in my development installation.

It seems to monkey patch the to_html method of the YouTube Oneboxer.

I don’t know if it is significant, but the original Onebox’s to_html method returns the /embed/ URL format…

github.com

discourse/onebox/blob/eb783a5ccf38b20224294b14c068b3ba01ff2579/lib/onebox/engine/youtube_onebox.rb#L32


      
            else
              to_html
            end
          end
          
          def to_html
            if video_id
              <<-HTML
                <iframe width="#{WIDTH}"
                        height="#{HEIGHT}"
                        src="https://www.youtube.com/embed/#{video_id}?#{embed_params}"
                        frameborder="0"
                        allowfullscreen>
                </iframe>
              HTML
            elsif list_id
              <<-HTML
                <iframe width="#{WIDTH}"
                        height="#{HEIGHT}"
                        src="https://www.youtube.com/embed/videoseries?list=#{list_id}&wmode=transparent&rel=0&autohide=1&showinfo=1&enablejsapi=1"
                        frameborder="0"

Whereas the lazy-yt plugin uses the /watch?v= URL format.

github.com

discourse/discourse/blob/b86198198fe904899f6b31edfbb7c05873a95625/plugins/lazy-yt/plugin.rb#L43


      
          
            # Put in the LazyYT div instead of the iframe
            escaped_title = ERB::Util.html_escape(video_title)
          
            <<~EOF
            <div class="onebox lazyYT lazyYT-container"
                 data-youtube-id="#{video_id}"
                 data-youtube-title="#{escaped_title}"
                 #{size_restricted ? size_tags.map { |t| "data-#{t}" }.join(' ') : ""}
                 data-parameters="#{embed_params}">
              <a href="https://www.youtube.com/watch?v=#{video_id}" target="_blank">
                <img class="ytp-thumbnail-image"
                     src="#{thumbnail_url}"
                     #{size_restricted ? size_tags.join(' ') : ""}
                     title="#{escaped_title}">
              </a>
            </div>
            EOF
          else
            yt_onebox_to_html
          end

Is there anything else I can do to show there is a problem that needs some form of attention? Next post will explain what I think is the root cause.

Topic		Replies	Views
Youtube embeds missing Support	51	3845	June 8, 2024
Youtube embeddings have stopped working for servers in Europe Bug	38	4306	April 10, 2021
Instagram oneboxing Feature	111	10942	November 27, 2020
Onebox Twitter embedding Feature	45	9799	October 5, 2020
Twitter Native Embed Theme component	81	4920	August 14, 2024

Youtube videos onebox embedding stopped working

Result of first curl using /watch? route

Result of the second curl to redirected URL - still on the /watch? route

Result of third curl - this time using the /embed/ route

lazy-yt plugin seems to re-write URLs in /watch format

Related topics