Some sites are blocked from oneboxing

I had an onebox preview issue since updating on May 4. Not all of hyperlinks can create preview box successfully.

However, those hyperlinks all work fine on meta until today, I finally found a hyperlink which have similar* problem on meta.
*different error code

Please advice. Thanks for help.

1 Like

I tried to Onebox that link on my forum and got the same result.
I noticed this right off when I checked out that link. It returned 2 exceptions. The link cannot be Oneboxed because of this. That’s why you/we got that message in the Onebox.

1 Like

@jamie.wilson has been investigating issues with medium oneboxes we have a fix in the pipeline

4 Likes

Yes, this looks like the same issue that generates a 500 error. Did we address this @jamie.wilson?

3 Likes

The link provided by @SkyLin is Oneboxing once more:

It looks like Medium made some changes on their end that exposed some shortcomings in how we generate Oneboxes. Those issues have now been addressed on our end and merged in, making Oneboxing more robust in general (and not just for Medium).

4 Likes

This issue has been solved after updating. Thanks a lot!

Comparing bbb44698...a24b6daa · discourse/discourse · GitHub

But my original problem* still exist. Is there any further action I can do to work this out? Thanks again.

*Onebox works fine on meta but failed on my site for a same url:

1 Like

I just tried all 3 of your links, the one for Medium and the 2 for Udemy, on my site and here on Meta. All 3 Oneboxed fine on my site and here on Meta.

2 Likes

Thanks for kind info.

I tried the way mentioned on Amazon onebox fails on my site, but works here and found 403 on udemy as return code. I even cloned a site with old backup and it has the same problem. I wonder if other reason cause the issue, say, setting?

Please advice. Thanks for help.

root@localhost:~# curl https://www.udemy.com/course/certified-kubernetes-application-developer/ -o deleteme.html --verbose
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:-- 0* Trying 2606:4700::6810:4255:443... * TCP_NODELAY set
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--  0* Connected to www.udemy.com (2606:4700::6810:4255) port 443  (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [3726 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Udemy, Inc.; CN=*.udemy.com
*  start date: Sep 10 15:45:40 2020 GMT
*  expire date: Oct 12 15:45:40 2021 GMT
*  subjectAltName: host "www.udemy.com" matched cert's "*.udemy.com"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign RSA OV SSL CA 2018
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55ba70848e10)
} [5 bytes data]
> GET /course/certified-kubernetes-application-developer/ HTTP/2
> Host: www.udemy.com
> user-agent: curl/7.68.0
> accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
< HTTP/2 403
< date: Sat, 15 May 2021 09:15:50 GMT
< content-type: text/plain; charset=UTF-8
< content-length: 16
< x-frame-options: SAMEORIGIN
< cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-c                                                                             heck=0, pre-check=0
< expires: Thu, 01 Jan 1970 00:00:01 GMT
< cf-request-id: 0a10e892d400000ac0a696a000000001
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-c                                                                             gi/beacon/expect-ct"
< set-cookie: __cf_bm=b04492251e1a16bfdad619155f03441f5707d134-1621070150-1800-AS6cROKp+jLfN9ubIJem+TuX5dANVYbshBzqeRTZhheBx6lOtfM8RPRDSGDLEObLD8tWwjbrp+D5Sh4RcCTXGNw=; path=/; expires=Sat, 15-May-21 09:45:50 GMT; domain=.udemy.com; HttpOnly; Secure; SameSite=None
< x-content-type-options: nosniff
< server: cloudflare
< cf-ray: 64fb4397bea70ac0-NRT
<
{ [16 bytes data]
100    16  100    16    0     0     10      0  0:00:01  0:00:01 --:--:--    10
* Connection #0 to host www.udemy.com left intact

1 Like

Where did that come from? 1970? That’s over 50 years ago.

2 Likes

Comes from the response of this command:

curl https://www.udemy.com/course/certified-kubernetes-application-developer/ -o deleteme.html --verbose

1 Like

And here comes the response of Amazon with 503 error code.

root@localhost:~# curl https://www.amazon.com/Deep-Learning-Python-Francois-Chollet/dp/1617294438 -o deleteme.html --verbose
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 162.219.225.118:443...
* TCP_NODELAY set
* Connected to www.amazon.com (162.219.225.118) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [4447 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=www.amazon.com
*  start date: Apr 26 00:00:00 2021 GMT
*  expire date: Apr 25 23:59:59 2022 GMT
*  subjectAltName: host "www.amazon.com" matched cert's "www.amazon.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global CA G2
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55993f5fae10)
} [5 bytes data]
> GET /Deep-Learning-Python-Francois-Chollet/dp/1617294438 HTTP/2
> Host: www.amazon.com
> user-agent: curl/7.68.0
> accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [177 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
} [5 bytes data]
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
< HTTP/2 503
< server: Server
< date: Mon, 17 May 2021 01:37:18 GMT
< content-type: text/html
< strict-transport-security: max-age=47474747; includeSubDomains; preload
< x-amz-rid: 72RYS75V0EMTMX4RK38T
< vary: Content-Type,Accept-Encoding,X-Amzn-CDN-Cache,X-Amzn-AX-Treatment,User-Agent
< last-modified: Wed, 30 Sep 2020 23:54:00 GMT
< etag: "a6f-5b0909d09d600"
< accept-ranges: bytes
< content-length: 2671
<
{ [1139 bytes data]
100  2671  100  2671    0     0   3604      0 --:--:-- --:--:-- --:--:--  3599
* Connection #0 to host www.amazon.com left intact
root@localhost:~#

And one thing, the onebox preview for Amazon is OK now after Jamie’s update code. The preview before updating, please refer to this image.

1 Like

Good on the Amazon link. Are the udemy links working now or are you still getting a 403 on those? I still don’t have any problem Oneboxing those.

1 Like

Thanks again.

All udemy hyperlinks failed on my site.

I just tried another forum based on discourse and found the same error code 403.

forum site: https://forum.quantifiedself.com/

testing 1: OK
Amazon https://www.amazon.com/Deep-Learning-Python-Francois-Chollet/dp/1617294438

testing 2: NG with error code 500. I guess this site doesn’t upgrade to the most update version yet.
Medium 15 Free Courses to Learn Python in 2021 | by javinpaul | The Startup | Medium

testing 3: NG, the same error code 403, which is exactly the same as mine.
udemy Kubernetes Certified Application Developer (CKAD) Training | Udemy

1 Like

I test all discourse based sites I know, the result as below:

onebox site 1 site 2 site 3 site 4 site 5 site 6 site 7 site 8
Amazon OK OK OK OK OK OK OK OK
medium OK NG OK NG NG OK NG NG
udemy NG NG OK OK OK OK NG OK
location* Tokyo California California California California Paris California England
Ireland Virginia Chicago
Africa Alaska

*test with Check host

site 1. https://vip.studycamp.tw/
site 2. https://forum.quantifiedself.com/
site 3. https://discussions.udacity.com/
site 4. https://forum.uipath.com/
site 5. https://forums.envato.com/
site 6. https://rideabl.es/
site 7. https://blenderartists.org/
site 8. https://se23.life/

4 Likes

You are lucky, the only one that works is the amazon.com
With all other amazon sites the onebox preview does not work.

2 Likes

The sites you’re testing against may be blocking the curl user agent. The Oneboxing process will use the user agent from a real browser, depending on the URL you’re trying to Onebox. At the moment, the user agent it uses in such cases is:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15

Try running your curl tests with the -A flag. e.g.:

curl -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15" https://www.udemy.com/course/certified-kubernetes-application-developer/ -o deleteme.html --verbose

What is the result of that command on your end?

If it’s still failing:

  1. Have you been running an unusually large number of Oneboxing requests against these sites? You may have been temporarily blocked by the sites in question and find things start working after waiting an appropriate period of time (this is up to the site operators, and there’s nothing we can do about it from our end unfortunately).

  2. Where is your server located? Do you have access to a machine in a different location that you run tests from? Even testing from your personal machine instead of your production server may get you different results.

4 Likes

Thanks for advice.

The return code is still 403:

root@localhost:~# curl -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15" https://www.udemy.com/course/certified-kubernetes-application-developer/ -o deleteme.html --verbose
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 2606:4700::6810:4155:443...
* TCP_NODELAY set
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0* Connected to www.udemy.com (2606:4700::6810:4155) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [3726 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Udemy, Inc.; CN=*.udemy.com
*  start date: Sep 10 15:45:40 2020 GMT
*  expire date: Oct 12 15:45:40 2021 GMT
*  subjectAltName: host "www.udemy.com" matched cert's "*.udemy.com"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign RSA OV SSL CA 2018
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x5591c76d8e10)
} [5 bytes data]
> GET /course/certified-kubernetes-application-developer/ HTTP/2
> Host: www.udemy.com
> user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15
> accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
< HTTP/2 403
< date: Mon, 17 May 2021 23:59:37 GMT
< content-type: text/plain; charset=UTF-8
< content-length: 16
< x-frame-options: SAMEORIGIN
< cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< expires: Thu, 01 Jan 1970 00:00:01 GMT
< cf-request-id: 0a1e5e6b920000952193363000000001
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< set-cookie: __cf_bm=3224e014d6b0b4eca1cf8a59c6c180928b2ffa0f-1621295977-1800-AU4UK/k3sl61kbMPYizcsl1TI7NAHkvE1mfjoKIPdrvGAJJeYUF8e72hQljXNwVTO+BcqfsAXxoiQDEy1cL+aVA=; path=/; expires=Tue, 18-May-21 00:29:37 GMT; domain=.udemy.com; HttpOnly; Secure; SameSite=None
< x-content-type-options: nosniff
< server: cloudflare
< cf-ray: 6510ccf28d0a9521-NRT
<
{ [16 bytes data]
100    16  100    16    0     0     10      0  0:00:01  0:00:01 --:--:--    10
* Connection #0 to host www.udemy.com left intact

Yes, I did. But the clone site didn’t post any content more than 20 days, it has the same problem.

Besides, based on the testing table I mentioned yesterday, site 2 & site 7 have the same issue and I believe it’s first udemy related post on these sites.

My server site is located in Japan. I can move the clone site to other area like US for testing later today, and I will update again then.

Update:

  1. After 11 hours, it’s the same 403 after moving cloned testing site from Tokyo to California. I will move to Europe for testing later, and I will update again tomorrow)
  2. The same 403 error code in London.

Thanks again.

2 Likes

I’ve done a little more testing from my side.

I can successfully curl the udemy URL from my residential ISP (North America) and also from a machine hosted at a fancy datacenter (North America).

Attempting to do so from a popular hosting provider (North America) fails. Attempting to do so from a different popular hosting provider (Europe) also fails.

When it fails, the content body that is returned to me is:

error code: 1020

which is coming from:

This post from Cloudflare explains what their 1020 error means, and what you can do to try to resolve it:

5 Likes

Currently 2.7.0.beta8, if that helps at all.

2 Likes