I had an onebox preview issue since updating on May 4. Not all of hyperlinks can create preview box successfully.
However, those hyperlinks all work fine on meta until today, I finally found a hyperlink which have similar* problem on meta.
*different error code
Please advice. Thanks for help.
I tried to Onebox that link on my forum and got the same result.
I noticed this right off when I checked out that link. It returned 2 exceptions. The link cannot be Oneboxed because of this. That’s why you/we got that message in the Onebox.
@jamie.wilson has been investigating issues with medium oneboxes we have a fix in the pipeline
Yes, this looks like the same issue that generates a 500 error. Did we address this @jamie.wilson?
The link provided by @SkyLin is Oneboxing once more:
It looks like Medium made some changes on their end that exposed some shortcomings in how we generate Oneboxes. Those issues have now been addressed on our end and merged in, making Oneboxing more robust in general (and not just for Medium).
This issue has been solved after updating. Thanks a lot!
Comparing bbb44698...a24b6daa · discourse/discourse · GitHub
But my original problem* still exist. Is there any further action I can do to work this out? Thanks again.
*Onebox works fine on meta but failed on my site for a same url:
I just tried all 3 of your links, the one for Medium and the 2 for Udemy, on my site and here on Meta. All 3 Oneboxed fine on my site and here on Meta.
Thanks for kind info.
I tried the way mentioned on Amazon onebox fails on my site, but works here and found 403 on udemy as return code. I even cloned a site with old backup and it has the same problem. I wonder if other reason cause the issue, say, setting?
Please advice. Thanks for help.
root@localhost:~# curl https://www.udemy.com/course/certified-kubernetes-application-developer/ -o deleteme.html --verbose
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 2606:4700::6810:4255:443... * TCP_NODELAY set
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0* Connected to www.udemy.com (2606:4700::6810:4255) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [3726 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=California; L=San Francisco; O=Udemy, Inc.; CN=*.udemy.com
* start date: Sep 10 15:45:40 2020 GMT
* expire date: Oct 12 15:45:40 2021 GMT
* subjectAltName: host "www.udemy.com" matched cert's "*.udemy.com"
* issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign RSA OV SSL CA 2018
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55ba70848e10)
} [5 bytes data]
> GET /course/certified-kubernetes-application-developer/ HTTP/2
> Host: www.udemy.com
> user-agent: curl/7.68.0
> accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
< HTTP/2 403
< date: Sat, 15 May 2021 09:15:50 GMT
< content-type: text/plain; charset=UTF-8
< content-length: 16
< x-frame-options: SAMEORIGIN
< cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-c heck=0, pre-check=0
< expires: Thu, 01 Jan 1970 00:00:01 GMT
< cf-request-id: 0a10e892d400000ac0a696a000000001
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-c gi/beacon/expect-ct"
< set-cookie: __cf_bm=b04492251e1a16bfdad619155f03441f5707d134-1621070150-1800-AS6cROKp+jLfN9ubIJem+TuX5dANVYbshBzqeRTZhheBx6lOtfM8RPRDSGDLEObLD8tWwjbrp+D5Sh4RcCTXGNw=; path=/; expires=Sat, 15-May-21 09:45:50 GMT; domain=.udemy.com; HttpOnly; Secure; SameSite=None
< x-content-type-options: nosniff
< server: cloudflare
< cf-ray: 64fb4397bea70ac0-NRT
<
{ [16 bytes data]
100 16 100 16 0 0 10 0 0:00:01 0:00:01 --:--:-- 10
* Connection #0 to host www.udemy.com left intact
Where did that come from? 1970? That’s over 50 years ago.
Comes from the response of this command:
curl https://www.udemy.com/course/certified-kubernetes-application-developer/ -o deleteme.html --verbose
And here comes the response of Amazon with 503 error code.
root@localhost:~# curl https://www.amazon.com/Deep-Learning-Python-Francois-Chollet/dp/1617294438 -o deleteme.html --verbose
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 162.219.225.118:443...
* TCP_NODELAY set
* Connected to www.amazon.com (162.219.225.118) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [4447 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=www.amazon.com
* start date: Apr 26 00:00:00 2021 GMT
* expire date: Apr 25 23:59:59 2022 GMT
* subjectAltName: host "www.amazon.com" matched cert's "www.amazon.com"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert Global CA G2
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55993f5fae10)
} [5 bytes data]
> GET /Deep-Learning-Python-Francois-Chollet/dp/1617294438 HTTP/2
> Host: www.amazon.com
> user-agent: curl/7.68.0
> accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [177 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
} [5 bytes data]
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
< HTTP/2 503
< server: Server
< date: Mon, 17 May 2021 01:37:18 GMT
< content-type: text/html
< strict-transport-security: max-age=47474747; includeSubDomains; preload
< x-amz-rid: 72RYS75V0EMTMX4RK38T
< vary: Content-Type,Accept-Encoding,X-Amzn-CDN-Cache,X-Amzn-AX-Treatment,User-Agent
< last-modified: Wed, 30 Sep 2020 23:54:00 GMT
< etag: "a6f-5b0909d09d600"
< accept-ranges: bytes
< content-length: 2671
<
{ [1139 bytes data]
100 2671 100 2671 0 0 3604 0 --:--:-- --:--:-- --:--:-- 3599
* Connection #0 to host www.amazon.com left intact
root@localhost:~#
And one thing, the onebox preview for Amazon is OK now after Jamie’s update code. The preview before updating, please refer to this image.
Good on the Amazon link. Are the udemy links working now or are you still getting a 403 on those? I still don’t have any problem Oneboxing those.
Thanks again.
All udemy hyperlinks failed on my site.
I just tried another forum based on discourse and found the same error code 403.
forum site: https://forum.quantifiedself.com/
testing 1: OK
Amazon https://www.amazon.com/Deep-Learning-Python-Francois-Chollet/dp/1617294438
testing 2: NG with error code 500. I guess this site doesn’t upgrade to the most update version yet.
Medium 15 Free Courses to Learn Python in 2021 | by javinpaul | The Startup | Medium
testing 3: NG, the same error code 403, which is exactly the same as mine.
udemy Kubernetes Certified Application Developer (CKAD) Training | Udemy
I test all discourse based sites I know, the result as below:
| onebox | site 1 | site 2 | site 3 | site 4 | site 5 | site 6 | site 7 | site 8 |
|---|---|---|---|---|---|---|---|---|
| Amazon | OK | OK | OK | OK | OK | OK | OK | OK |
| medium | OK | NG | OK | NG | NG | OK | NG | NG |
| udemy | NG | NG | OK | OK | OK | OK | NG | OK |
| location* | Tokyo | California | California | California | California | Paris | California | England |
| Ireland | Virginia | Chicago | ||||||
| Africa | Alaska |
*test with Check host
site 1. https://vip.studycamp.tw/
site 2. https://forum.quantifiedself.com/
site 3. https://discussions.udacity.com/
site 4. https://forum.uipath.com/
site 5. https://forums.envato.com/
site 6. https://rideabl.es/
site 7. https://blenderartists.org/
site 8. https://se23.life/
You are lucky, the only one that works is the amazon.com
With all other amazon sites the onebox preview does not work.
The sites you’re testing against may be blocking the curl user agent. The Oneboxing process will use the user agent from a real browser, depending on the URL you’re trying to Onebox. At the moment, the user agent it uses in such cases is:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15
Try running your curl tests with the -A flag. e.g.:
curl -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15" https://www.udemy.com/course/certified-kubernetes-application-developer/ -o deleteme.html --verbose
What is the result of that command on your end?
If it’s still failing:
Have you been running an unusually large number of Oneboxing requests against these sites? You may have been temporarily blocked by the sites in question and find things start working after waiting an appropriate period of time (this is up to the site operators, and there’s nothing we can do about it from our end unfortunately).
Where is your server located? Do you have access to a machine in a different location that you run tests from? Even testing from your personal machine instead of your production server may get you different results.
Thanks for advice.
The return code is still 403:
root@localhost:~# curl -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15" https://www.udemy.com/course/certified-kubernetes-application-developer/ -o deleteme.html --verbose
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 2606:4700::6810:4155:443...
* TCP_NODELAY set
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0* Connected to www.udemy.com (2606:4700::6810:4155) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [3726 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=California; L=San Francisco; O=Udemy, Inc.; CN=*.udemy.com
* start date: Sep 10 15:45:40 2020 GMT
* expire date: Oct 12 15:45:40 2021 GMT
* subjectAltName: host "www.udemy.com" matched cert's "*.udemy.com"
* issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign RSA OV SSL CA 2018
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x5591c76d8e10)
} [5 bytes data]
> GET /course/certified-kubernetes-application-developer/ HTTP/2
> Host: www.udemy.com
> user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15
> accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
< HTTP/2 403
< date: Mon, 17 May 2021 23:59:37 GMT
< content-type: text/plain; charset=UTF-8
< content-length: 16
< x-frame-options: SAMEORIGIN
< cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< expires: Thu, 01 Jan 1970 00:00:01 GMT
< cf-request-id: 0a1e5e6b920000952193363000000001
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< set-cookie: __cf_bm=3224e014d6b0b4eca1cf8a59c6c180928b2ffa0f-1621295977-1800-AU4UK/k3sl61kbMPYizcsl1TI7NAHkvE1mfjoKIPdrvGAJJeYUF8e72hQljXNwVTO+BcqfsAXxoiQDEy1cL+aVA=; path=/; expires=Tue, 18-May-21 00:29:37 GMT; domain=.udemy.com; HttpOnly; Secure; SameSite=None
< x-content-type-options: nosniff
< server: cloudflare
< cf-ray: 6510ccf28d0a9521-NRT
<
{ [16 bytes data]
100 16 100 16 0 0 10 0 0:00:01 0:00:01 --:--:-- 10
* Connection #0 to host www.udemy.com left intact
Yes, I did. But the clone site didn’t post any content more than 20 days, it has the same problem.
Besides, based on the testing table I mentioned yesterday, site 2 & site 7 have the same issue and I believe it’s first udemy related post on these sites.
My server site is located in Japan. I can move the clone site to other area like US for testing later today, and I will update again then.
Update:
Thanks again.
I’ve done a little more testing from my side.
I can successfully curl the udemy URL from my residential ISP (North America) and also from a machine hosted at a fancy datacenter (North America).
Attempting to do so from a popular hosting provider (North America) fails. Attempting to do so from a different popular hosting provider (Europe) also fails.
When it fails, the content body that is returned to me is:
error code: 1020
which is coming from:
This post from Cloudflare explains what their 1020 error means, and what you can do to try to resolve it:
https://community.cloudflare.com/t/community-tip-fixing-error-1020-access-denied/66439
Currently 2.7.0.beta8, if that helps at all.
The new build solve this issue, the onebox preview of udemy shows again. Not sure which part fix it.