Youtube embeds missing

[neounix (Dark Matter)]

@Iceman

root@cont-web-only:/var/www/discourse# time rake posts:rebake_match['youtube',string,100]
           116 / 76598 (0.2%)-

Hi @Iceman

You are not wrong. You are just trying to solve a problem, perhaps frustrated your YT embeds are not working…

When you run this:

# time rake posts:rebake_match['youtube',string,100]
           116 / 76598 (0.2%)- 

Because the delay is set to 100, it will only rebake one post per 100 seconds. You have nearly 77K posts to rebake (using this pattern), so that would take about 90 days to rebake. When I used 100, it was only an example, not a recommendation, sorry to confuse you if I did

Even if you change it to a 10 second delay, your rebake will take about 9 days.

What rate do you want to actually rebake your “YT cakes”?

[Overgrow (OG)]

@neounix

Edit that post and add this: icemanworkingonthisheadache

You can easily Run “Rebuild HTML” as admin by clicking button under each post.

[Iceman]

Thanks @neounix

Tried on a single post with a a YT link:

root@cont-web-only:/var/www/discourse# rake posts:rebake_match['icemanworkingonthisheadache',string,5]
        1 / 1 (100.0%)
1 posts done!

Then went to check on that post and it still shows just the link, no onebox. (Checked with no Cache, no Cookies, different PC even )

@Overgrow

You can easily Run “Rebuild HTML” as admin by clicking button under each post.

Thanks @Overgrow.

Also tried that both before and after the rebake command (the one above) on the same and another post, without success.

[neounix (Dark Matter)]

@Iceman

Then went to check on that post and it still shows just the link, no onebox.

Hey @iceman,

So, that 's a big clue; and now we know that even rebaking a single post with the rake task does not solve your issue (also confirmed that the HTML rebuild from the UI does not work as suggested by @Overgrow) , which is what you were trying to tell us before.

Thanks for the step-by-step troubleshooting.

Let me think about this for a while; and in the meantime, others might have some ideas now that the problem is becoming more clear.

---

Edit:

---

@Iceman

Post one of your (not resolving) YT links (maybe where you added the ‘icemanworkingonthisheadache’ text) here and lets confirm one box works here (and I will also try on one of my staging sites).

[Iceman]

Sure thing @neounix

It does happen with any YT video, but for the sake of consistency, two that I tried specifically for this:

Franz Kafka: Chronicler of Darkness

Bram Stoker: Resurrecting the Vampire

---

Edit: Yep, they do work here.

As a note, I can’t get them to show on the preview panel on my instalaltion.

[neounix (Dark Matter)]

@Iceman

It does happen with any YT video, but for the sake of consistency, two that I tried specifically for this:

But all other “not YT” one boxes are rendering as they should on your site @Iceman ?

[Iceman]

Hi @neounix

Yes, all other oneboxes work as expected: Discourse, Wordpress, SoundCloud, Facebook, Twitter, Vimeo, etc. Only YT shows as an URL.

[riking (Kane York)]

One thing to try is to try signing into Google and visit the YouTube site while proxied through your server and see if it gives you any “interesting” error messages.

[Overgrow (OG)]

@Iceman

As a note, I can’t get them to show on the preview panel on my instalaltion.

Well if you can’t get onebox working in the composer for newly added post there is no point in rebaking old posts.

You need to get onebox working for new posts first (at least get it displayed correctly in the preview panel).

and btw… HTML rebuild from the UI = rebake

[Iceman]

So, after a week of penitence, I made some progress but ultimately hit a wall.

• All the details referred above are still in play.

• After going above and beyond with debugging, turns out that the issue was the following: When I created the new Droplet I used the opportunity to update a few things I wasn’t able to do back then with the original one. One of those was IPv6. Turns out that Digital Ocean’s IPv6 Ranges are mostly banned by Google for some cosmic reason. So, word of advice to the future: If you are setting up a DO Discourse Installation, don’t set up IPv6, because you can’t remove that IPv6 later, you will have to change Droplet.

• After correcting that, everything worked as expected. However, due to normal growth (and some recent tactics by Alphabet) I’m back to square zero because now I get this error and I don’t know of any way to “stop” Discourse from processing oneboxes (I even tried adding the YT domain to the Blacklist, with no success). This is because according to that post I linked “it should go away after an hour”, but I can’t freeze my rebakes… or can I? Should I? The other options rely on changing stuff on the server’s side but that would be… adding more IPs? Forcing “X-Forwarded-For” on Onebox calls?

• Also, a bonus question: Why does Discourse “take the load” of calling YT? I mean, this error suggests that it is the Server’s IP that is being flagged as doing all the calls (yes, it is because of the onebox unfurl), but… why doesn’t that fall under the user loading the page? Performance?

[neounix (Dark Matter)]

@Iceman

and I don’t know of any way to “stop” Discourse from processing oneboxes (I even tried adding the YT domain to the Blacklist, with no success).

Hi @Iceman

As a kludge, I have tested this (not elegant) CSS (for you) and it seems to work to stop onebox (from processing); as a kind of “hit a fly with a hammer” approach, which you could turn on and off (add and subtract) until you come up with something better. Give it a try and see:

.onebox-body{
    display:none;
}

Hope this helps.

Note:

I tested this on some onebox links and the onebox(es) disappeared and the link remained; but did not test in detail, sorry.

[Overgrow (OG)]

Just stop current processing. Do you know what triggered rebake that is running? Is it background job? If it is then just lower rebake old posts count setting to zero.

You can then use rebake of selected youtube posts (matching with regexp) with delays…

Rebake all posts matching a pattern

Want to rebake all the posts matching a string or regular expression? Let’s do it! Access Your Site First connect to your Droplet via SSH, and enter the Docker container for your Discourse instances: cd /var/discourse ./launcher enter app Rebake All Posts Containing a Specific String Run this command: rake posts:rebake_match["pattern"] Replace pattern with the string you want to match. Note that the match performed will be case insensitive. rake posts:rebake_match[""] The ab…

[Iceman]

First of all, thank you both @neounix @Overgrow

Regarding this:

@neounix

As a kludge, I have tested this (not elegant) CSS (for you) and it seems to work to stop onebox (from processing); as a kind of “hit a fly with a hammer” approach, which you could turn on and off (add and subtract) until you come up with something better. Give it a try and see:

.onebox-body{
    display:none;
}

Hope this helps.

Will test it asap. As longs as it blocks it for me to test, more than happy

Regarding @Overgrow’s question:

@Overgrow

Just stop current processing. Do you know what triggered rebake that is running? Is it background job? If it is then just lower rebake old posts count setting to zero.

You can then use rebake of selected youtube posts (matching with regexp) with delays…

I recently learned (thanks to you guys) how to do custom rebakes and do some queries and dark magic. Howevver, can I ask how can I know/query this:

@Overgrow

Just stop current processing. Do you know what triggered rebake that is running? Is it background job? If it is then just lower rebake old posts count setting to zero.

I mean… is that on the Ruby console or by killing something on Sidekiq…?

[riking (Kane York)]

rebake old posts count controls how many posts flagged with the various rake tasks are processed every 15 minutes.

[Iceman]

Hi @riking,

Thanks for your input. Just a quick question… how can I change that setting? Rails Console?

[neounix (Dark Matter)]

@Iceman

Just a quick question… how can I change that setting? Rails Console

Yo @Iceman

It is a setting in the admin UI

[Iceman]

God do I feel dumb Thanks @neounix!

I can’t set it to zero, though. The UI says that the value should be between 1 and 2000000000. So I guess I will set it to 1 and combine it with your hack… that should allow me to see if the ban lifts.

(Because the other options are way heavier, being new IPs, LBs or directly forcing X-Forwarded-For on everything, and I don’t want to screw up how Discourse works )

Will update soon!

[neounix (Dark Matter)]

@Iceman

I can’t set it to zero, though. The UI says that the value should be between 1 and 2000000000

FYI … It you need to set it to zero you can more-than-likely do it with a direct DB UPDATE query (or Rails, which I cannot speak intelligently about)

[Iceman]

Welp, no luck. Left it with value 1 and the hack to not display anything and the Ban is still up 5h later. Worst part? Obviously there is no way to contact them, so I’m just guessing ways to solve it.

[Overgrow (OG)]

You need to be patient. It may take up to 2-3 days before the ban is lifted in my experience. Just make sure there are no further requests to that IP…

You can check if there are still rebake jobs to run. Hint is in the code (this line selects posts that need rebaking in the background limited by rebake old posts count setting):

https://github.com/discourse/discourse/blob/75b1298e997ce7dda904cc88b668a55eb13b457f/app/models/post.rb#L611

You are looking for following posts:

WHERE (((baked_version IS NULL) OR (baked_version < 2)) AND (deleted_at IS NULL))