How to avoid potential issues when rebaking all posts?

[Canapin (Coin-coin le Canapin)]

Hi!

Rebaking all posts is sometimes a recommended action for various reasons.

After my own experience of a rebake experience that raised issues and seen/foresee other potential others, I’d like to know how they can be avoided.

• If I rebake my 2M posts, it will trigger too many requests to Youtube and my IP will be blacklisted, preventing Discourse from generating previews.

• If oneboxes (with titles, thumbnails, excerpts copied in Discourse database - cooked field) original URLs went to be broken or redirected, it seems that the oneboxes will break and that we will lose this information.

• I have old Facebook and Instagram oneboxes that display nicely. Nowadays, they don’t (they display in an iframe, see this commit).
  Additional info: Instagram embedding features and layout (note that the layout has changed a bit since then, and the iframes now are cropping content) and Facebook oneboxes not working on my forums (it seems even Discord can’t have “trusted” IP…? It’s a public link)

• I decided to drop the support of Facebook (and thus Instagram to my knowledge) on my forums for several reasons. If I rebake all my posts, I suppose that every link that was previously properly oneboxed will break. Is that right?

To circumvent certain of these issues, "Onebox Assistant", crawl for those previews reliably! could help, but not for all. For example, it couldn’t help me with Facebook previews.

---

What can be done to preserve existing oneboxed information during a rebake?

[Ed_S]

@MarcP

Is there a pattern perhaps we could use to rebake all posts except the ones with an embed/onebox?

Want to rebake all the posts matching a string or regular expression? Let’s do it! Access Your Site First connect to your Droplet via SSH, and enter the Docker container for your Discourse instances: cd /var/discourse ./launcher enter app Rebake All Posts Containing a Specific String Run this command: rake posts:rebake_match[“pattern”] Replace pattern with the string you want to match. Note that the match performed will be case insensitive. rake posts:rebake_match[“slight_smile”] The …

It feels to me that we need an enhancement to rebake, to be more careful

• rate limit on selected sites or perhaps on all sites
• inherit the original onebox if the refetch fails for any reason

In other words, I think we need a non-damaging rebake, at least as a selectable option.

(There will be some discourse communities who value an up to date 404, or who don’t value old posts at all, but there will also be communities which very much want to preserve old threads intact.)

[Ed_S]

Could this maybe be reworded and recategorised as a feature request?

[Stephen]

@Ed_S

inherit the original onebox if the refetch fails for any reason

Is there any value at all in automatically re-fetching the contents of a tweet? I would be fine with rebake skipping fetching one boxed content again unless a box were ticked.

[Ed_S]

That’s a good thought - it’s more robust, and less likely to fall foul of rate-limiting trouble. Leave all oneboxes alone unless specified.

[simon]

@Canapin

If I rebake my 2M posts, it will trigger too many requests to Youtube and my IP will be blacklisted , preventing Discourse from generating previews.

Thanks for posting that. It could impact something I’m working on. Do you have any idea how many requests it takes to trigger the limit?

Looking at the YouTube API docs, it seems that they allow up to 10k GET requests per 24 hours, but this is for requests made with an API key: YouTube Data API Overview  |  Google for Developers. It’s not clear to me how rate limits for unauthenticated API requests to pull in the video preview images are rate limited.

I don’t think there is a rate limit applied to videos being played via an iframe, but it would be good to confirm that: YouTube Player API Reference for iframe Embeds  |  YouTube IFrame Player API  |  Google for Developers.

[Canapin (Coin-coin le Canapin)]

@simon

Do you have any idea how many requests it takes to trigger the limit?

I don’t know.

I resolved the issue by using Onebox Assistant without any API. Just the plugin enabled. No idea how it resolved my problem. I don’t know either if it would work nowadays.

"Onebox Assistant", crawl for those previews reliably!

Not related, but I used the plugin when rebaking all my posts after a huge import (1,6 M posts), including posts containing Youtube videos. Youtube ended by blacklisting my IP after too much embedding. Installing and enabling Onebox Assistant resolved this issue, and I didn’t need it after the rebake. I disabled it but didn’t remove it (just in case, you know). It was perfect for this task.