I’d love some advice on what to do here, but also have some ideas about how this might be better handled.
What might be happening
One theory is that our server may have been identified by YouTube as potentially farming music videos and we’re getting limited / blocked.
We’re a really unremarkable little forum in the UK with meagre traffic, but we have a couple of threads (actually one split in two due to size) of 10K + 2K posts of music videos. It’s a musical chain where the next poster simply posts a song related, often in some tangential way, to the previous post.
We have other threads with YouTube links, of course, but this one is particularly (~100%) dense with music.
Following a rebake at the weekend, I’m guessing that YouTube looked at the activity of the Oneboxer trying to grab headers for lots of music videos and their algorithm put us on the naughty step.
I subsequently, re-tried to bake posts and that has presumably confirmed YouTube’s suspicions that all we do from this IP address is attempt to download music videos.
Might be Related to Digital Ocean
So Googling 429 errors on YouTube, and ignoring all of the YouTube API results, eventually points to a group of similar sounding issues that all say that their IP addresses have been grey-listed/black-listed/banned as a result of being on virtual servers - Digital Ocean’s name was specifically mentioned.
The suggestion is that YouTube may ‘ban’ a series of IP addresses and that other, legitimate servers, are often pulled in as collateral damage.
Here is one such problem and potential solution.
https://support.google.com/youtube/thread/21697789?hl=en
and here is another from a developer at medium.com who works on the embed.ly API.
https://support.google.com/youtube/thread/21939228?hl=en
The suggested resolution in the first post was to disable ip6 on the droplet.
Does anyone know if that sounds likely, or if it would cause any issues?
I’ve held off doing this at the moment so I can collect any other data that might be needed to aid resolution.
What should happen?
Firstly, I understand the reaction here is likely to be that my experience is an edge-case and that no change is needed.
I can understand that. But if @marcozambi and I have found the same issue in a relatively short period of time then it suggests that it could be something that others will see. Perhaps YouTube has recently become more officious in its policing of embedding?
I’d ask you to remember that at the moment my forum is in complete limbo. I have tens of thousands of links that have failed to be Oneboxed, but I don’t know where they are and I can only think of a total rebake to find them all which will almost certainly get more negative attention from YouTube.
At the very least, Onebox cannot fail silently when it hits a rate-limit error. It has to bring it to the attention of admins and, where possible, pause the process that tripped the rate-limit issue.
Options for bringing this to the attention of the forum owner
I don’t think this is easy. At all.
I see Onebox as playing a number of distinctly different roles
a) it builds Oneboxes in real-time as posts are composed,
b) it responds to rebakes of old posts (with often already known links) in the background and headless
c) it is asked to handle masses of background job posts during a migration / rebake
If a rate-limit error were to occur during scenario a) then the author would recognise that the Onebox hadn’t expanded and may complain to the forum admins - it might be noticed. I guess it’s also possible to grab the single error during composition and push an error or message to admins?
In scenario b) the failure may happen during background processing of the post and therefore it would need to be either retried or if a retry count had been reached it may need to be failed and the forum admins notified.
In scenario c) the wider context of lots of failures happening at the same time would need to be known. At the moment I don’t believe there is any overarching control of the post rebake process such that 10,000 429 errors coming back from Onebox might be able to recognised as a bigger problem than sending 10,000 individual messages to admins.
In reality, a failure such as that would mean that all calls to Onebox for YouTube (might be other providers as well) video expansion would need to be put on hold until we’re no longer rate-limited.
Alternative approaches
Rate limit outgoing requests
I guess the prevention-is-better-than-the-cure axiom might dictate that a Discourse rate-limiter could be put in front of the requests so that Discourse would then hold requests back on some settings-configured basis.
This would play havoc with re-bakes and migrations.
Onebox Assistant
Perhaps we all need commercial relationships with organisations that cache and proxy embeds such as the one used by @merefield in his Onebox Assistant plugin?
Whitelist the 'bot
Could we perhaps find a way to whitelist the Discourse 'bot with YouTube? Probably too open to abuse.
YouTube API
Just like the Onebox does with Twitter, perhaps we could use the YouTube API if the rate limits are greater?