After Blood, Sweat and Tears, I think it’s solved.
However, I’m not particuarly “proud” of the solution but hey, it’s Google, they won’t talk nor explain anything to you, so… conclusions:
-
First of all, one important lesson: Don’t enable IPv6 on DigitalOcean if you are using Discourse, because their IPv6 range is blocked by YouTube.
-
After the IPv6 change was fixed, due to increasing traffic, regardless of host (changed a couple of times, what a journey), what happened after that was that YouTube was IP-Blocking my Discourse installation, due to the quantity of YouTube videos posted on the site and how Discourse loads them.
-
In order to check this block, you need to either use your server as proxy to one with a browser or just do a curl and search for this line: “Sorry for the interruption. We have been receiving a large volume of requests from your network.” (there is this topic for reference)
-
Thanks to the help from @riking @neounix and @Overgrow I did a series of commands (that you can read above) to try to either stop, limit, or change the rate at which we bake the YouTube embeds. For most sites that would be enough, but we had the increased drama of being migrated after I tried a couple of hosts, so all the previous posts needed baking. As a matter of fact, limiting it to 1 every hour kind of solved it at first. But I guess my community just really likes to share videos because that didn’t last long.
-
Obviously there is no feedback or help from YouTube here, except for a couple of threads on their forum with the error and all the comments saying “yeah I also have that issue” but no solutions there.
-
Given the circumstances, remembering that infomercial logic of “There has to be another way!” I opted for a “Rambo” approach: Purchased another IP address. Then added a cron that switches the outbound IP address every hour. Issue solved.
It is expected that, if the site keeps growing and people just keep sharing that YouTube love I may need to acquire a third IP. But hey, until I figure out a correct way of doing a “distributed discourse” in a K8S or something, it’s as good as it gets.
Not the most elegant of solutions, I know.
Once again, thanks for all the help (and mostly patience, because I know I’m very n00bish with all the Rails/Sidekiq/RubyConsole combo, but I’m trying to improve by reading the Discourse Code).
Thanks!