Imgur images not rendering because server is rate limited

Development

AI-generated summary

The issue of Imgur images not rendering on a Discourse forum is caused by the server being rate limited by Imgur. winston-yallow reported that after importing 120k topics with images from Imgur, the images would initially render correctly but fail to load after reloading the page. pfaffman identified the issue as rate limiting by Imgur, which prevents the server from downloading the images.

winston-yallow explained that the server is making requests to Imgur to cache the images locally, and pfaffman suggested that this behavior can be turned off or worked around by using a proxy server or importing topics on a different machine and restoring the backup to production.

1

I have added a post via the API. It contains the following snippet:

![Here is an example](https://i.imgur.com/LaUXRKW.png)

As visible here, it is correctly rendered in the editor:

image
image932×323 41 KB

Initially after saving, it is correctly shown in the topic too. However, after reloading the website it is rendered like this:
image

What I already tried:

  • check that the raw content in the database is correct
  • from the web interface, select “Rebuild HTML”
  • from the server console, perform rake posts:rebake (no errors)
  • disabled browser cache to ensure I don’t see an old render
  • restarted the app via ./launcher restart app
  • restarted the whole server

I don’t know what else to do at this point. Any tips on how to debug this further?

This is a fresh standard install using docker. I am on version 3.2.0.beta4-dev

2

To clarify, does it still happen if you create the message manually?

Also, this is a random idea, but did you try inserting a new line above to see if the image will appear?

On my side, I tried using the API or doing it manually, and I could not reproduce it. :thinking:.

3

I’m not familiar with that format to add image links, but I wondered if the issue could be connected to the block hotlinked media setting?

4

Thanks for the idea, but this setting is not enabled on my instance.

5

Your server has been rate limited by imgur. You can read the image with your browser, but not download it from your server.

If you try to pull the image from your server with curl or wget, you’ll get an error.

6

Just tested this, and yes it does happen there too. So it does not seem to be related to the API (I’ll adjust the topic title).

No difference, this still happens. As for reproduction, I am able to reproduce this reliably using this as the full topic content:

Test
![some alt text](https://i.imgur.com/LaUXRKW.png)

image
image725×749 41.1 KB

7

Oof, I see. This makes sense since before I ran into this, I did import 120k topics. So that must have fetched a lot of images from imgur.
Thanks for pointing this out, I simply thought the client does all image requests!

I find it interesting that the server does these requests, I assume it is to cache them locally in case the original disappears? I will look for ways to spread the process out a bit, so that the 120k posts get build over a longer period of time to prevent the server from being blacklisted or rate limited. I mean for this server it’s not a problem, it’s just on integration. But once I do the import on prod I’d like to prevent this :sweat_smile:

Continued as a new question: How to spread out import over longer timer to prevent running into rate limits of external services

8

Right.

Unless you turn off the settings that enable this behavior.

So do the import on another machine and restore the backup to prod. A more complicated solution would be to contrive to have your server retrieve stuff via a proxy server.