Avatars take a long time to load after moving to S3-compatible R2

Hello,

I just migrated to R2 and everything went perfectly. All images have the S3 CDN URL link. However, I noticed an issue: avatars are taking a long time to load. On average, it takes about 3 to 4 seconds, whether I’m clicking on a user’s avatar or looking inside a post. Is this expected?

hmmm, i suspect it could be one of 3 different issues, but the most likely to me is on-the-fly resizing.

1. on-the-fly avatar resizing

when you migrated your uploads to R2, it moved the original images; however, discourse uses many different sizes of avatars (eg 45px for posts, 120px for the user card).

if those specific optimized sizes didn’t migrate perfectly, or haven’t been generated yet, dscourse has to generate them synchronously at the moment the user clicks them:

  1. discourse downloads the original avatar from R2 to the local server
  2. resizes it using imagemagick
  3. uploads the new size back to R2
  4. redirects the browser to the new URL, and this process takes 3 to 4 seconds

to verify: hard-refresh the page - if the avatar takes 3-4 seconds the first time, but loads instantly the second time, this is exactly what is happening.

to fix: this will naturally fix itself as users browse and the sizes are generated. but you can fix it instantly, by forcing the server to pre-generate all avatars in the background by ssh’ing into your server and running:

./launcher enter app
rake avatars:refresh

2. the 3 second IPv6 timeout

if the avatars are taking 3-4 seconds every time even after multiple refreshes, they are likely hitting a networking timeout.

cloudflare R2 api endpoints are dual-stack in that they use both ipv4 and ipv6. if your server droplet has an ipv6 address assigned to it, but the host’s ipv6 gateway isn’t routed properly, ruby’s internal connection to the R2 bucket will attempt ipv6 first, hang for 3 second (this is the default linux TCP timeout), fail and then instantly succeed using ipv4.

to verify: ssh into the server and run:

curl -I -6 https://cloudflare.com

if it hang for a few seconds and fails, the server’s ipv6 is broken, causing every internal S3 api check to suffer a 3-second delay.

to fix: you will need to either fix the ipv6 routing in in your host control panel or maybe even disable ipv6 on the droplet entirely

3. gravatar delays

if your site is configured to check for gravatar updates it may be pinging gravatar’s external servers before rendering the avatar. if the server has a slow outbound connection (also often related to DNS or ipv6) it will likely block the avatar render.

to verify: run this on your server
curl -I -6 https://gravatar.com
if it hangs for 3 seconds the ipv6 is broken (see above)

gravatar-related fix: in your discourse settings go to automatically download gravatars, turn it off temporarily, and see if that fixes it - i don’t think this is the problem but if it is, i you can leave the setting off, or fix the ipv6 routing as in 2 above, or maybe change the DNS resolver.

Thank you for your quick response. I think I had already tried the ‘rake avatars:refresh’ before, but I’m not absolutely sure.

What used to work for me to see the avatar open immediately was clicking on it a first time; on the second time, it would open instantly. But that’s probably due to caching. Also, I just tested your second tip and it returns a “HTTP/2 301”, with several other lines’. Same thing for tip 3. I’ll run avatars:refresh again in a few days, as I needed to restore a snapshot. Thanks again!

Gravatar

server: nginx
date: Mon, 22 Jun 2026 19:29:00 GMT
content-type: text/html; charset=utf-8
content-length: 0
content-language: en
expires: Wed, 11 Jan 1984 05:00:00 GMT
cache-control: no-cache, must-revalidate, max-age=0
x-redirect-by: Gravatar
location: https://en.gravatar.com/
alt-svc: h3=":443"; ma=86400
strict-transport-security: max-age=31536000; includeSubdomains; preload

CF

HTTP/2 301
date: Mon, 22 Jun 2026 19:27:00 GMT
content-type: text/html
content-length: 167
location: https://www.cloudflare.com/
cache-control: max-age=3600
expires: Mon, 22 Jun 2026 20:26:59 GMT
set-cookie: __cf_bm=eBP2aJ7Eg30nHPuvMMNxxKrgNtcNwKs0WDgnYyONeus-1782156420-1.0.1.1-sXpW27iuhGDF615cOfwNFybH4IMxgvZy3uA_3X_o..402T_3KSgT7CSymipL5RjdpGe3raWEqsVxQFFLPKRoDjfoT7B.0rqyDt.osbkOF98; path=/; expires=Mon, 22-Jun-26 19:57:00 GMT; domain=.cloudflare.com; HttpOnly; Secure; SameSite=None
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=QfYqSekEDPJHC2k%2BMjHN0cGjz172tmUWe2GSR8EgwNLh3TGjFYkQ0vwPxlzY1NcBcKFOMaAi4FlgjqjhETOOtHf%2BH9KdQSvqN3OME2Uh1i4nHIw%2Fy1qkvSpf4jxDchM7CaDW80tJkjBV4OqF"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
strict-transport-security: max-age=15780000; includeSubDomains
server: cloudflare
cf-ray: a0fda5d8ecd6b26d-LAX
alt-svc: h3=":443"; ma=86400

yea given your reply, i am almost certain it is issue #1 because the curl command results for cloudflare and gravatar appear to be as expected. give rake avatars:refresh a try whenever it is convenient for you and let me know if it works.

Hey Lilly, i’m still geting the same issue. Even after the rake avatars:refresh It’s even has the same issue with /latest. I tested it cleaning the cache from my browser and from cloudflare, but no success yet. Maybe i need to wait for some time? i’m testing it in a forum with 4500 users

don’t clear your browser or cloudflare caches anymore - when you run rake avatars:refresh for that many users, it doesn’t happen instantly. but rather it puts thousands of jobs into sidekiq to process in the background, which can take several hours depending on your server’s cpu. sorry about that, i should have mentioned sidekiq and that it an take awhile depending on how many users.

go to your-forum.com/sidekiq/queues and watch the queue. wait for it to completely empty out. once Sidekiq is finished, all the sizes should be in your R2 bucket permanently, and i think your avatar loading should return to normal speed.

Ok, I think something else is going on. I have nothing in my queues. But if I click on any user’s avatar, this shows up in ‘tail -f log/production.log’: Sent file /var/www/discourse/tmp/avatar_proxy/3689d91eb5e1013beef831c585b5e62edeeecbd6.jpeg (0.2ms)

oh wow ok. this is likely a smoking gun and points to a different issue.

avatar_proxy in the logs usually means discourse is refusing to serve the avatar directly from cloudflare R2 CDN. instead, discourse is aggressively intercepting the request and downloading the image from R2 to the local server’s /tmp folder, and then using ruby to serve the image to the browser. so i think this is completely bypassing the CDN and explains the 3-second delay - i suspect the server is manually fetching and loading the file on every single request :grimacing:

discourse uses the avatar_proxy in a few very specific scenarios and usually it is a privacy or security setting that forces the server to mask the external URL.

check these settings in admin - site settings:

find external system avatars url - if there is anything in that box (like /letter_avatar_proxy/v4/...), delete it so it is empty. that should stop discourse from proxying default letter avatars. also worth checking uploaded avatars allowed groups and make sure it says TL_0.

maybe doublecheck DISCOURSE_S3_CDN_URL to make sure it is correct without a trailing slash or typo?

remap custom avatars:
it seems likely that your database still contains the raw R2 bucket URLs instead of your new CDN URL; because they don’t match, your forum is likely proxying them for security reasons.

check in the rails console to see exactly what discourse is fighting with:

./launcher enter app
rails c

pick a username with a slow loading avatar

u = User.find_by_username("the_selected_username")
u.user_avatar.custom_upload.url

if the output returns a raw bucket URL your previous remaps didn’t catch everything (perhaps it might have missed a subdomain or a scheme).

to fix, ssh into your server, go back to your container again (not rails) (./launcher enter app), and run the remap tool (again lol) to swap the raw URL for your CDN URL:

discourse remap "https://<your-raw-cloudflare-url>.r2.cloudflarestorage.com" "https://cdn.your-domain.com"

then run it a second time using // instead of https:// just in case.

btw, just out of curiousity what host service are you using? i have the same general setup as you and i haven’t experienced this issue yet. so i’m also interested in your configuration and i want to try to reproduce it somehow.

the url i get shows the s3 cdn URL, and i can open the image in the browser.

I will pass the S3 for now, as i dont really need it at this time.

and i am using Advinservers for a longtime now

thank you for your help, much appreciated

I’ve finished further testing and can confirm the issue is not related to R2. The same behavior (slow-loading avatars on posts and when clicking on a username) persists even with a properly configured AWS S3. It is also not a firewall issue, as “ufw status” confirms that the firewall is currently disabled. I plan to conduct more tests this weekend in a staging environment, where I can keep the forum running for longer periods and take it offline if necessary without any issues.

Do you have both site and s3 CDNs configured?

Yes, i’m using both. The bucket is connected with cloudfront for the s3 cdn url, and the same for discourse cdn url

For the r2 tests, i was not using discourse cdn url.

Except not on the one that doesn’t work?

And the R2 with no discourse CDN is where you’re having trouble?

I won’t pretend that I understand exactly what your issue is or just how the avatar images work, but I’d get the CDN wired up before you do more testing, and it could be that the issue is that you moved to a new bucket and the issue is that those images need to get regenerated.

Avatars here load from what looks like a Discourse cdn: https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/lilly/48/555832_2.png

Do the user profiles load faster after they’ve loaded the first time?

Again, I don’t promise that I understand and haven’t looked at the code, but a guess is that these are getting served by the Discourse CDN, and Discourse then counts on subsequent requests to pull from the CDN. I think this explains why it doesn’t work (or works slowly) on the R2 version that doesn’t have a Discourse CDN.

maybe i am not grasping what you mean here, but i have 2 sites running R2 object storage that don’t experience this issue. :woman_shrugging:t2:

Do they not have Discourse CDNs? If they don’t, I’m wrong. If they do, then it may be that not having a Discourse CDN (when you have an s3 bucket?) causes this issue.

But it seems weird that avatars would behave differently with S3 than without it.

Actually, I tested this with 4 different configurations:

  1. R2 without Discourse CDN

  2. R2 with Discourse CDN

  3. AWS S3 with Discourse CDN

  4. AWS S3 without Discourse CDN

In all cases, I used the S3 CDN URL: files.mydomain.com.

For the cases with the Discourse CDN, I used: cdn.mydomain.com.

The problem is that in every scenario, the avatar loading is always very slow.

If I open a topic, I see the avatars loading one by one. However, this only happens once. For example, if I go to admin/users, I only see the nicknames, and then the avatars start loading one by one.

If I click on a nickname, the user card opens and the avatar appears 3-4 seconds later. This also only happens once; if I click it again, the avatar appears immediately, likely due to caching.

PS: When performing each test, I restore the snapshot from when I wasn’t using S3/R2, delete the bucket, and start over.

i’m just going to guess that it has nothing to do with object storage configuration. i think your avatars are slow across all four of those setups because your bottleneck isn’t the storage provider; but rather it’s the network latency between your site droplet and your bucket, or your server’s CPU is struggling to resize the images on the fly. i don’t know anything about your server/CDN configuration and what distances are involved here. but i think once the edge cache is built, it shouldn’t matter what storage you use as long as you stick with one and let the cache build itself. however i’m just theorizing at this point because i have no other ideas. :woman_shrugging:t2: :grinning_cat_with_smiling_eyes:

I don’t know what to think anymore either. For R2, I was using the Western Europe (WEUR) region, and for S3, I was using eu-north-1. These are my VPS specs:

AMD Turin Processor (4 vCores) 8GB DDR5 ECC Memory 256GB NVMe SSD Storage 5TB Bandwidth (10 Gbit) Located in Los Angeles, CA

Maybe I should test with a region in the United States next time? i dont think i can do that with R2.

That’s what I would expect. It’s slow to generate all of the avatars. The rake tasks will make it happen, but they take a long time, especially if there are many users. The first time you access a user, the avatar gets generated and it takes a few seconds to run an external program that processes the image data into various sizes. After that, everything is fine. I think you just need to run the rake task and wait for it to generate all of the avatars in the various sizes?

yea i guess that is what i was getting at here but maybe wasn’t explaining properly:

but i think you said it was loading slow every time. :thinking:

i think every time you change your s3 configuration or clear your cache you are starting starting over. if you have lots of user avatars this will take a long time.