Avatar not loaded in a multi container infrastructure


#1

Hi,

We are using a multi container infrastructure

  • 3 front-ends in SGN/JPN/IND
  • RDS & Elastic cache REDIS in SG
  • We have read replicas deployed close to our front-ends
  • We use S3 in SG to store our pictures (no replication)

We use Route 53 latency based routing to find the best server for our users.

When I load our main page from Singapore it works fine. I can see all our avatars. If i do the same thing but using a VPN to Tokyo, I see what seems to be a default avatar.

image

Any idea what is happening?

Seb


Serve all S3 images from CDN
(Bhanu Sharma) #2

are all the front ends configured properly to load resources from the same S3 bucket?


#3

Our S3 bucket is public and all front-ends share the same db (so the same settings)


(Bhanu Sharma) #4

I’d start investigating by checking if all the front ends are actually loading resources from S3 first.


#5

It seems that discourse is proxying avatars? It tries to load them from

https://%domain%/user_avatar/%domain%/%username%/45/1427_1.png

I don’t see errors in the logs (nginx or discourse). All requests going to /user_avatars/* returns 200 OK

Here is my S3 configuration:


(Bhanu Sharma) #6

Assuming it is something like:
https://d11a6trkgmumsb.cloudfront.net/user_avatar/meta.discourse.org/itsbhanusharma/32/82149_1.png
It’d be interesting to see what is loaded at the Tokyo instance! is it the same url over there too that returns the generic invisible man avatar ?

Ps: are you using cloudfront for distribution or s3 bucket ??


#7

For now we use plain s3 bucket. we will enable cdn later once we go live.


#8

I guess the problem is related to our latency based routing. Would this impact the way discourse is serving avatar?


#9

For instance this link does not show you my actual image


(Bhanu Sharma) #10

if this is a non-essential test site, you may try to test if images uploaded locally are shown on all the instances first.

if it is something that is being used in production, maybe @zogstrip can help you with some rake magic. I’m not very experienced with rake.


#11

So i did some tests:
1/ In tokyo thumbnails are broken but i can click and see the images correctly. In SG, everything is working fine
2/ After enabling cloudfront for S3, images got their url changed to https://xxx.cloudfront… URLS used by Avatars remained the same. no change on this

When I inspect broken images, I can see that links are incorrect?
Thumbnails should be pointing to cloudfront as well and not to the local server?


(Bhanu Sharma) #12

Yes, if You have CDN enabled, everything should be loaded from the CDN endpoint and not Your local server. That’s what a CDN exists for.

I didn’t really understand this.

You can try rebaking posts on the affected node if it’s loading the right link but not rendering them correctly.


(Jay Pfaffman) #13

I’ve seen similar behavior on single container installs. I have not yet started working on why. I don’t think it has to do with multiple containers or S3.


(Bhanu Sharma) #14

I had this happen once after an update on a system that was distributed across multiple containers and used s3 for uploads etc. It was fixed on it’s own after rebuild so maybe it’s some sort of caching or rendering issue?


#15

Hi Guys,

Finally I managed to reproduce the problem on my single instance after moving to S3 + CDN. All my images moved to ‘Cloudfront’ BUT avatars are still loaded from

https://%domain%/user_avatar/%domain%/joan/120/1457_1.png

I tried to rebuild/rebake all posts but still get the same url for avatars.

Any ideas? Is there a rake command i need to run?

Thx
Seb


(Bhanu Sharma) #16

I’m trying to repro this. Will update my reply if I find something.

Update:
I’ve tried configuring S3 for uploads and cloudfront as distribution endpoint.
everything works as expected:
after migrate_to_s3 rake task, all the local media is uploaded to the bucket.
The cloudfront endpoint linked to s3 serves all the files.

Hence, it may be something with your install.
@pfaffman I want to help on this but I think my limited knowledge with discourse core ends here.


(Jay Pfaffman) #17

This is exactly what I’m seeing on a site that’s not multisite.


(Bhanu Sharma) #18

Try adding the cdn url to the app.yml and rebuilding.


(Jay Pfaffman) #19

That seems like the problem. It’s adding the hostname as if there were a CDN. Oh, but that seems fine, as the URLs sort of work, it’s just that all of them are the generic person placeholder.


(Bhanu Sharma) #20

On my forum, I did some digging and it turns out that even after s3 and cloudfront are configured, only large images are being served from there and the avatars are from the server itself!
If it’s some bug, I’m happy to help.
One such forum is located at: discourse.orng.co
Avatars are being loaded from my own server but they load just fine.