Problems with avatar uploads due to S3 changes

Support
1

Hi everyone,

Somehow I keep having issues with avatars. I’m on the last version (v2.2.0.beta3 +3).

This is something noticed a few weeks ago but I’m only having the time to debug now. At first, I thought it could be a problem with the customizations I did through my theme plugin, but I don’t think that’s the case.

Some images show up but some others don’t. Has anyone seem something like that?

Here’s an example of an image that should be showing up (both urls seem correct):
https://app.thewatercooler.io/user_avatar/app.thewatercooler.io/tammycravit/45/1089_1.png

And here’s another one that works:
https://app.thewatercooler.io/user_avatar/app.thewatercooler.io/robertwilliams/45/1456_1.png

Here are all my S3 settings:

screencapture-app-thewatercooler-io-admin-site_settings-category-all_results-2018-10-12-12_58_57
screencapture-app-thewatercooler-io-admin-site_settings-category-all_results-2018-10-12-12_58_571938×3738 807 KB

1 Like
2

I think the problem might be related to Gravatar. I’m checking the database and looks like the users where the images are not showing up are the ones that Discourse is trying to download from Gravatar (with no success).

[31] pry(main)> u.user_avatar.update_gravatar!
=> nil
[32] pry(main)> u.refresh_avatar
=> nil
[33] pry(main)> u.user_avatar
=> #<UserAvatar:0x000055e10932f038
 id: 306,
 user_id: 303,
 custom_upload_id: 655,
 gravatar_upload_id: 1282,
 last_gravatar_download_attempt: Fri, 12 Oct 2018 20:08:17 UTC +00:00,
 created_at: Wed, 01 Nov 2017 20:19:44 UTC +00:00,
 updated_at: Mon, 03 Sep 2018 07:30:36 UTC +00:00>
[34] pry(main)> u.small_avatar_url
=> "https://app.thewatercooler.io/user_avatar/app.thewatercooler.io/valedeoro/45/1282_1.png"
[35] pry(main)> u.refresh_avatar
=> nil
[36] pry(main)>

And I can confirm that the Gravatar file exists in the S3 bucket by checking the associated object and searching the listed file on Amazon:

[38] pry(main)> u.user_avatar.gravatar_upload

I’m also noticing a really long delay to render some of the images that display correctly.

3 Likes
3

It is already the weekend for him but @tgxworld can check when it is his Monday

4

Thank you Jeff! No rush at all. I’m running the rake tasks to refresh the avatars now, and will keep this post updated :slight_smile:

1 Like
5

Running the rake tasks had no results. I will try to debug more tomorrow.

2 Likes
6

Can you be sure to update to latest, I did notice some issues with one of our once off jobs did play up on meta https://github.com/discourse/discourse/commit/8d06731484f84ee680d12d355c3b82ddc1191140

3 Likes
8

Does Upload.find(1282) return an upload record?

9

Thanks for replying. Yes, it does and the file exists in the bucket too.

[1] pry(main)> Upload.find(1282)
=> #<Upload:0x00005627d96730e8
 id: 1282,
 user_id: 303,
 original_filename: "gravatar.png",
 filesize: 201626,
 width: 360,
 height: 360,
 url: "//thewatercooler.s3.dualstack.us-east-1.amazonaws.com/original/2X/b/bdbc5de045297d6279f41cafe5486799823c526b.png",
 created_at: Mon, 03 Sep 2018 07:30:35 UTC +00:00,
 updated_at: Mon, 03 Sep 2018 07:30:36 UTC +00:00,
 sha1: "bdbc5de045297d6279f41cafe5486799823c526b",
 origin: "https://www.gravatar.com/avatar/e1e727c941ed663e9da7123bf05921cc.png?s=360&d=404",
 retain_hours: nil,
 extension: "png",
 thumbnail_width: 360,
 thumbnail_height: 360>

@sam updating from Beta to Stable now and I’ll post back here in a bit. Should I run the rake tasks to refresh everything again?

10

Interesting, what about upload.optimized_images?

11

That returns an empty array:

[3] pry(main)> upload.optimized_images
=> []
[4] pry(main)>
12

Hmm that is the problem here. I’ll have a look at this tomorrow to see if I can reproduce with the given image.

1 Like
13

Thank you so much! I’m now updating to get Sam’s commit and will run the rake tasks to refresh again, and see what happens.

14

The problem seems to be widespread actually. Now I’m on tests-passed branch, and I ran both refresh and clean rake tasks, but no luck. The weird thing is that I’m also noticing the same issue with some users that have custom_uploads as well. The ones with custom uploads have optimized_images, but deleting them has no effect either.

I also noticed that the reference that discourse has for the file is not the same URL that I can get from the bucket itself. For example, this the link I get from S3 directly (using their UI):

https://s3-us-west-2.amazonaws.com/thewatercooler/original/2X/d/de05564186252ab863d84df88dc83bfa45b0dd70.jpeg

But this is what I get from the Discourse’s database:

//thewatercooler.s3.dualstack.us-east-1.amazonaws.com/original/2X/d/de05564186252ab863d84df88dc83bfa45b0dd70.jpeg

But that’s not the S3 endpoint I have set on my settings page (I have the default). I just changed my s3 endpoint to https://s3-us-west-2.amazonaws.com but no luck either.

If you think having some Gravatar images might help, here’s a list of the one that I noticed the issue:

https://www.gravatar.com/avatar/a65aa3101e9ae2cae094ff045527cf4f.png?s=360&amp;d=404
https://www.gravatar.com/avatar/b19d019cab25762612252799f44a2b96.png?s=360&d=404
https://www.gravatar.com/avatar/1fc4324f4e1fe24c6587f900ff795229.png?s=360&d=404
https://www.gravatar.com/avatar/186f9af07d20464bae0e5be5ca060a17.png?s=360&d=404
https://www.gravatar.com/avatar/bcf814a93c75a521821ceebb74e6df90.png?s=360&d=404
https://www.gravatar.com/avatar/1face0cac8f8d45e23699ec830c04954.png?s=360&d=404
https://www.gravatar.com/avatar/2822ffe52dff31cb708edd4469704d2f.png?s=360&d=404
https://www.gravatar.com/avatar/bcf814a93c75a521821ceebb74e6df90.png?s=360&d=404
https://www.gravatar.com/avatar/5d04a3641d34189806ac548cf24b282e.png?s=360&d=404
https://www.gravatar.com/avatar/600c032e2bcbe65a04676c33add4b8d2.png?s=360&d=404
https://www.gravatar.com/avatar/add7942f5530b8d9a3bb06f9aac1baf6.png?s=360&d=404
1 Like
15

Did you happen to reconfigure SiteSetting.s3_region recently? It looks like that site setting has been configured to be us-east-1 when it should be us-west-2.

https://thewatercooler.s3.dualstack.us-west-2.amazonaws.com/original/2X/d/de05564186252ab863d84df88dc83bfa45b0dd70.jpeg works while https://thewatercooler.s3.dualstack.us-east-1.amazonaws.com/original/2X/d/de05564186252ab863d84df88dc83bfa45b0dd70.jpeg doesn’t.

2 Likes
16

Yeah, I had to change after one of the last updates (I explained what happened here Avatar upload failing)

I’m gonna write a rake task to update all the images. I wasn’t sure if changing the region could have affected it since it always worked before the other thread.

I will post my gist with my script to update the images in a bit to maybe help others in the future.

Thanks a lot!

17

Does that mean the region was incorrectly set initially before it was corrected? You can remap the URLs with bundle exec script/discourse remap //thewatercooler.s3.dualstack.us-east-1.amazonaws.com //thewatercooler.s3.dualstack.us-west-2.amazonaws.com. Please take a backup before running that in case it doesn’t remap correctly.

18

Awesome, will run the script now. I don’t understand what happened. I never touched the bucket settings on Discourse and my bucket was always west. For some reason it worked before. What I’m noticing is that images stored under 1x don’t use the dualstack url format and all my old uploads are 1x. Don’t know if something changed in the code recently that could have affected that.

Here’s an example:

"//thewatercooler.s3.amazonaws.com/original/1X/03eb07a5e93b9e2cb80c63d3fd86a310fb18968c.png"
"//thewatercooler.s3.amazonaws.com/original/1X/073fe33269897de4d0d7973b7f43e0c61a355f4f.png"
"//thewatercooler.s3.amazonaws.com/original/1X/8a34f55c0451703d672181f46cc9c9c4e8b9b457.png"
"//thewatercooler.s3.amazonaws.com/original/1X/9ee647f24b1d3d32653facb9da5d56469d22b6f9.jpeg"
"//thewatercooler.s3.amazonaws.com/original/1X/6f96e830f8f68e1b3232b33934569707f5406b1b.jpg"
"//thewatercooler.s3.dualstack.us-east-1.amazonaws.com/original/2X/8/868a566f2f56afe1a3e6fd16997e5a021a535a8b.jpeg"
"//thewatercooler.s3.dualstack.us-east-1.amazonaws.com/original/2X/9/9f175016fdd217566c4803d45bc8f99d996fe9b9.jpeg"
"//thewatercooler.s3.dualstack.us-east-1.amazonaws.com/original/2X/c/c3cd9870fa16a4875583183b833462cc8b1d40ff.jpeg"
"//thewatercooler.s3.dualstack.us-east-1.amazonaws.com/original/2X/4/40ef5880701b79c2968a284840925e632b4a9ad2.jpeg"
"//thewatercooler.s3.dualstack.us-east-1.amazonaws.com/original/2X/c/c90759b41f603fd18b75a41221850d40f0245ea9.jpeg"

All the 1X urls work independent of the region but the 2X ones have the region on the url.

19

Does OnceoffLog.where(job_name: "CorrectMissingDualstackUrls") return anything for you?

20

I just ran Remap, but checking OnceOfflLog shows this:

[#<OnceoffLog:0x00005639fcad5e58 id: 25, job_name: "CorrectMissingDualstackUrls", created_at: Fri, 07 Sep 2018 18:05:52 UTC +00:00, updated_at: Fri, 07 Sep 2018 18:05:52 UTC +00:00>]
21

Running Remap and the rake tasks to refresh and clean the avatars solved the problem.

Thank you so much for all the help!!

4 Likes