How can I recover missing avatars?

Support
1

we have faced an issue similar to this bug :

https://meta.discourse.org/t/missing-user-profile-pictures/93844

is there any way to recover the missing avatars?

p.s.: I tried to use this guide and ran rake avatars:refresh but nothing happened.

4 Likes
2

this problem is like a virus and spreading towards more avatars! a strange observation was this:

in the firefox some avatars are shown, while they are missing in the chrome. this doesn’t mean, in firefox no avatar is missing. there are missing avatars in firefox as well!

firefox snapshot

image
image1339Ă—712 167 KB

chrome snapshot

image
image1366Ă—727 167 KB

has this bug really been fixed?

2 Likes
3

I’ve come across the same issue and I’m currently investigating it. In the case I’m investigating it’s connected with external storage.

@Pad_Pors Are you still seeing this issue? Also, did you, or do you have uploads stored in s3?

4 Likes
4

good to hear that an expert is investigating it :ok_hand:

yes, we are still facing this issue, and yes, we used to have uploads stored on s3, but not anymore.

1 Like
5

Same for us @angus, we do have S3. let us know if you need some logs or else. Many thanks

1 Like
6

In the case I’m looking at, the missing avatar images were moved to the tombstone folder in s3 after a command line upgrade and a manual upgrade of postgres from 10 to 12. I’m still not sure why exactly.

Finding the uploads

@Jeremie_Leroy @Pad_Pors If you want to see if the same is true in your case, the way I checked for the missing avatars in my s3 tombstone folder is as follows

  1. Obtained the sha1 (the 16 character string in the upload url) of an avatar upload I knew was broken after the migration. I did this by taking the upload id from the broken avatar upload url (it’s the first part of the file name, e.g. for 6254_2.png the id is 6254, and then using that id to look up the upload’s sha1 in recent database dump. If you’re uncomfortable with the command line you can visualise your data in a dump using a postgres GUI like Postico – a modern PostgreSQL client for the Mac.

  2. Did a search for the sha1 in the tomstone folder of the relevant s3 bucket using the AWS CLI.

    aws s3api list-objects --bucket <bucket_name> --query "Contents[?contains(Key, <sha1>)]" --prefix "tombstone"

    (you have to change <bucket_name> and <sha1>)

If it works, you will get a list of results that looks like this

{
  "Key": "tombstone/original/2X/d/d7b553ff276fca054c7090e859ef5339fd1f936e.jpg",
  "LastModified": "2020-05-16T11:45:03+00:00",
  "ETag": ## string of letters and numbers,
  "Size": 64580,
  "StorageClass": "STANDARD",
  "Owner": {
     "ID": ## string of letters and numbers
   }
}

Note, 5/16 was when I performed the upgrade. All incorrectly tombstoned avatars have timestamps from around then. I suspect the issue took a while to show up in production, and did so intermittently, due to caching.

Recovery

As far as I understand, UploadRecovery (and it’s associated rake task) is specific to post uploads, and doesn’t handle tombstoned avatars.

I’m currently looking adding (patching) a new method to UploadRecovery that uses recover_from_s3 to recover the avatar uploads.

If anyone knows of an easier way to recover incorrectly tombstoned avatars in s3, I’m all ears.

@tgxworld any ideas?

4 Likes
7

this is tricky for me, is there other people in our situation, so it will be worth to work on an update ? @sam @codinghorror

1 Like
8

thanks @angus for sharing the checking path, but we are not using AWS as well as S3 anymore, in fact if I’m not wrong the problem get started after we migrated from AWS.

I’d be happy if the problem is being fixed, but at the moment it’s much easier for me to ask those few users to re-upload their avatars, and wish that the problem do not spread into new users!

1 Like
9

I’ve addressed this successfully on the site I’m managing by using an adapted version of the same logic that recovers incorrectly tombstoned images in posts: https://github.com/discourse/discourse/blob/master/lib/upload_recovery.rb#L135.

@Jeremie_Leroy Part of the reason I posted those steps above is that there are various possible causes of missing avatars. To know if this is going to work for you, you first have to confirm that there are incorrectly tombstoned avatars, i.e. what the cause of the issue in your case is.

I also understand that undertaking that analysis is very hard if you’re not familiar with the technical aspects of Discourse. I’ve put the solution I used into a plugin which you can try on your own site. See instructions below.

@Pad_Pors If you’re ok just asking people to re-upload avatars I would just do that and not attempt this fix.

Instructions

Please note that this is a fix for missing avatars when:

  • Uploads are stored in s3
  • Avatars are missing because they are incorrectly tombstoned
  • The upload record of the avatar still exists

Do this at a time of low activity on your forum and perform a full backup first.

Install this plugin

https://github.com/angusmcleod/discourse-s3-avatar-recovery

The plugin adds recover_avatars to UploadRecovery, and uses an adapted version of the same restore method used by the main recover method used to recover missing uploads in posts.

This will keep a copy of the “recovered” avatars in the tombstone folder for redundancy. These copies will be removed by the purge_deleted_uploads background job which runs according to the period set by the purge_deleted_uploads_grace_period_days site setting.

Enter the rails console

To use the method you need to first ssh into your server, enter the docker instance and start a rails console:

./launcher enter app
rails c

Do a test run

To see which uploads the method is going to try and recover from tombstone, first do a test run:

UploadRecovery.new(dry_run: true, stop_on_error: false).recover_avatars

This will give you a list of usernames and s3 file links. These are the avatar files the method will attempt to move out of tombstone on a real run:

Example output

user1 tombstone/original/2X/b/bc84397936074854226f1c6e9016099b7fc0aca7.jpeg
user2 tombstone/original/2X/b/b8588e7e45804406f2cbe86f2362c2ccf7f13155.jpg
user3 tombstone/original/2X/8/81a4e3b70cdc35e8f61bca87d276d0253152804a.jpeg

Compare this with the users missing avatars on your site.

Do a real run

Change dry_run to true to do a real run

UploadRecovery.new(dry_run: false, stop_on_error: false).recover_avatars

Check

Once the job is complete check your live site in a new incognito window (to avoid caching issues). Once you’re done make sure you close down the active ssh conection to your server. You don’t want to accidentially enter the wrong thing in the open rails console.

3 Likes