Missing user profile backgrounds

Hi,

It seems we have the following issue on an instance running Discourse stable v2.2.0+4:

I noticed this morning after updating to 2.2 but I’m not sure the issue came after this upgrade. It appears we had this error log every 10 minutes for several days:

Job exception: undefined method `recover_user_profile_backgrounds' for #<UploadRecovery:0x00007f866f769ba0 @dry_run=false>
/var/www/discourse/app/jobs/onceoff/recover_user_profile_backgrounds.rb:22:in `execute_onceoff'
/var/www/discourse/app/jobs/onceoff.rb:21:in `execute'
/var/www/discourse/app/jobs/base.rb:137:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.4/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/app/jobs/base.rb:127:in `block in perform'
/var/www/discourse/app/jobs/base.rb:123:in `each'
/var/www/discourse/app/jobs/base.rb:123:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:187:in `execute_job'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:169:in `block (2 levels) in process'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/var/www/discourse/lib/sidekiq/pausable.rb:81:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:168:in `block in process'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:139:in `block (6 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_retry.rb:98:in `local'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:138:in `block (5 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq.rb:36:in `block in <module:Sidekiq>'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:134:in `block (4 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:199:in `stats'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:129:in `block (3 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_logger.rb:8:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:128:in `block (2 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_retry.rb:73:in `global'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:127:in `block in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/logging.rb:48:in `with_context'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:126:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:167:in `process'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:85:in `process_one'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:73:in `run'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:16:in `watchdog'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:25:in `block in safe_thread'

After the update from 2.1 to 2.2, the error no longer happens but the profile backgrounds are still not showing.

What should be done to recover them?

2 Likes

Any information on this? Should the uploads be considered lost?

Alan is on vacation but I do remember something about this. Any advice @sam?

Looks to me like a once off is running on an old source tree.

The actual method looks like it is there:

https://github.com/discourse/discourse/blob/bdbf77dc38f93c5314458f213bbd67b601828089/lib/upload_recovery.rb#L51-L51

Can you try rebuilding your container to see if this goes away?

cd /var/discourse
./launcher rebuild app
3 Likes

As mentioned, the error for the missing method is no longer raised since the rebuild to update to 2.2. However, that did not change the issue regarding the missing profile background.

Just performed a rebuild this morning to v2.2.1+9 but it did not change anything. No error logs, just getting 403 errors on profile background images. The URL format is like this:

https://[...].s3.dualstack.eu-west-1.amazonaws.com/original/1X/[...].jpg
https://[...].s3.dualstack.eu-west-1.amazonaws.com/original/2X/c/[...].jpg
https://[...].s3.dualstack.eu-west-1.amazonaws.com/original/2X/2/[...].jpg

Should the links be remapped? @tgxworld any advice here?

It’s not a central feature in our community (not sure users have even noticed yet) but I would like to avoid telling them that their background are lost.

I remember dualstack being weird. You may wish to search for that.

The onceoff job was backported to stable in this commit:

https://github.com/discourse/discourse/commit/cf60ae32ea8c6b9a898cf64ea9d82970e26e38dd

However, the UploadRecovery.new.recover_user_profile_backgrounds method did not exist at this point, see this file:

https://github.com/discourse/discourse/blob/cf60ae32ea8c6b9a898cf64ea9d82970e26e38dd/lib/upload_recovery.rb

So I guess I should manually run the onceoff job now that 2.2.1 is installed. How can I do that and is it safe?

2 Likes

ssh into your server

cd /var/discourse
./launcher enter app
rails c
Jobs::RecoverUserProfileBackgrounds.new.execute_onceoff({})

Yes.

1 Like

Thanks! The command should be prefixed with Jobs:::

[3] pry(main)> Jobs::RecoverUserProfileBackgrounds.new.execute_onceoff({})
=> nil

However, this does not seem to help. I will keep digging.

1 Like

Actually, I can’t find the uploads on S3. I ran a recursive ls on the upload bucket, and grep the results using the SHA1 of selected missing backgrounds.

Not sure where to go from there. I used the command below to extract the list of users who set a background, and will probably tell them that their uploads are lost.

User.find(UserProfile.where("profile_background IS NOT NULL OR card_background IS NOT NULL").pluck(:user_id)).pluck(:username)

Sorry to bump this topic again. Any additional info about this? Otherwise, I can only conclude that data can be lost in upgrading Discourse.

For example, can these uploads be found elsewhere or under another SHA1?