Missing user profile backgrounds

support
1

Hi,

It seems we have the following issue on an instance running Discourse stable v2.2.0+4:

I noticed this morning after updating to 2.2 but I’m not sure the issue came after this upgrade. It appears we had this error log every 10 minutes for several days:

Job exception: undefined method `recover_user_profile_backgrounds' for #<UploadRecovery:0x00007f866f769ba0 @dry_run=false>
/var/www/discourse/app/jobs/onceoff/recover_user_profile_backgrounds.rb:22:in `execute_onceoff'
/var/www/discourse/app/jobs/onceoff.rb:21:in `execute'
/var/www/discourse/app/jobs/base.rb:137:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.4/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/app/jobs/base.rb:127:in `block in perform'
/var/www/discourse/app/jobs/base.rb:123:in `each'
/var/www/discourse/app/jobs/base.rb:123:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:187:in `execute_job'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:169:in `block (2 levels) in process'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/var/www/discourse/lib/sidekiq/pausable.rb:81:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:168:in `block in process'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:139:in `block (6 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_retry.rb:98:in `local'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:138:in `block (5 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq.rb:36:in `block in <module:Sidekiq>'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:134:in `block (4 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:199:in `stats'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:129:in `block (3 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_logger.rb:8:in `call'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:128:in `block (2 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_retry.rb:73:in `global'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:127:in `block in dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/logging.rb:48:in `with_context'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:126:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:167:in `process'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:85:in `process_one'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:73:in `run'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:16:in `watchdog'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:25:in `block in safe_thread'

After the update from 2.1 to 2.2, the error no longer happens but the profile backgrounds are still not showing.

What should be done to recover them?

2 Likes
2

Any information on this? Should the uploads be considered lost?

3

Alan is on vacation but I do remember something about this. Any advice @sam?

4

Looks to me like a once off is running on an old source tree.

The actual method looks like it is there:

https://github.com/discourse/discourse/blob/bdbf77dc38f93c5314458f213bbd67b601828089/lib/upload_recovery.rb#L51-L51

Can you try rebuilding your container to see if this goes away?

cd /var/discourse
./launcher rebuild app
3 Likes
5

As mentioned, the error for the missing method is no longer raised since the rebuild to update to 2.2. However, that did not change the issue regarding the missing profile background.

6

Just performed a rebuild this morning to v2.2.1+9 but it did not change anything. No error logs, just getting 403 errors on profile background images. The URL format is like this:

https://[...].s3.dualstack.eu-west-1.amazonaws.com/original/1X/[...].jpg
https://[...].s3.dualstack.eu-west-1.amazonaws.com/original/2X/c/[...].jpg
https://[...].s3.dualstack.eu-west-1.amazonaws.com/original/2X/2/[...].jpg

Should the links be remapped? @tgxworld any advice here?

It’s not a central feature in our community (not sure users have even noticed yet) but I would like to avoid telling them that their background are lost.

7

I remember dualstack being weird. You may wish to search for that.

8

The onceoff job was backported to stable in this commit:

https://github.com/discourse/discourse/commit/cf60ae32ea8c6b9a898cf64ea9d82970e26e38dd

However, the UploadRecovery.new.recover_user_profile_backgrounds method did not exist at this point, see this file:

https://github.com/discourse/discourse/blob/cf60ae32ea8c6b9a898cf64ea9d82970e26e38dd/lib/upload_recovery.rb

So I guess I should manually run the onceoff job now that 2.2.1 is installed. How can I do that and is it safe?

2 Likes
9

ssh into your server

cd /var/discourse
./launcher enter app
rails c
Jobs::RecoverUserProfileBackgrounds.new.execute_onceoff({})

Yes.

1 Like
10

Thanks! The command should be prefixed with Jobs:::

[3] pry(main)> Jobs::RecoverUserProfileBackgrounds.new.execute_onceoff({})
=> nil

However, this does not seem to help. I will keep digging.

1 Like
11

Actually, I can’t find the uploads on S3. I ran a recursive ls on the upload bucket, and grep the results using the SHA1 of selected missing backgrounds.

Not sure where to go from there. I used the command below to extract the list of users who set a background, and will probably tell them that their uploads are lost.

User.find(UserProfile.where("profile_background IS NOT NULL OR card_background IS NOT NULL").pluck(:user_id)).pluck(:username)

12

Sorry to bump this topic again. Any additional info about this? Otherwise, I can only conclude that data can be lost in upgrading Discourse.

For example, can these uploads be found elsewhere or under another SHA1?