Hi, It seems we have the following issue on an instance running Discourse stable v2.2.0+4: [image] Missing header profile images on Meta Bug Has anyone else noticed that the header/background images on user profiles are missing here on Meta? S3 gives a 403 (forbidden) error. [image] I noticed this morning after updating to 2.2 but I’m not sure the issue came after this upgrade. It appears we had this error log every 10 minutes for several days: Job exception: undefined method `recover_user_profile_backgrounds' for # /var/www/discourse/app/jobs/onceoff/recover_user_profile_backgrounds.rb:22:in `execute_onceoff' /var/www/discourse/app/jobs/onceoff.rb:21:in `execute' /var/www/discourse/app/jobs/base.rb:137:in `block (2 levels) in perform' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.4/lib/rails_multisite/connection_management.rb:63:in `with_connection' /var/www/discourse/app/jobs/base.rb:127:in `block in perform' /var/www/discourse/app/jobs/base.rb:123:in `each' /var/www/discourse/app/jobs/base.rb:123:in `perform' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:187:in `execute_job' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:169:in `block (2 levels) in process' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:128:in `block in invoke' /var/www/discourse/lib/sidekiq/pausable.rb:81:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:133:in `invoke' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:168:in `block in process' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:139:in `block (6 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_retry.rb:98:in `local' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:138:in `block (5 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq.rb:36:in `block in ' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:134:in `block (4 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:199:in `stats' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:129:in `block (3 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_logger.rb:8:in `call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:128:in `block (2 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_retry.rb:73:in `global' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:127:in `block in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/logging.rb:48:in `with_context' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/logging.rb:42:in `with_job_hash_context' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:126:in `dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:167:in `process' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:85:in `process_one' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:73:in `run' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:16:in `watchdog' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:25:in `block in safe_thread' After the update from 2.1 to 2.2, the error no longer happens but the profile backgrounds are still not showing. What should be done to recover them?

Any information on this? Should the uploads be considered lost?

Looks to me like a once off is running on an old source tree. The actual method looks like it is there: lib/upload_recovery.rb bdbf77dc3 def recover_user_profile_backgrounds Can you try rebuilding your container to see if this goes away? cd /var/discourse ./launcher rebuild app

[image] gkln: After the update from 2.1 to 2.2, the error no longer happens but the profile backgrounds are still not showing. As mentioned, the error for the missing method is no longer raised since the rebuild to update to 2.2. However, that did not change the issue regarding the missing profile background.

Just performed a rebuild this morning to v2.2.1+9 but it did not change anything. No error logs, just getting 403 errors on profile background images. The URL format is like this: https://[...].s3.dualstack.eu-west-1.amazonaws.com/original/1X/[...].jpg https://[...].s3.dualstack.eu-west-1.amazonaws.com/original/2X/c/[...].jpg https://[...].s3.dualstack.eu-west-1.amazonaws.com/original/2X/2/[...].jpg Should the links be remapped? @tgxworld any advice here? It’s not a central feature in our community (not sure users have even noticed yet) but I would like to avoid telling them that their background are lost.

I remember dualstack being weird. You may wish to search for that.

The onceoff job was backported to stable in this commit: https://github.com/discourse/discourse/commit/cf60ae32ea8c6b9a898cf64ea9d82970e26e38dd However, the UploadRecovery.new.recover_user_profile_backgrounds method did not exist at this point, see this file: lib/upload_recovery.rb cf60ae32e class UploadRecovery def initialize(dry_run: false) @dry_run = dry_run end def recover(posts = Post) posts.where("raw LIKE '%upload:\/\/%'").find_each do |post| begin analyzer = PostAnalyzer.new(post.raw, post.topic_id) analyzer.cooked_stripped.css("img").each do |img| if dom_class = img["class"] if (Post.white_listed_image_classes & dom_class.split).count > 0 next end end if img["data-orig-src"] if @dry_run puts "#{post.full_url} #{img["data-orig-src"]}" This file has been truncated. show original So I guess I should manually run the onceoff job now that 2.2.1 is installed. How can I do that and is it safe?

[image] gkln: So I guess I should manually run the onceoff job […]. How can I do that? ssh into your server cd /var/discourse ./launcher enter app rails c Jobs::RecoverUserProfileBackgrounds.new.execute_onceoff({}) [image] gkln: is it safe? Yes.

Thanks! The command should be prefixed with Jobs::: [3] pry(main)> Jobs::RecoverUserProfileBackgrounds.new.execute_onceoff({}) => nil However, this does not seem to help. I will keep digging.

Actually, I can’t find the uploads on S3. I ran a recursive ls on the upload bucket, and grep the results using the SHA1 of selected missing backgrounds. Not sure where to go from there. I used the command below to extract the list of users who set a background, and will probably tell them that their uploads are lost. User.find(UserProfile.where("profile_background IS NOT NULL OR card_background IS NOT NULL").pluck(:user_id)).pluck(:username)

Missing user profile backgrounds

Product Support

codinghorror (Jeff Atwood) February 5, 2019, 10:09pm 3

Alan is on vacation but I do remember something about this. Any advice @sam?

Topic		Replies	Views
Missing header profile images on Meta Bug	3	1041	October 1, 2018
Old image uploads become broken images Product Support	63	9656	May 24, 2021
Images are disappearing off of s3! Bug	56	6164	September 20, 2018
How can I migrate files from the old upload scheme (default/XXX) to the new one (default/original/{1,2}X)? Product Support	17	3630	May 17, 2019
Broken Images and Their S3 URLs Product Support	32	5359	August 31, 2020

Missing user profile backgrounds

Related topics