Upgrade broke images

AI-generated summary

Users report that upgrading to Discourse 2.5.0.beta6 caused images to break, returning 403 Forbidden errors from S3. kkowalczyk notes that S3 returns 403 instead of 404 for missing files, suggesting images may have been deleted. Attempts to recover images using rake uploads:recover_from_tombstone and rake posts:missing_uploads were unsuccessful. The rake uploads:missing command fails with a PostgreSQL syntax error. SweaterVest confirms the same issue. masterakay also reports the problem and references a related thread. The issue appears widespread, with multiple users unable to restore missing images despite trying various recovery methods.

I’ve updated to 2.5.0.beta6 (via web interface in /admin) and now some of the images are missing because s3 returns 403.

Here’s an example page: https://forum.sumatrapdfreader.org/t/outlook-folders-previewer-change-set-zoom-width/2506/5

Here’s what shows in JavaScript console:

ec89198b693dd221d603373294f36ea14c5c25f1.png:1 GET https://discourse-sumatra-backup.s3.dualstack.us-east-1.amazonaws.com/original/2X/e/ec89198b693dd221d603373294f36ea14c5c25f1.png 403 (Forbidden)
Image (async)
(anonymous) @ _application-f4f68c0db9c527ecb2ed2efb83107d37ca9e6e73caaadc4ab4761618775063f1.js:1
(anonymous) @ _application-f4f68c0db9c527ecb2ed2efb83107d37ca9e6e73caaadc4ab4761618775063f1.js:1
(anonymous) @ _application-f4f68c0db9c527ecb2ed2efb83107d37ca9e6e73caaadc4ab4761618775063f1.js:1
e7cb414e4237ed797a603eeea49d7ed468ac2aee.png:1 GET https://discourse-sumatra-backup.s3.dualstack.us-east-1.amazonaws.com/original/2X/e/e7cb414e4237ed797a603eeea49d7ed468ac2aee.png 403 (Forbidden)
Image (async)
(anonymous) @ _application-f4f68c0db9c527ecb2ed2efb83107d37ca9e6e73caaadc4ab4761618775063f1.js:1
(anonymous) @ _application-f4f68c0db9c527ecb2ed2efb83107d37ca9e6e73caaadc4ab4761618775063f1.js:1
(anonymous) @ _application-f4f68c0db9c527ecb2ed2efb83107d37ca9e6e73caaadc4ab4761618775063f1.js:1
5:1 Error while trying to use the following icon from the Manifest: https://forum.sumatrapdfreader.org/manifest.webmanifest (Download error or resource isn't a valid image)

Unfortunately s3 is lying and seems to return 403 for non-existing files (instead of 404) so it’s more likely the images are no longer in s3 than having their permissions mangled.

I did what Old image uploads become broken images - #31 by qtpngo suggests i.e.:

cd /var/discourse
./launcher enter app
rake uploads:recover_from_tombstone

That didn’t fix it.

Any ideas how to fix it?

I have the same issue – 403 forbidden for images after upgrade to 2.5.0.beta6

enable s3 uploads = yes

@vinothkannans Any suggestions here?

To recap: all I did was upgrade Discourse via web interface and many images seem to have been deleted from S3 backup. Example page: No preview in Windows Explorer

Any suggestion on how to re-create / re-upload those images?

Things I’ve tried and didn’t fix it:

rake uploads:recover_from_tombstone
rake posts:missing_uploads

returns:

2 uploads are missing.
2 of 6867 posts are affected.

Clearly much more than 2 images are missing.

rake uploads:missing

Currently it seems to be completely broken:

default has uploads on S3!
validating without inventory is likely to take an enormous amount of time.
We recommend you run SKIP_EXTERNAL=1 rake uploads:missing to skip validating if on a multisite.
rake aborted!
PG::SyntaxError: ERROR:  syntax error at end of input
LINE 1: INSERT INTO verified_ids VALUES
                                        ^
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-2.0.2/lib/patches/db/pg.rb:56:in `exec'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-2.0.2/lib/patches/db/pg.rb:56:in `exec'
/var/www/discourse/lib/file_store/s3_store.rb:290:in `list_missing'
/var/www/discourse/lib/file_store/s3_store.rb:197:in `list_missing_uploads'
/var/www/discourse/lib/tasks/uploads.rake:396:in `list_missing_uploads'
/var/www/discourse/lib/tasks/uploads.rake:389:in `block (2 levels) in <main>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:68:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.3.0/lib/rails_multisite/connection_management.rb:78:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:374:in `block in <main>'
/var/www/discourse/lib/tasks/uploads.rake:400:in `block in <main>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:missing_files
(See full trace by running task with --trace)

Additionally, I don’t know if this is relevant but I have this:

root@discourse-2gb-nyc2-01:/var/discourse/shared/standalone/uploads# ls default/original/1X/
1af3725386cf0eb1cd763f19df548c5f95646710.png  b87e5cba637562a00e005839efa6319ba0e8c405.png
29727643e4a789cd3d7daf25fe010290aefedc25.png  e652ec24c6af582019d8224ae5b93dcccedd73b4.png
86122817b67f304240c8d690c63c5cf784193061.png  f8d4190982be2130c75caa6775726088bbc15496.png
root@discourse-2gb-nyc2-01:/var/discourse/shared/standalone/uploads# ls default/optimized/1X/
86122817b67f304240c8d690c63c5cf784193061_1_120x120.png  86122817b67f304240c8d690c63c5cf784193061_1_60x60.png
86122817b67f304240c8d690c63c5cf784193061_1_135x135.png  86122817b67f304240c8d690c63c5cf784193061_1_64x64.png
86122817b67f304240c8d690c63c5cf784193061_1_180x180.png  86122817b67f304240c8d690c63c5cf784193061_1_67x67.png
86122817b67f304240c8d690c63c5cf784193061_1_20x20.png    86122817b67f304240c8d690c63c5cf784193061_1_75x75.png
86122817b67f304240c8d690c63c5cf784193061_1_240x240.png  86122817b67f304240c8d690c63c5cf784193061_1_90x90.png
86122817b67f304240c8d690c63c5cf784193061_1_25x25.png    86122817b67f304240c8d690c63c5cf784193061_1_96x96.png
86122817b67f304240c8d690c63c5cf784193061_1_30x30.png    b87e5cba637562a00e005839efa6319ba0e8c405_1_40x40.png
86122817b67f304240c8d690c63c5cf784193061_1_32x32.png    b87e5cba637562a00e005839efa6319ba0e8c405_1_50x50.png
86122817b67f304240c8d690c63c5cf784193061_1_360x360.png  b87e5cba637562a00e005839efa6319ba0e8c405_1_90x90.png
86122817b67f304240c8d690c63c5cf784193061_1_37x37.png    e652ec24c6af582019d8224ae5b93dcccedd73b4_1_240x240.png
86122817b67f304240c8d690c63c5cf784193061_1_40x40.png    e652ec24c6af582019d8224ae5b93dcccedd73b4_1_50x50.png
86122817b67f304240c8d690c63c5cf784193061_1_45x45.png    e652ec24c6af582019d8224ae5b93dcccedd73b4_1_64x64.png
86122817b67f304240c8d690c63c5cf784193061_1_48x48.png    e652ec24c6af582019d8224ae5b93dcccedd73b4_1_90x90.png
86122817b67f304240c8d690c63c5cf784193061_1_50x50.png

So the image for https://discourse-sumatra-backup.s3.amazonaws.com/original/1X/1490d7c52a757954a9aad983f7d0994bb5f5392a.png is no longer in my local file system

Same issue. My Thread:

@kkowalczyk This is what we have tried, nothing so far.