Migration to S3 error

I read around 10 diff topics discussing similar error topics on Meta here, but couldn’t find the solution. So in the end decided to start this conversation.

I created a new ‘test’ discourse site (2.7.0 beta8 374ff4c54c).
Then created my first post having just 1 image.
Then changed the ‘upload’ location to aws S3.
Then commanded

DISCOURSE_S3_BUCKET="bhtestb/uploads" DISCOURSE_S3_REGION="ap-south-1" DISCOURSE_S3_ACCESS_KEY_ID="MyKey" DISCOURSE_S3_SECRET_ACCESS_KEY="MySecret" DISCOURSE_S3_CDN_URL="https://bhtestb.s3.ap-south-1.amazonaws.com/uploads" rake uploads:migrate_to_s3

This mainly seems to give two errors:

  • “rake aborted” : Don’t know why it says ‘aborted’?
  • FileStore::ToS3MigrationError: 1 of 12 uploads are not migrated to S3. S3 migration failed for db ‘default’. : Why it says ‘migration failed’?

Full Output of the command is:

Migrating uploads to S3 for 'default'...
Uploading files to S3...
 - Listing local files
 => 5 files
 - Listing S3 files
. => 5 files
 - Syncing files to S3

Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
1 posts were flagged for a rebake
rake aborted!
FileStore::ToS3MigrationError: 1 of 12 uploads are not migrated to S3. S3 migration failed for db 'default'.
/var/www/discourse/lib/file_store/to_s3_migration.rb:131:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:78:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:357:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:65:in `migrate'
/var/www/discourse/lib/tasks/uploads.rake:123:in `migrate_to_s3'
/var/www/discourse/lib/tasks/uploads.rake:102:in `block in migrate_to_s3_all_sites'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-3.0.0/lib/rails_multisite/connection_management.rb:80:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-3.0.0/lib/rails_multisite/connection_management.rb:90:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:100:in `migrate_to_s3_all_sites'
/var/www/discourse/lib/tasks/uploads.rake:96:in `block in <main>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:migrate_to_s3
(See full trace by running task with --trace)
1 Like

Did you follow Using Object Storage for Uploads (S3 & Clones)

2 Likes

Yes. I used/am using Aws S3 upload location successfully for 2 years for my main website (this is a sandbox/trial website which I’m first experimenting so that I could do the same on my main website later on).

I followed that guide, told by you, exactly.

2 Likes

The thing that I always forget is to add the stanza that does the upload assets to S3.

2 Likes

I don’t think I missed any part/stanza of the S3 Policy/Procedure. There are 2 main parts/stanzas only (in addition to ‘Effect’): ‘Action’ and ‘Resource’.

Moreover, my 1st bucket was working ok, without me changing anything in policy or anything else. It just got some problem when I tried to change the bucket without enough knowledge at that point of time.

1 Like