I setup a new host so I can have a ‘staging’ environment. I re-created the same discourse installation with what should be the same version. I’m running version: 2.8.2.
First comment, as of version 2.8.2 my back up size has dropped from 282MB to around 90MB. Not sure why but I’m going to just with some intelligence added in that I’m taking advantage of.
I downloaded the latest archive from my forum, and uploaded to local storage on the new staging environment.
The restore fails due to:
[2022-02-27 19:41:18] ALTER TABLE
[2022-02-27 19:41:18] ALTER TABLE
[2022-02-27 19:41:18] Migrating the database...
[2022-02-27 19:43:00]
[2022-02-27 19:43:00] Reconnecting to the database...
[2022-02-27 19:43:00] Reloading site settings...
[2022-02-27 19:43:00] Disabling outgoing emails for non-staff users...
[2022-02-27 19:43:02] Disabling readonly mode...
[2022-02-27 19:43:02] Clearing category cache...
[2022-02-27 19:43:02] Reloading translations...
[2022-02-27 19:43:02] Remapping uploads...
[2022-02-27 19:43:02] Remapping 'https://forum.geekbeacon.org' to 'https://forum-staging.geekbeacon.org'
[2022-02-27 19:43:08] Restoring uploads, this may take a while...
[2022-02-27 19:43:36] EXCEPTION: 8 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'.
[2022-02-27 19:43:36] /var/www/discourse/lib/file_store/to_s3_migration.rb:132:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:87:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:373:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:66:in `migrate'
/var/www/discourse/lib/file_store/s3_store.rb:317:in `copy_from'
/var/www/discourse/lib/backup_restore/uploads_restorer.rb:62:in `restore_uploads'
/var/www/discourse/lib/backup_restore/uploads_restorer.rb:44:in `restore'
/var/www/discourse/lib/backup_restore/restorer.rb:61:in `run'
/var/www/discourse/script/spawn_backup_restore.rb:23:in `restore'
/var/www/discourse/script/spawn_backup_restore.rb:36:in `block in <main>'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `fork'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `<main>'
[2022-02-27 19:43:36] Trying to rollback...
[2022-02-27 19:43:36] Rolling back...
[2022-02-27 19:43:36] Cleaning stuff up...
[2022-02-27 19:43:36] Dropping functions from the discourse_functions schema...
[2022-02-27 19:43:36] Removing tmp '/var/www/discourse/tmp/restores/default/2022-02-27-194051' directory...
[2022-02-27 19:43:36] Unpausing sidekiq...
[2022-02-27 19:43:36] Marking restore as finished...
[2022-02-27 19:43:36] Notifying 'csgeek' of the end of the restore...
This is your issue. Maybe use the same S3 bucket and use the same bucket? Maybe see that the hidden setting that’s called something like "download S3 in backup " is turned on. You’ll need to search or look on on the source for the setting name.
Or maybe you need to see that the rest of your uploads on the production site are on S3.
I have no S3 configured I’m guessing maybe the old server has some old data that didn’t get mapped to a valid S3 location.
I went back to the older server and I have 2 buckets configured, 1 for backups and 1 for media. I followed the guide and setup AWS S3 including the CDN.
When I run rake uploads:migrate_to_s3 which failed, I follwed that with a posts:rebake and that seems to have reduced the number of errors, but it still fails:
Please note that migrating to S3 is currently not reversible!
[CTRL+c] to cancel, [ENTER] to continue
Migrating uploads to S3 for 'default'...
Uploading files to S3...
- Listing local files
=> 208 files
- Listing S3 files
. => 978 files
- Syncing files to S3
................................................................................................................................................................................................................
Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
15 posts were flagged for a rebake
rake aborted!
FileStore::ToS3MigrationError: 1 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'.
/var/www/discourse/lib/file_store/to_s3_migration.rb:132:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:87:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:373:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:66:in `migrate'
/var/www/discourse/lib/tasks/uploads.rake:123:in `migrate_to_s3'
/var/www/discourse/lib/tasks/uploads.rake:102:in `block in migrate_to_s3_all_sites'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-4.0.0/lib/rails_multisite/connection_management.rb:80:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-4.0.0/lib/rails_multisite/connection_management.rb:90:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:100:in `migrate_to_s3_all_sites'
/var/www/discourse/lib/tasks/uploads.rake:96:in `block in <main>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => uploads:migrate_to_s3
(See full trace by running task with --trace)
Is there a way to run migrate_to_s3 in a verbose mode to identify the post that’s the culprit?
FileStore::ToS3MigrationError: 1 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'.
In that case the best thing to do is fix up the post(s) that have uploads in the wrong place. But I think that there’s a case in which that test can have a false positive. In that case, you can use a switch on the rake task the will have it pause so you can fix things up. I can’t find it quicky, and it’s not at Backup discourse from the command line where it should be. I’m in the middle of a task, so I can’t find it now.
That’ll pause the restore and you can muck with the database in another terminal to fix it. Or, I think, you can just stop it before it does that check.
Oh damn, I hadn’t thought about stopping the restore after the database and before the uploads.
I don’t need to move the uploads from S3 — can I just migrate the frontend and the database?
Can you tell me which option allows you to stop the rake on posts and identify the problematic ones? That would be greatly appreciated.