I’ve been running into issues trying to run a restore on our Staging Discourse instance. Staging is running v2.4.0.beta1 +36. Any idea where the breakdown might be or where to look? Thanks in advance!
Below is the end of the log output:
[2019-07-16 20:08:12] ALTER TABLE
[2019-07-16 20:08:12] ALTER TABLE
[2019-07-16 20:08:12] ALTER TABLE
[2019-07-16 20:08:12] ALTER TABLE
[2019-07-16 20:08:12] Migrating the database...
[2019-07-16 20:08:16] Reconnecting to the database...
[2019-07-16 20:08:16] Reloading site settings...
[2019-07-16 20:08:16] Disabling outgoing emails for non-staff users...
[2019-07-16 20:08:16] Clearing emoji cache...
[2019-07-16 20:08:16] Disabling readonly mode...
[2019-07-16 20:08:16] Clear theme cache
[2019-07-16 20:08:22] Extracting uploads...
[2019-07-16 20:08:40] Migrating uploads to S3...
[2019-07-16 20:08:46] Restore process was cancelled!
[2019-07-16 20:08:46] Trying to rollback...
[2019-07-16 20:08:46] Rolling back...
[2019-07-16 20:08:47] Cleaning stuff up...
[2019-07-16 20:08:47] Removing tmp '/var/www/discourse/tmp/restores/default/2019-07-16-200516' directory...
[2019-07-16 20:08:48] Unpausing sidekiq...
[2019-07-16 20:08:48] Marking restore as finished...
Below is the output after running discourse restore BACKUP_FILENAME from the command line. Any feedback is appreciated, thanks!
Disabling outgoing emails for non-staff users...
Clearing emoji cache...
Disabling readonly mode...
Clear theme cache
Extracting uploads...
Migrating uploads to S3...
Checking if default already migrated...
524 of 9474 uploads are not migrated to S3. S3 migration failed for db 'default'.
321 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'.
Looking for missing uploads on: default
Fixing missing uploads:
..........................................................................................................
116 post uploads are missing.
116 uploads are missing.
106 of 116 are old scheme uploads.
98 of 83342 posts are affected.
rake posts:missing_uploads identified 98 issues. S3 migration failed for db 'default'.
No posts require rebaking
Migrating uploads to S3 for 'default'...
Some uploads were not migrated to the new scheme. Please run these commands in the rails console
SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)
Restore process was cancelled!
Trying to rollback...
Rolling back...
Cleaning stuff up...
Removing tmp '/var/www/discourse/tmp/restores/default/2019-07-22-172918' directory...
Unpausing sidekiq...
Marking restore as finished...
Notifying 'system' of the end of the restore...
Finished!
[FAILED]
Restore done.
I’m pretty sure I’ve just been hit by this too, I believe this should be marked as a bug.
(if it’s different feel free to move to separate new topic)
Kinda important that restoring backups works.
Note that the reason for failure is not obvious when using the admin UI and clicking Restore next to the backup name, you get:
...
Migrating uploads to S3...
Restore process was cancelled!
...
Completing the restore via the command line gets you more detail:
discourse enter app
discourse restore example-net-2020-01-02-033557-v20191219112000.tar.gz
...
Reconnecting to the database...
Reloading site settings...
Disabling outgoing emails for non-staff users...
Clearing emoji cache...
Disabling readonly mode...
Clear theme cache
Extracting uploads...
Remapping uploads...
Remapping '//forum-example-net.s3.dualstack.eu-west-2.amazonaws.com/' to '/uploads/default/'
optimized_images=3
uploads=1
Migrating uploads to S3...
Checking if default already migrated...
6 of 12 uploads are not migrated to S3. S3 migration failed for db 'default'.
1 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'.
Looking for missing uploads on: default
0 post uploads are missing.
Please provide the following environment variables
- DISCOURSE_S3_BUCKET
- DISCOURSE_S3_REGION
and either
- DISCOURSE_S3_ACCESS_KEY_ID
- DISCOURSE_S3_SECRET_ACCESS_KEY
or
- DISCOURSE_S3_USE_IAM_PROFILE
Restore process was cancelled!
Trying to rollback...
Rolling back...
Cleaning stuff up...
Dropping function from the discourse_functions schema
Removing tmp '/var/www/discourse/tmp/restores/default/2020-01-06-222212' directory...
Unpausing sidekiq...
Marking restore as finished...
Notifying 'system' of the end of the restore...
Finished!
[FAILED]
Restore done.
I added a little debug code to uploads.rake just before Please provide the following environment variables to dump the environment variables:
puts "ENV: " + ENV.inspect
ENV contained no DISCOURSE_S3_* variables set.
Is there a meaningful reason this isn’t pulling this data from the settings?
No need to convert this into a bug. It’s on my plate and I’ve already spent a huge amount of time refactoring the restore process, adding lots of tests and making it more reliable. I’ll make a couple more tweaks to make restoring to S3 less likely to break and to output more information in the admin UI.
AFAIK the backup/restore has been reworked, but I just found out that this is still an issue.
An attempt on beta11 to restore a backup which had enable s3 uploads enabled still fails with
[2020-02-18 09:51:38] Restoring uploads, this may take a while...
[2020-02-18 09:51:38] EXCEPTION: Please provide the following environment variables:
- DISCOURSE_S3_BUCKET
- DISCOURSE_S3_REGION
and either
- DISCOURSE_S3_ACCESS_KEY_ID
- DISCOURSE_S3_SECRET_ACCESS_KEY
or
- DISCOURSE_S3_USE_IAM_PROFILE
[2020-02-18 09:51:38] /var/www/discourse/lib/file_store/to_s3_migration.rb:34:in `s3_options_from_env'
The S3 access credentials are present in the restored database so there is no need to require these to be in an environment variable as well.
Providing the environment variables yield a failure as well
Restoring uploads, this may take a while...
Checking if db8015 already migrated...
200 of 206 uploads are not migrated to S3. S3 migration failed for db 'db8015'.
5 posts are not remapped to new S3 upload URL. S3 migration failed for db 'db8015'.
No posts require rebaking
Migrating uploads to S3 for 'db8015'...
Uploading files to S3...
- Listing local files
=> 21 files
- Listing S3 files
. => 16 files
- Syncing files to S3
.....................
Updating the URLs in the database...
Removing old optimized images...
Flagging all posts containing lightboxes for rebake...
5 posts were flagged for a rebake
EXCEPTION: 183 of 206 uploads are not migrated to S3. S3 migration failed for db 'db8015'.
/var/www/discourse/lib/file_store/to_s3_migration.rb:127:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:74:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:350:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:61:in `migrate'
/var/www/discourse/lib/file_store/s3_store.rb:203:in `copy_from'
/var/www/discourse/lib/backup_restore/uploads_restorer.rb:48:in `restore_uploads'
/var/www/discourse/lib/backup_restore/uploads_restorer.rb:30:in `restore'
/var/www/discourse/lib/backup_restore/restorer.rb:58:in `run'
script/discourse:143:in `restore'
No idea why this is failing.
Most uploads were at s3, so the “200 of 206 uploads are not migrated to S3” and “183 of 206 uploads are not migrated to S3” are incorrect. The amount of 21 local files is correct, and there are approximately 200 uploads in s3 (could be 206 as well). I do not recognize any of the other numbers (183, 16).
No idea why the restore process is trying to move more uploads to s3 either? It should just take the local images from the backup and leave the uploads in s3 alone? Or am I overlooking something?
In the end I ended up hacking the enable_s3_uploads setting in the database dump to false, but that caused everything to be remapped back to local. And since there were a few images still local, it took a lot of work to figure out which ones needed to be mapped back to s3 and which one should not.
Mixing local uploads with uploads stored on S3 isn’t supported. Yes, I know, it’s possible to end up in such a state when someone switches from local to S3 and doesn’t migrate existing uploads to S3, but that’s another story…
Restoring a backup always includes remapping uploads if the system detects any changes which affect upload URLs. That includes switching between standalone and multisite, switching between local uploads and S3 as well as changes to the S3 and CDN settings. All uploads are restored at the correct location based on settings, either locally or on S3.
From time to time we run into backups were the automatic remapping and migration to S3 fails for various reasons. You can expect so see more improvements early in the 2.5 development cycle.