Restore failed at "EXCEPTION: x of y uploads are not migrated to S3. S3 migration failed for db 'default'."

AI-generated summary

The user dltj describes a failed Discourse restore due to S3 upload migration issues, specifically an error stating that 465 of 751 uploads failed to migrate. The root cause was a failed Postgres 10-to-12 upgrade on an outdated server. To resolve it, dltj postponed the Postgres update in the app.yml file, disabled the enable_s3_uploads setting before creating a backup, and successfully restored the site on a new server. RBoy asks for clarification on the steps, particularly whether the setting was disabled via the Admin GUI or app.yml, and whether any broken links appeared after the restore. dltj confirms the change was made in app.yml, that the restore worked, and that no broken links were observed.

This has been sitting as a draft on meta.discourse.org as I worked through the issue on my Discourse server. I’ve been able to get around the problem described below, but that I couldn’t do what was effectively a disaster restore (a restore without the benefit of being able to get into the old, broken server) is concerning. (The root cause was a failed Postgres 10-to-12 upgrade, but I was able to postpone the update in the Discourse app.yml file long enough to disable the enable_s3_uploads site setting before creating the backup. I’ve now done that backup, uploaded it to the new server, restored it, and have the Discourse site running again. Original message:


I am reconstructing a Discourse install from backup stored on S3. The old installation was very old—first installed in June 2016. I started the restore from a fresh installation because of a failed Postgres 10-to-12 upgrade. Rather than fix that…since this box was running an no longer supported version of Ubuntu…I thought I would just take the latest backup and load it onto the new server. The restore fails, however. In the logs, the point at which I’m running into a problem is here:

[2020-11-23 22:27:32] Reloading translations...
[2020-11-23 22:27:32] Remapping uploads...
[2020-11-23 22:27:32] Restoring uploads, this may take a while...
[2020-11-23 22:28:38] EXCEPTION: 465 of 751 uploads are not migrated to S3. S3 migration failed for db 'default'.
[2020-11-23 22:28:38] /var/www/discourse/lib/file_store/to_s3_migration.rb:131:in `raise_or_log'
/var/www/discourse/lib/file_store/to_s3_migration.rb:78:in `migration_successful?'
/var/www/discourse/lib/file_store/to_s3_migration.rb:357:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:65:in `migrate'
/var/www/discourse/lib/file_store/s3_store.rb:238:in `copy_from'
/var/www/discourse/lib/backup_restore/uploads_restorer.rb:48:in `restore_uploads'
/var/www/discourse/lib/backup_restore/uploads_restorer.rb:30:in `restore'
/var/www/discourse/lib/backup_restore/restorer.rb:64:in `run'
/var/www/discourse/script/spawn_backup_restore.rb:23:in `restore'
/var/www/discourse/script/spawn_backup_restore.rb:36:in `block in <main>'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `fork'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `<main>'
[2020-11-23 22:28:38] Trying to rollback...

This sounds similar to Restore process cancelled at migrating uploads to S3 step - #5 by Jim.Morrison, but it didn’t offer any guidance on how to fix the issue.

The before-you-post message in the right pane of this compose window also suggested Failed to restore DB, but that didn’t have anything helpful either.

I’m facing the same issue here but I haven’t been able to resolve it:

Could you explain in more details what exactly you did here? Was this settings disabled in the Admin → Settings GUI page or was it on in app.yml? So after the backup, you were able to restore it and get everything back? Did you turn on this setting again after the backup? Thanks

This was editing the app.yml file, and yes I was able to get everything back. Unfortunately, I didn’t create any notes from my work 2 years ago, and the details have been lost with time. I didn’t re-enable “include uploads in backup”, but I think I will now — hopefully whatever issue I had before is fixed.

Did you find any broken links to attachments in the posts after the restore?

Not that I can recall. Good luck!