Restore from backup fails due to S3 migration

Product Support

AI-generated summary

Restoring a Discourse instance from a backup fails due to S3 migration issues. Senthil reports this issue, and pfaffman suggests disabling S3 uploads during the restore process. However, this doesn’t work in Senthil’s case.

A workaround is to disable the enable_s3_uploads site setting before creating the backup, as suggested by gerhard. This approach works for Senthil.

If a backup was made with S3 uploads enabled, a manual fix is required. MuhYal successfully restored their instance by editing the dump.sql file, replacing all instances of t and true with f and false, respectively. gerhard recommends using a suitable editor, such as SciTE, to handle large files.

1

I tried to migrate my discourse instance to a new server, and restore via UI and CLI fail during restore operation. It has to do with S3 migration.

I also had these settings enabled in rails console

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)

The relevant log from the failure was:

Remapping '//thinnal.s3.dualstack.us-east-1.amazonaws.com/' to '/uploads/default/'
incoming_emails=2
optimized_images=3851
post_revisions=86
posts=618
stylesheet_cache=10
topic_links=107
topics=150
uploads=419
user_histories=20
Migrating uploads to S3...
Checking if default already migrated...
1971 of 1977 uploads are not migrated to S3. S3 migration failed for db 'default'.
266 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'.
Looking for missing uploads on: default

0 post uploads are missing.

No posts require rebaking
Migrating uploads to S3 for 'default'...
Some uploads were not migrated to the new scheme. Please run these commands in the rails console

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)
Restore process was cancelled!
Trying to rollback...
Rolling back...
Cleaning stuff up...
Dropping function from the discourse_functions schema
Removing tmp '/var/www/discourse/tmp/restores/default/2019-09-02-140750' directory...
Unpausing sidekiq...
Marking restore as finished...
Notifying 'system' of the end of the restore...
Finished!
[FAILED]
Restore done.
2

There is a similar report of you search. I think the solution was to turn off S3 uploads when you restore?

3

Thanks @pfaffman Let me try that.

Do you know of any command line (rails console) way to disable s3 upload?

4

The Uploads to S3 was disabled during the restore. It still failed.

5

That’s a known problem. As a workaround, you could temporarily disable the enable_s3_uploads site setting before creating the backup. You can re-enable on the new server after the restore.

6

Thank you!. Yes, I disabled upload to S3 before taking the backup, and then the restore was successful.

7

Hi,

What to do if a backup was made while it was enabled :slight_smile:

8

I solved;

I open dump.sql with EmEditor program and search enable_s3_uploads, S3 and all “t” to “f” all “true” to “false”.

Restore successful :slight_smile:

9

Well, then you’ll have to get your hands dirty. Sorry about that. I’m planning to start refactoring and improving backups and restores next week, but it can take a while until everything’s peachy again.

That’s great! I was going to suggest exactly those steps. :wink:

For anyone reading this in the future: If you need to edit the dump.sql (which might be quite large), I suggest Scintilla and SciTE as editor. It handles those large files without problems.