Restore Failure - S3 (compatible) backup

markersocial · July 16, 2020, 2:16pm

So I’ve been trying to restore a backup, but getting errors. Looks like it has to do with the s3 backups.

Important to note, this is with a S3 compatible service (Scaleway). However I’m unsure if this error is specific to the compatible service or not, setting it up worked very seamlessly and has been working well. If it is specific to Scaleway, then I’ll likely stop using their service as I understand that only AWS S3 is officially supported.

I used this guide to set up: Using Object Storage for Uploads (S3 Clones) so this is in the app.yml file.

  after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets

Used the same configuration for Scaleway too in the app.yml file (did not set it up in the admin settings as it seemed unnecessary): scaleway config

I’ve tried both from the admin area and command line on the current server, as well as configuring a new server (copied the app.yml over) and restoring via command line. Got the same errors.

[STARTED]
'system' has started the restore!
Marking restore as running...
Making sure /var/www/discourse/tmp/restores/default/2020-07-16-131434 exists...
Downloading archive to tmp directory...
#<Thread:0x000055c73a831df8@/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.66.0/lib/aws-sdk-s3/file_downloader.rb:116 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
	1: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.66.0/lib/aws-sdk-s3/file_downloader.rb:120:in `block (3 levels) in thread_batches'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.66.0/lib/aws-sdk-s3/file_downloader.rb:128:in `write': undefined method `split' for nil:NilClass (NoMethodError)
EXCEPTION: undefined method `split' for nil:NilClass
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.66.0/lib/aws-sdk-s3/file_downloader.rb:128:in `write'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.66.0/lib/aws-sdk-s3/file_downloader.rb:120:in `block (3 levels) in thread_batches'
Trying to rollback...
There was no need to rollback
Cleaning stuff up...
Removing tmp '/var/www/discourse/tmp/restores/default/2020-07-16-131434' directory...
Unpausing sidekiq...
Marking restore as finished...
Notifying 'system' of the end of the restore...
#<Thread:0x000055c73a831510@/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.66.0/lib/aws-sdk-s3/file_downloader.rb:116 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
	1: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.66.0/lib/aws-sdk-s3/file_downloader.rb:120:in `block (3 levels) in thread_batches'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.66.0/lib/aws-sdk-s3/file_downloader.rb:128:in `write': undefined method `split' for nil:NilClass (NoMethodError)
#<Thread:0x000055c73a8316c8@/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.66.0/lib/aws-sdk-s3/file_downloader.rb:116 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
	1: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.66.0/lib/aws-sdk-s3/file_downloader.rb:120:in `block (3 levels) in thread_batches'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.66.0/lib/aws-sdk-s3/file_downloader.rb:128:in `write': undefined method `split' for nil:NilClass (NoMethodError)
#<Thread:0x000055c73a8319e8@/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.66.0/lib/aws-sdk-s3/file_downloader.rb:116 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
	1: from /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.66.0/lib/aws-sdk-s3/file_downloader.rb:120:in `block (3 levels) in thread_batches'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.66.0/lib/aws-sdk-s3/file_downloader.rb:128:in `write': undefined method `split' for nil:NilClass (NoMethodError)
Finished!
[FAILED]
Restore done.

Maybe the error is just for downloading the backup?

Falco · July 16, 2020, 4:36pm

Sounds like it.

I would move backup location to local in the app.yml, rebuild, download the backup file manually and restore from console.

markersocial · July 16, 2020, 4:44pm

Ah, that makes sense. Thanks @Falco!

I’ll give it a shot in the morning and report back~

markersocial · July 17, 2020, 7:31am

Thanks @Falco, appreciate it! Worked completely smoothly

I migrated to a new server, seems like it would have been fine staying on the same server as well.

In case someone else wants to reference this later, the steps in more detail where (note that my s3 settings were just in the app.yml file as specified here, nothing customised in the admin settings):

On source site, if not a rollback, enable ‘disable emails’ (might not be necessary) and turn on read-only mode (remember to switch these back on the new instance once migration is complete). Create backup, once complete, consider stopping the old instance also (./launcher stop app). Regardless of if a rollback or not, update DNS A records to point to the new server IP. You can do these in a more elegant way or order to minimise downtime, downtime wasn’t a concern for my case (rollback, not launched forum).
Install Discourse on the new server and migrate everything custom in the app.yml over including s3 settings, have the Discourse versions be the same or close to the same.
Comment out these two records (other s3 settings in the app.yml can be left as is):
DISCOURSE_S3_BACKUP_BUCKET: BucketName
DISCOURSE_BACKUP_LOCATION: s3
Manually download desired backup from s3 or s3 compatible service.
Navigate to /var/discourse/shared/standalone/backups and create a new folder called ‘default’ if it doesn’t exist (it won’t if it’s a new install). Then, while in the backups directory run (this changes folder permissions to match what this folder would normally have if Discourse created a local backup - not sure if necessary):
chown -R 1000:www-data default
Upload backup into the backups/default folder using a SFTP client, don’t rename the backup file.
Rebuild app:
cd /var/discourse
./launcher rebuild app
Enter app, enable backups and restore (rename BackupFileName.tar.gz):

./launcher enter app
discourse enable_restore
discourse restore BackupFileName.tar.gz

Once complete, uncomment the two s3 backup file lines in the app.yml from step 2 and rebuild app.
You can delete the local backups/default folder and contained backup (/var/discourse/shared/standalone/backups).

Reference:

system · August 16, 2020, 7:31am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Using Scaleway s3-compatible object storage Support	27	3699	April 6, 2021
Help restoring - system hung at midnight Installation	26	2202	June 13, 2022
Backup Upload to S3 Fails on scaleway (multipart upload) Support	9	1215	March 23, 2021
Restore failed at "EXCEPTION: x of y uploads are not migrated to S3. S3 migration failed for db 'default'." Installation	4	987	June 10, 2022
Finding UI generated backup and restoring site Support	22	2402	April 10, 2021

Restore Failure - S3 (compatible) backup

Related topics