Testing Restore - not working

[Crius]

Hey there, it’s your favourite bug finder, support bothering user.

I was testing the restore this afternoon and two times out of two, ended up with this:

ERROR:  could not extend file "base/16384/3294755": wrote only 4096 of 8192 bytes at block 43985
HINT:  Check free disk space.
EXCEPTION: psql failed: HINT:  Check free disk space.
/var/www/discourse/lib/backup_restore/database_restorer.rb:92:in `restore_dump'
/var/www/discourse/lib/backup_restore/database_restorer.rb:26:in `restore'
/var/www/discourse/lib/backup_restore/restorer.rb:51:in `run'
script/discourse:149:in `restore'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.2.2/lib/thor/command.rb:27:in `run'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.2.2/lib/thor/invocation.rb:127:in `invoke_command'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.2.2/lib/thor.rb:392:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.2.2/lib/thor/base.rb:485:in `start'
script/discourse:290:in `<top (required)>'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli/exec.rb:58:in `load'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli/exec.rb:58:in `kernel_load'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli/exec.rb:23:in `run'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli.rb:492:in `exec'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli.rb:34:in `dispatch'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli.rb:28:in `start'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/exe/bundle:45:in `block in <top (required)>'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/exe/bundle:33:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Trying to rollback...
Rolling back...
Cleaning stuff up...
Dropping functions from the discourse_functions schema...
Removing tmp '/var/www/discourse/tmp/restores/default/2023-08-19-145440' directory...
Unpausing sidekiq...
Marking restore as finished...
Notifying 'system' of the end of the restore...
Finished!
[FAILED]
Restore done.

Special mention to the

Finished!
[FAILED]
Restore done.

That made me kinda laugh
I’d say that this is really bad, on a scale to “not that bad” to “incredibly bad”.

I restored in a staging environment (a separated VMs we use for testing and messing about) after doing a clean:

• launcher stop app
• rm -rf /var/discourse/shared
• launcher rebuild app

The restore I tried is the last one created by the automated restore and backed up in an S3.

Then downloaded via the link provided via mail and restored using this guide:

Restore a backup from the command line

Here’s how to restore a Discourse backup from the command line, without ever booting the Discourse web UI. This is handy when you’re moving servers. Prerequisites Download the latest backup file from source Discourse instance. The destination discourse needs to be bootstrapped (run ./discourse-setup or copy your existing app.yml) Make sure that the destination Discourse instance is on the latest version. Update it if necessary. Transfer the backup SSH into the destination server, or otherwise…

The backup was 10GB. On the VM there were roughly 110GB free when I started the restore.

Edit: Discourse version 3.2.0.beta1-dev (fbf7b106cc)

[itsbhanusharma (Bhanu Sharma)]

@Crius

ERROR:  could not extend file "base/16384/3294755": wrote only 4096 of 8192 bytes at block 43985
HINT:  Check free disk space.
EXCEPTION: psql failed: HINT:  Check free disk space.

You’re out of disk space. Make sure you have at least 3x the size of your backup available on this server.

[Crius]

Nope. That’s an “hint” as in “it’s a common issue”.

As stated in the post straight above your reply:

The backup was 10GB. On the VM there were roughly 110GB free when I started the restore.

(edit with image in GB for ease of reading)

[Crius]

Investigating further, it seems that something is wrong in the data backupped, either the file is corrupted somehow or the cooking messed up something.

I’ll try and download another backup to see if the issue is the same across backup files to eliminate the possibility of a problem in downloading the file.

[Crius]

Weirder thing, loading up that topic and that message have literally zero issues for discourse. It loads up and render without problems.

I’ve manually intervened and edited that post. Launching a manual backup now to test again.