Testing Restore - not working

Product Support Self-hosting

AI-generated summary

In a discussion titled ‘Testing Restore - not working’ under the Installation category with ‘backups’ tag, Crius reports that restoring a 10GB backup fails twice with a PostgreSQL error suggesting insufficient disk space — despite having 110GB free. The error logs end with a contradictory ‘[FAILED] Restore done’, which Crius finds amusing. itsbhanusharma initially suggests the server lacks disk space, recommending 3x the backup size, but Crius counters with evidence showing ample free space. Further investigation by Crius hints at possible corruption or processing errors in the backup file itself, and they plan to test with another backup. Interestingly, Crius notes the problematic post loads fine in Discourse, so they manually edit it and trigger a new manual backup to retest the restore process. The issue remains unresolved, pointing toward potential data integrity or restore script problems rather than storage limits.

1

Hey there, it’s your favourite bug finder, support bothering user.

I was testing the restore this afternoon and two times out of two, ended up with this:

ERROR:  could not extend file "base/16384/3294755": wrote only 4096 of 8192 bytes at block 43985
HINT:  Check free disk space.
EXCEPTION: psql failed: HINT:  Check free disk space.
/var/www/discourse/lib/backup_restore/database_restorer.rb:92:in `restore_dump'
/var/www/discourse/lib/backup_restore/database_restorer.rb:26:in `restore'
/var/www/discourse/lib/backup_restore/restorer.rb:51:in `run'
script/discourse:149:in `restore'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.2.2/lib/thor/command.rb:27:in `run'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.2.2/lib/thor/invocation.rb:127:in `invoke_command'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.2.2/lib/thor.rb:392:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.2.2/lib/thor/base.rb:485:in `start'
script/discourse:290:in `<top (required)>'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli/exec.rb:58:in `load'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli/exec.rb:58:in `kernel_load'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli/exec.rb:23:in `run'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli.rb:492:in `exec'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli.rb:34:in `dispatch'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli.rb:28:in `start'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/exe/bundle:45:in `block in <top (required)>'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.13/exe/bundle:33:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Trying to rollback...
Rolling back...
Cleaning stuff up...
Dropping functions from the discourse_functions schema...
Removing tmp '/var/www/discourse/tmp/restores/default/2023-08-19-145440' directory...
Unpausing sidekiq...
Marking restore as finished...
Notifying 'system' of the end of the restore...
Finished!
[FAILED]
Restore done.

Special mention to the

Finished!
[FAILED]
Restore done.

That made me kinda laugh :roll_eyes:
I’d say that this is really bad, on a scale to “not that bad” to “incredibly bad”.

I restored in a staging environment (a separated VMs we use for testing and messing about) after doing a clean:

  • launcher stop app
  • rm -rf /var/discourse/shared
  • launcher rebuild app

The restore I tried is the last one created by the automated restore and backed up in an S3.

Then downloaded via the link provided via mail and restored using this guide:

The backup was 10GB. On the VM there were roughly 110GB free when I started the restore.

Edit: Discourse version 3.2.0.beta1-dev (fbf7b106cc)

2

You’re out of disk space. Make sure you have at least 3x the size of your backup available on this server.

3

Nope. That’s an “hint” as in “it’s a common issue”.

As stated in the post straight above your reply:

The backup was 10GB. On the VM there were roughly 110GB free when I started the restore.

image
image555×159 4.38 KB

(edit with image in GB for ease of reading)

4

Investigating further, it seems that something is wrong in the data backupped, either the file is corrupted somehow or the cooking messed up something.

image
image1414×819 64.8 KB

I’ll try and download another backup to see if the issue is the same across backup files to eliminate the possibility of a problem in downloading the file.

5

Weirder thing, loading up that topic and that message have literally zero issues for discourse. It loads up and render without problems.

I’ve manually intervened and edited that post. Launching a manual backup now to test again.