I can't get a good backup - it fails after after_create_hook

Bug
1

My installation hasn’t been backed up in a while. Even though it’s set to be automatic - I didn’t get any failures either…

Tonight I did a manual backup and it said it failed. This was the end of the log:

[2015-12-19 01:21:09] pg_dump: creating CONSTRAINT embeddable_hosts_pkey
[2015-12-19 01:21:09] Disabling readonly mode...
[2015-12-19 01:21:09] Finalizing backup...
[2015-12-19 01:21:09] Updating dump for more awesomeness...
[2015-12-19 01:21:14] Creating archive: e-liquid-recipes-forum-2015-12-19-012022.tar.gz
[2015-12-19 01:21:14] Making sure archive does not already exist...
[2015-12-19 01:21:14] Creating empty archive...
[2015-12-19 01:21:14] Archiving metadata...
[2015-12-19 01:21:15] Archiving data dump...
[2015-12-19 01:21:17] Archiving uploads...
[2015-12-19 01:22:15] Gzipping archive...
[2015-12-19 01:25:13] Executing the after_create_hook for the backup
[2015-12-19 01:25:13] EXCEPTION: No such file or directory - /var/www/discourse/public/backups/default/e-liquid-recipes-forum-2015-12-19-012022.tar.gz
[2015-12-19 01:25:13] /var/www/discourse/app/models/backup.rb:75:in `size'
/var/www/discourse/app/models/backup.rb:75:in `block in create_from_filename'
/var/www/discourse/app/models/backup.rb:72:in `tap'
/var/www/discourse/app/models/backup.rb:72:in `create_from_filename'
/var/www/discourse/lib/backup_restore/backuper.rb:266:in `after_create_hook'
/var/www/discourse/lib/backup_restore/backuper.rb:49:in `run'
/var/www/discourse/lib/backup_restore/backup_restore.rb:161:in `block in start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `fork'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:13:in `backup!'
/var/www/discourse/app/controllers/admin/backups_controller.rb:31:in `create'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/abstract_controller/base.rb:198:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_controller/metal/rendering.rb:10:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.4/lib/active_support/callbacks.rb:117:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.4/lib/active_support/callbacks.rb:117:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.4/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.4/lib/active_support/callbacks.rb:505:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.4/lib/active_support/callbacks.rb:505:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.4/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.4/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.4/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/abstract_controller/callbacks.rb:19:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_controller/metal/rescue.rb:29:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.4/lib/active_support/notifications.rb:164:in `block in instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.4/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.4/lib/active_support/notifications.rb:164:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/abstract_controller/base.rb:137:in `process'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionview-4.2.4/lib/action_view/rendering.rb:30:in `process'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.7/lib/mini_profiler/profiling_methods.rb:106:in `block in profile_method'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_controller/metal.rb:196:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_controller/metal.rb:237:in `block in action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_dispatch/routing/route_set.rb:76:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_dispatch/routing/route_set.rb:76:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_dispatch/routing/route_set.rb:45:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_dispatch/routing/mapper.rb:49:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_dispatch/journey/router.rb:43:in `block in serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_dispatch/journey/router.rb:30:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_dispatch/journey/router.rb:30:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_dispatch/routing/route_set.rb:821:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.2.2/lib/omniauth/builder.rb:59:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/conditionalget.rb:38:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
/var/www/discourse/lib/middleware/anonymous_cache.rb:129:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_dispatch/middleware/flash.rb:260:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:225:in `context'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:220:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_dispatch/middleware/cookies.rb:560:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/query_cache.rb:36:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.4/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.4/lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.4/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-1.0.1/lib/logster/middleware/reporter.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.4/lib/rails/rack/logger.rb:38:in `call_app'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.4/lib/rails/rack/logger.rb:22:in `call'
/var/www/discourse/config/initializers/100-quiet_logger.rb:10:in `call_with_quiet_assets'
/var/www/discourse/config/initializers/100-silence_logger.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.4/lib/action_dispatch/middleware/request_id.rb:21:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/methodoverride.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/runtime.rb:18:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/sendfile.rb:113:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.7/lib/mini_profiler/profiler.rb:276:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/message_bus-1.1.1/lib/message_bus/rack/middleware.rb:60:in `call'
/var/www/discourse/lib/middleware/request_tracker.rb:73:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.4/lib/rails/engine.rb:518:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.4/lib/rails/application.rb:165:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.4/lib/rails/railtie.rb:194:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.4/lib/rails/railtie.rb:194:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:580:in `process_client'
/var/www/discourse/lib/scheduler/defer.rb:85:in `process_client'
/var/www/discourse/lib/middleware/unicorn_oobgc.rb:95:in `process_client'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:674:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:529:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:140:in `start'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-4.9.0/bin/unicorn:126:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `<main>'
[2015-12-19 01:25:13] Notifying 'daath' of the end of the backup...
[2015-12-19 01:25:14] Removing old backups...
[2015-12-19 01:25:14] Cleaning stuff up...
[2015-12-19 01:25:14] Removing '.tar' leftovers...
[2015-12-19 01:25:14] Removing tmp '/var/www/discourse/tmp/backups/default/2015-12-19-012022' directory...
[2015-12-19 01:25:14] Unpausing sidekiq...
[2015-12-19 01:25:14] Marking backup as finished...
[2015-12-19 01:25:14] Finished!
1 Like
2

Welcome to the club!

A question which is pending … is, or was, your host low on disk space at some point?

https://meta.discourse.org/t/automatic-daily-backups-do-not-work/36809/

https://meta.discourse.org/t/backup-issue-undefined-method-every-for-jobs-createbackup-class/36818/

3

this looks like a different error though …

4

Yes, it was low at some point… Deleted some backups and cleaned up a bit - can’t remember if it ran OK after that?!.. Might not have actually…

5

I just noticed it says

The path seems wrong. Shouldn’t it be

/var/discourse/shared/standalone/backups/default

?

6

Yes, there’s no clear point what’s going on here… the instance I mention here has never been low on space, and the errors in /logs don’t look like any other samples here.

Also, two other discourse instances I manage have no problems.

7

When you enter the running container, can you cd into /var/www/discourse/public/backups/default/?

8

Ah, yes I can do that:

/var/discourse$ sudo ./launcher ssh app
lastlog_openseek: Couldn’t stat /var/log/lastlog: No such file or directory
lastlog_openseek: Couldn’t stat /var/log/lastlog: No such file or directory

Welcome to Discourse Docker

Use: rails, rake or discourse to execute commands in production

root@forum:~# cd /var/www/discourse/public/backups/default/
root@forum:/var/www/discourse/public/backups/default# ll
total 8
drwxr-xr-x 2 discourse www-data 4096 Dec 20 14:43 ./
drwxr-xr-x 3 discourse www-data 4096 Jan 29 2015 …/
root@forum:/var/www/discourse/public/backups/default#

1 Like
9

I have 9.7GB free space - I just monitored free space as I started a backup - could it be that it exhausted the free space and failed silently? I wan’t paying good enough attention, but at some point free space was at 370MB…

10

Do you have enable_s3_backups enabled? If so, can you disable it and try backing up again?

11

I just tried - Same result and output :confused:

12

My backups are still failing :frowning:

13

Is your Discourse version the latest?

14

I always keep it updated…

15

NOTE: If I make a backup without files, it completes and uploads to my S3 bucket.

Now this is fine, but I would like to keep a backup of uploaded images/files as well - I could live without that, if I could use S3 to keep files/uploads directly, but I don’t know how to set it up properly:

16

Is the linked post relevant? Just wondering. :slight_smile:

I don’t think Discourse sends the backup as a stream, by that I mean, as normal backup systems do, the backup is first created on the disk, uploaded to S3, and then deleted from the disk(?).

The fact it works when using “without files” is a strong indicator that it’s running out of space during the backup.

You should run df -h on the droplet, perhaps on a timer, over and over again when running a backup. You should see the disk become full, then suddenly have free space (I don’t think Discourse keeps a backup that has failed, for obvious reasons).

I hope this helps!

17

As I wrote - I would be OK without a backup of the files if I could use S3 storage for storing the files - I’m just linking to a thread where I explain how I can’t get it to work properly.

That might indeed be the reason, though I never observed that it ran out of space. But a looong time ago I tried this and it went to about 300 MB free space and then stopped…