Disk space and backup failure notifications

I’m seeing frequent backup failures without any notifications, messages or emails to either myself as an admin or the “system” user.

Both automatic and manual backups fail.

Here is the example log from an manually executed backup (small amount removed to fit post limits):

[2016-02-16 23:32:47] 'DeanMarkTaylor' has started the backup!
...
[2016-02-16 23:33:34] pg_dump: processing data for table "versions"
[2016-02-16 23:33:34] pg_dump: dumping contents of table versions
[2016-02-16 23:33:34] pg_dump: executing SEQUENCE SET versions_id_seq
[2016-02-16 23:33:34] pg_dump: processing data for table "warnings"
[2016-02-16 23:33:34] pg_dump: dumping contents of table warnings
[2016-02-16 23:33:34] pg_dump: executing SEQUENCE SET warnings_id_seq
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT actions_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT api_keys_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT application_requests_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT badge_groupings_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT badge_types_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT badges_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT blocked_emails_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT categories_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT categories_search_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT category_custom_fields_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT category_featured_topics_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT category_featured_users_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT category_groups_pkey
[2016-02-16 23:33:34] Disabling readonly mode...
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT category_users_pkey
[2016-02-16 23:33:34] Finalizing backup...
[2016-02-16 23:33:34] Updating dump for more awesomeness...
[2016-02-16 23:33:49] Creating archive: xxx-xx-2016-02-16-233247.tar.gz
[2016-02-16 23:33:49] Making sure archive does not already exist...
[2016-02-16 23:33:49] Creating empty archive...
[2016-02-16 23:33:49] Archiving metadata...
[2016-02-16 23:33:49] Archiving data dump...
[2016-02-16 23:33:58] Archiving uploads...
[2016-02-16 23:37:15] Gzipping archive...
[2016-02-16 23:45:26] Executing the after_create_hook for the backup
[2016-02-16 23:45:26] EXCEPTION: No such file or directory - /var/www/discourse/public/backups/default/xxx-xx-2016-02-16-233247.tar.gz
[2016-02-16 23:45:26] /var/www/discourse/app/models/backup.rb:75:in `size'
/var/www/discourse/app/models/backup.rb:75:in `block in create_from_filename'
/var/www/discourse/app/models/backup.rb:72:in `tap'
/var/www/discourse/app/models/backup.rb:72:in `create_from_filename'
/var/www/discourse/lib/backup_restore/backuper.rb:266:in `after_create_hook'
/var/www/discourse/lib/backup_restore/backuper.rb:49:in `run'
/var/www/discourse/lib/backup_restore/backup_restore.rb:161:in `block in start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `fork'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:13:in `backup!'
/var/www/discourse/app/controllers/admin/backups_controller.rb:31:in `create'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/abstract_controller/base.rb:198:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/rendering.rb:10:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:117:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:117:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/abstract_controller/callbacks.rb:19:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/rescue.rb:29:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `block in instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.5.1/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/abstract_controller/base.rb:137:in `process'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionview-4.2.5.1/lib/action_view/rendering.rb:30:in `process'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.8/lib/mini_profiler/profiling_methods.rb:106:in `block in profile_method'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_controller/metal.rb:196:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_controller/metal.rb:237:in `block in action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:74:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:74:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:43:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/mapper.rb:49:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/router.rb:43:in `block in serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/router.rb:30:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/router.rb:30:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:815:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/builder.rb:63:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/conditionalget.rb:38:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
/var/www/discourse/lib/middleware/anonymous_cache.rb:129:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/flash.rb:260:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:225:in `context'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:220:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/cookies.rb:560:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.5.1/lib/active_record/query_cache.rb:36:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-1.0.1/lib/logster/middleware/reporter.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.1/lib/rails/rack/logger.rb:38:in `call_app'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.1/lib/rails/rack/logger.rb:22:in `call'
/var/www/discourse/config/initializers/100-quiet_logger.rb:10:in `call_with_quiet_assets'
/var/www/discourse/config/initializers/100-silence_logger.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/request_id.rb:21:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/methodoverride.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/runtime.rb:18:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/sendfile.rb:113:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.8/lib/mini_profiler/profiler.rb:282:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/message_bus-2.0.0.beta.2/lib/message_bus/rack/middleware.rb:60:in `call'
/var/www/discourse/lib/middleware/request_tracker.rb:73:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.1/lib/rails/engine.rb:518:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.1/lib/rails/application.rb:165:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.1/lib/rails/railtie.rb:194:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.1/lib/rails/railtie.rb:194:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:562:in `process_client'
/var/www/discourse/lib/scheduler/defer.rb:85:in `process_client'
/var/www/discourse/lib/middleware/unicorn_oobgc.rb:95:in `process_client'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:658:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:132:in `start'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/bin/unicorn:126:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `<main>'
[2016-02-16 23:45:26] Notifying 'DeanMarkTaylor' of the end of the backup...
[2016-02-16 23:45:26] Removing old backups...
[2016-02-16 23:45:26] Cleaning stuff up...
[2016-02-16 23:45:26] Removing '.tar' leftovers...
[2016-02-16 23:45:27] Removing tmp '/var/www/discourse/tmp/backups/default/2016-02-16-233247' directory...
[2016-02-16 23:45:27] Unpausing sidekiq...
[2016-02-16 23:45:27] Marking backup as finished...
[2016-02-16 23:45:27] Finished!

The backups when created are created they are 8.6GB in size, so in this case it’s likely a space issue. However my main worry is that there are no notifications of failure.

root@forum:/var/discourse/shared/standalone/backups/default# df -h
Filesystem                 Size  Used Avail Use% Mounted on
udev                       2.0G  4.0K  2.0G   1% /dev
tmpfs                      396M  340K  396M   1% /run
/dev/disk/by-label/DOROOT   59G   38G   18G  68% /
none                       4.0K     0  4.0K   0% /sys/fs/cgroup
none                       5.0M     0  5.0M   0% /run/lock
none                       2.0G  1.1M  2.0G   1% /run/shm
none                       100M     0  100M   0% /run/user
none                        59G   38G   18G  68% /var/lib/docker/aufs/mnt/ff086f43e49a8dd91ab098f8a4db6c9b222b1c838a573d464ff0db7eee29c7e3
shm                         64M     0   64M   0% /var/lib/docker/containers/354e2015381ef26aaf4057f838969d54c96718c3c7491c32f01c3db2b4bef66a/shm
root@forum:/var/discourse/shared/standalone/backups/default# ls -l -h
total 8.6G
-rw-r--r-- 1 1000 www-data 8.6G Feb 14 22:38 xxx-xx-2016-02-15-033149.tar.gz

My current version (v1.5.0.beta10 +86):

Discourse 1.5.0.beta10 - https://github.com/discourse/discourse version 8e9a8472f4714a7e2e63e669aaa7b58c9c8cd0e8
1 лайк