Disk space and backup failure notifications

Bug
,
1

It appears that one or more of the following are true:

  1. There are no backup failed notifications
  2. low disk space notifications
  3. failure notifications are not working

I have recently experienced a disk space issue where Discourse was left in a Read-only state.
Although backups had failed for a few days before this was the first time it was left read-only.

I had received no notifications of backup errors or disk space issues at all.

In addition I didn’t get any notification again of the “download remote images” option being changed and it didn’t appear in the staff actions log.
https://meta.discourse.org/t/download-remote-images-setting-reset-during-upgrade/20357
I have PM’ed @zogstrip to reopen this one, because it doesn’t appear in any log I can’t tell when this option changed.

I would expect to get some notification when errors with backups occur.

I would not expect the system to be left read-only, although this time around this was the only warning sign there was an issue.

Here is a log you might wish to review:

[STARTED]
'system' has started the backup!
Marking backup as running...
Making sure '/var/www/discourse/tmp/backups/default/2015-01-18-030000' exists...
Making sure '/var/www/discourse/public/backups/default' exists...
Writing metadata to '/var/www/discourse/tmp/backups/default/2015-01-18-030000/meta.json'...
Enabling readonly mode...
Pausing sidekiq...
Waiting for sidekiq to finish running jobs...
Dumping the public schema of the database...
pg_dump: reading schemas
pg_dump: reading user-defined tables
pg_dump: reading extensions
pg_dump: reading user-defined functions
pg_dump: reading user-defined types
pg_dump: reading procedural languages
pg_dump: reading user-defined aggregate functions
pg_dump: reading user-defined operators
pg_dump: reading user-defined operator classes
pg_dump: reading user-defined operator families
pg_dump: reading user-defined text search parsers
pg_dump: reading user-defined text search templates
pg_dump: reading user-defined text search dictionaries
pg_dump: reading user-defined text search configurations
pg_dump: reading user-defined foreign-data wrappers
pg_dump: reading user-defined foreign servers
pg_dump: reading default privileges
pg_dump: reading user-defined collations
pg_dump: reading user-defined conversions
pg_dump: reading type casts
pg_dump: reading table inheritance information
pg_dump: reading event triggers
pg_dump: finding extension members
pg_dump: finding inheritance relationships
pg_dump: reading column info for interesting tables
pg_dump: finding the columns and types of table "schema_migrations"
pg_dump: finding the columns and types of table "topics"
pg_dump: finding default expressions of table "topics"
pg_dump: finding check constraints for table "topics"
pg_dump: finding the columns and types of table "posts"
pg_dump: finding default expressions of table "posts"
pg_dump: finding the columns and types of table "users"
pg_dump: finding default expressions of table "users"
pg_dump: finding the columns and types of table "versions"
pg_dump: finding default expressions of table "versions"
pg_dump: finding the columns and types of table "topic_views"
pg_dump: finding the columns and types of table "uploads"
pg_dump: finding default expressions of table "uploads"
pg_dump: finding the columns and types of table "topic_users"
pg_dump: finding default expressions of table "topic_users"
pg_dump: finding the columns and types of table "topic_links"
pg_dump: finding default expressions of table "topic_links"
pg_dump: finding the columns and types of table "post_timings"
pg_dump: finding the columns and types of table "message_bus"
pg_dump: finding default expressions of table "message_bus"
pg_dump: finding the columns and types of table "notifications"
pg_dump: finding default expressions of table "notifications"
pg_dump: finding the columns and types of table "incoming_links"
pg_dump: finding default expressions of table "incoming_links"
pg_dump: finding the columns and types of table "post_replies"
pg_dump: finding the columns and types of table "categories"
pg_dump: finding default expressions of table "categories"
pg_dump: finding the columns and types of table "category_featured_topics"
pg_dump: finding default expressions of table "category_featured_topics"
pg_dump: finding the columns and types of table "site_settings"
pg_dump: finding default expressions of table "site_settings"
pg_dump: finding the columns and types of table "user_open_ids"
pg_dump: finding default expressions of table "user_open_ids"
pg_dump: finding the columns and types of table "user_actions"
pg_dump: finding default expressions of table "user_actions"
pg_dump: finding the columns and types of table "post_actions"
pg_dump: finding default expressions of table "post_actions"
pg_dump: finding the columns and types of table "post_action_types"
pg_dump: finding default expressions of table "post_action_types"
pg_dump: finding the columns and types of table "topic_link_clicks"
pg_dump: finding default expressions of table "topic_link_clicks"
pg_dump: finding the columns and types of table "category_featured_users"
pg_dump: finding default expressions of table "category_featured_users"
pg_dump: finding the columns and types of table "twitter_user_infos"
pg_dump: finding default expressions of table "twitter_user_infos"
pg_dump: finding the columns and types of table "email_logs"
pg_dump: finding default expressions of table "email_logs"
pg_dump: finding the columns and types of table "topic_allowed_users"
pg_dump: finding default expressions of table "topic_allowed_users"
pg_dump: finding the columns and types of table "facebook_user_infos"
pg_dump: finding default expressions of table "facebook_user_infos"
pg_dump: finding the columns and types of table "invites"
pg_dump: finding default expressions of table "invites"
pg_dump: finding the columns and types of table "topic_invites"
pg_dump: finding default expressions of table "topic_invites"
pg_dump: finding the columns and types of table "user_visits"
pg_dump: finding default expressions of table "user_visits"
pg_dump: finding the columns and types of table "email_tokens"
pg_dump: finding default expressions of table "email_tokens"
pg_dump: finding the columns and types of table "drafts"
pg_dump: finding default expressions of table "drafts"
pg_dump: finding the columns and types of table "site_customizations"
pg_dump: finding default expressions of table "site_customizations"
pg_dump: finding the columns and types of table "draft_sequences"
pg_dump: finding default expressions of table "draft_sequences"
pg_dump: finding the columns and types of table "post_search_data"
pg_dump: finding the columns and types of table "user_search_data"
pg_dump: finding the columns and types of table "category_search_data"
pg_dump: finding the columns and types of table "github_user_infos"
pg_dump: finding default expressions of table "github_user_infos"
pg_dump: finding the columns and types of table "site_texts"
pg_dump: finding the columns and types of table "user_histories"
pg_dump: finding default expressions of table "user_histories"
pg_dump: finding the columns and types of table "groups"
pg_dump: finding default expressions of table "groups"
pg_dump: finding the columns and types of table "group_users"
pg_dump: finding default expressions of table "group_users"
pg_dump: finding the columns and types of table "category_groups"
pg_dump: finding default expressions of table "category_groups"
pg_dump: finding the columns and types of table "topic_allowed_groups"
pg_dump: finding default expressions of table "topic_allowed_groups"
pg_dump: finding the columns and types of table "post_uploads"
pg_dump: finding default expressions of table "post_uploads"
pg_dump: finding the columns and types of table "optimized_images"
pg_dump: finding default expressions of table "optimized_images"
pg_dump: finding the columns and types of table "screened_emails"
pg_dump: finding default expressions of table "screened_emails"
pg_dump: finding the columns and types of table "screened_urls"
pg_dump: finding default expressions of table "screened_urls"
pg_dump: finding the columns and types of table "oauth2_user_infos"
pg_dump: finding default expressions of table "oauth2_user_infos"
pg_dump: finding the columns and types of table "plugin_store_rows"
pg_dump: finding default expressions of table "plugin_store_rows"
pg_dump: finding the columns and types of table "user_stats"
pg_dump: finding default expressions of table "user_stats"
pg_dump: finding the columns and types of table "post_details"
pg_dump: finding default expressions of table "post_details"
pg_dump: finding the columns and types of table "screened_ip_addresses"
pg_dump: finding default expressions of table "screened_ip_addresses"
pg_dump: finding the columns and types of table "api_keys"
pg_dump: finding default expressions of table "api_keys"
pg_dump: finding the columns and types of table "post_revisions"
pg_dump: finding default expressions of table "post_revisions"
pg_dump: finding the columns and types of table "topic_embeds"
pg_dump: finding default expressions of table "topic_embeds"
pg_dump: finding the columns and types of table "top_topics"
pg_dump: finding default expressions of table "top_topics"
pg_dump: finding the columns and types of table "category_users"
pg_dump: finding default expressions of table "category_users"
pg_dump: finding the columns and types of table "single_sign_on_records"
pg_dump: finding default expressions of table "single_sign_on_records"
pg_dump: finding the columns and types of table "badge_types"
pg_dump: finding default expressions of table "badge_types"
pg_dump: finding the columns and types of table "badges"
pg_dump: finding default expressions of table "badges"
pg_dump: finding the columns and types of table "user_badges"
pg_dump: finding default expressions of table "user_badges"
pg_dump: finding the columns and types of table "color_schemes"
pg_dump: finding default expressions of table "color_schemes"
pg_dump: finding the columns and types of table "color_scheme_colors"
pg_dump: finding default expressions of table "color_scheme_colors"
pg_dump: finding the columns and types of table "user_custom_fields"
pg_dump: finding default expressions of table "user_custom_fields"
pg_dump: finding the columns and types of table "category_custom_fields"
pg_dump: finding default expressions of table "category_custom_fields"
pg_dump: finding the columns and types of table "group_custom_fields"
pg_dump: finding default expressions of table "group_custom_fields"
pg_dump: finding the columns and types of table "post_custom_fields"
pg_dump: finding default expressions of table "post_custom_fields"
pg_dump: finding the columns and types of table "topic_custom_fields"
pg_dump: finding default expressions of table "topic_custom_fields"
pg_dump: finding the columns and types of table "invited_groups"
pg_dump: finding default expressions of table "invited_groups"
pg_dump: finding the columns and types of table "google_user_infos"
pg_dump: finding default expressions of table "google_user_infos"
pg_dump: finding the columns and types of table "user_avatars"
pg_dump: finding default expressions of table "user_avatars"
pg_dump: finding the columns and types of table "user_profiles"
pg_dump: finding default expressions of table "user_profiles"
pg_dump: finding the columns and types of table "quoted_posts"
pg_dump: finding default expressions of table "quoted_posts"
pg_dump: finding the columns and types of table "badge_groupings"
pg_dump: finding default expressions of table "badge_groupings"
pg_dump: finding the columns and types of table "incoming_referers"
pg_dump: finding default expressions of table "incoming_referers"
pg_dump: finding the columns and types of table "incoming_domains"
pg_dump: finding default expressions of table "incoming_domains"
pg_dump: finding the columns and types of table "topic_search_data"
pg_dump: finding the columns and types of table "badge_posts"
pg_dump: finding the columns and types of table "permalinks"
pg_dump: finding default expressions of table "permalinks"
pg_dump: finding the columns and types of table "warnings"
pg_dump: finding default expressions of table "warnings"
pg_dump: finding the columns and types of table "user_fields"
pg_dump: finding default expressions of table "user_fields"
pg_dump: finding the columns and types of table "user_exports"
pg_dump: finding default expressions of table "user_exports"
pg_dump: flagging inherited columns in subtables
pg_dump: reading indexes
pg_dump: reading indexes for table "schema_migrations"
pg_dump: reading indexes for table "topics"
pg_dump: reading indexes for table "posts"
pg_dump: reading indexes for table "users"
pg_dump: reading indexes for table "versions"
pg_dump: reading indexes for table "topic_views"
pg_dump: reading indexes for table "uploads"
pg_dump: reading indexes for table "topic_users"
pg_dump: reading indexes for table "topic_links"
pg_dump: reading indexes for table "post_timings"
pg_dump: reading indexes for table "message_bus"
pg_dump: reading indexes for table "notifications"
pg_dump: reading indexes for table "incoming_links"
pg_dump: reading indexes for table "post_replies"
pg_dump: reading indexes for table "categories"
pg_dump: reading indexes for table "category_featured_topics"
pg_dump: reading indexes for table "site_settings"
pg_dump: reading indexes for table "user_open_ids"
pg_dump: reading indexes for table "user_actions"
pg_dump: reading indexes for table "post_actions"
pg_dump: reading indexes for table "post_action_types"
pg_dump: reading indexes for table "topic_link_clicks"
pg_dump: reading indexes for table "category_featured_users"
pg_dump: reading indexes for table "twitter_user_infos"
pg_dump: reading indexes for table "email_logs"
pg_dump: reading indexes for table "topic_allowed_users"
pg_dump: reading indexes for table "facebook_user_infos"
pg_dump: reading indexes for table "invites"
pg_dump: reading indexes for table "topic_invites"
pg_dump: reading indexes for table "user_visits"
pg_dump: reading indexes for table "email_tokens"
pg_dump: reading indexes for table "drafts"
pg_dump: reading indexes for table "site_customizations"
pg_dump: reading indexes for table "draft_sequences"
pg_dump: reading indexes for table "post_search_data"
pg_dump: reading indexes for table "user_search_data"
pg_dump: reading indexes for table "category_search_data"
pg_dump: reading indexes for table "github_user_infos"
pg_dump: reading indexes for table "site_texts"
pg_dump: reading indexes for table "user_histories"
pg_dump: reading indexes for table "groups"
pg_dump: reading indexes for table "group_users"
pg_dump: reading indexes for table "category_groups"
pg_dump: reading indexes for table "topic_allowed_groups"
pg_dump: reading indexes for table "post_uploads"
pg_dump: reading indexes for table "optimized_images"
pg_dump: reading indexes for table "screened_emails"
pg_dump: reading indexes for table "screened_urls"
pg_dump: reading indexes for table "oauth2_user_infos"
...
pg_dump: executing SEQUENCE SET plugin_store_rows_id_seq
pg_dump: processing data for table "post_action_types"
pg_dump: dumping contents of table post_action_types
pg_dump: executing SEQUENCE SET post_action_types_id_seq
pg_dump: processing data for table "post_actions"
pg_dump: dumping contents of table post_actions
pg_dump: executing SEQUENCE SET post_actions_id_seq
pg_dump: processing data for table "post_custom_fields"
pg_dump: dumping contents of table post_custom_fields
pg_dump: executing SEQUENCE SET post_custom_fields_id_seq
pg_dump: processing data for table "post_details"
pg_dump: dumping contents of table post_details
pg_dump: executing SEQUENCE SET post_details_id_seq
pg_dump: processing data for table "post_replies"
pg_dump: dumping contents of table post_replies
pg_dump: processing data for table "post_revisions"
pg_dump: dumping contents of table post_revisions
pg_dump: executing SEQUENCE SET post_revisions_id_seq
pg_dump: processing data for table "post_search_data"
pg_dump: dumping contents of table post_search_data
pg_dump: processing data for table "post_timings"
pg_dump: dumping contents of table post_timings
pg_dump: processing data for table "post_uploads"
pg_dump: dumping contents of table post_uploads
pg_dump: executing SEQUENCE SET post_uploads_id_seq
pg_dump: processing data for table "posts"
pg_dump: dumping contents of table posts
pg_dump: [archiver] could not write to output file: No space left on device
EXCEPTION: pg_dump failed
/var/www/discourse/lib/backup_restore/backuper.rb:171:in `dump_public_schema'
/var/www/discourse/lib/backup_restore/backuper.rb:37:in `run'
/var/www/discourse/lib/backup_restore/backup_restore.rb:161:in `block in start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `fork'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:13:in `backup!'
/var/www/discourse/app/jobs/scheduled/create_backup.rb:10:in `execute'
/var/www/discourse/app/jobs/base.rb:153:in `block (2 levels) in perform'
Notifying 'system' of the end of the backup...
Cleaning stuff up...
Removing tmp '/var/www/discourse/tmp/backups/default/2015-01-18-030000' directory...
Unpausing sidekiq...
Something went wrong while unpausing Sidekiq.
Disabling readonly mode...
--------------------------------------------
---------------- EXCEPTION -----------------
MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis/client.rb:110:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis.rb:416:in `block in del'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis.rb:37:in `block in synchronize'
/usr/local/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis.rb:37:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis.rb:415:in `del'
/var/www/discourse/lib/discourse_redis.rb:63:in `del'
/var/www/discourse/lib/discourse.rb:177:in `disable_readonly_mode'
/var/www/discourse/lib/backup_restore/backuper.rb:310:in `disable_readonly_mode'
/var/www/discourse/lib/backup_restore/backuper.rb:288:in `clean_up'
/var/www/discourse/lib/backup_restore/backuper.rb:61:in `run'
/var/www/discourse/lib/backup_restore/backup_restore.rb:161:in `block in start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `fork'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:13:in `backup!'
/var/www/discourse/app/jobs/scheduled/create_backup.rb:10:in `execute'
/var/www/discourse/app/jobs/base.rb:153:in `block (2 levels) in perform'
--------------------------------------------
============================================
================ EXCEPTION =================
MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis/client.rb:110:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis.rb:416:in `block in del'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis.rb:37:in `block in synchronize'
/usr/local/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis.rb:37:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis.rb:415:in `del'
/var/www/discourse/lib/discourse_redis.rb:63:in `del'
/var/www/discourse/lib/backup_restore/backup_restore.rb:141:in `clear_shutdown_signal!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:170:in `block in start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `fork'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:13:in `backup!'
/var/www/discourse/app/jobs/scheduled/create_backup.rb:10:in `execute'
/var/www/discourse/app/jobs/base.rb:153:in `block (2 levels) in perform'
============================================
Sidekiq heartbeat test failed, restarting

The difference between the backup that apparently left the instance in read-only mode and that of the days previous appears to be just these extra lines in the latest log:

/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:13:in `backup!'
/var/www/discourse/app/jobs/scheduled/create_backup.rb:10:in `execute'
/var/www/discourse/app/jobs/base.rb:153:in `block (2 levels) in perform'
Notifying 'system' of the end of the backup...
Cleaning stuff up...
Removing tmp '/var/www/discourse/tmp/backups/default/2015-01-18-030000' directory...
Unpausing sidekiq...
Something went wrong while unpausing Sidekiq.
Disabling readonly mode...
--------------------------------------------
---------------- EXCEPTION -----------------
MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis/client.rb:110:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis.rb:416:in `block in del'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis.rb:37:in `block in synchronize'
/usr/local/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis.rb:37:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis.rb:415:in `del'
/var/www/discourse/lib/discourse_redis.rb:63:in `del'
/var/www/discourse/lib/discourse.rb:177:in `disable_readonly_mode'
/var/www/discourse/lib/backup_restore/backuper.rb:310:in `disable_readonly_mode'
/var/www/discourse/lib/backup_restore/backuper.rb:288:in `clean_up'
/var/www/discourse/lib/backup_restore/backuper.rb:61:in `run'
/var/www/discourse/lib/backup_restore/backup_restore.rb:161:in `block in start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `fork'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:13:in `backup!'
/var/www/discourse/app/jobs/scheduled/create_backup.rb:10:in `execute'
/var/www/discourse/app/jobs/base.rb:153:in `block (2 levels) in perform'
--------------------------------------------
============================================
================ EXCEPTION =================
MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis/client.rb:110:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis.rb:416:in `block in del'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis.rb:37:in `block in synchronize'
/usr/local/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis.rb:37:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.2.0/lib/redis.rb:415:in `del'
/var/www/discourse/lib/discourse_redis.rb:63:in `del'
/var/www/discourse/lib/backup_restore/backup_restore.rb:141:in `clear_shutdown_signal!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:170:in `block in start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `fork'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:13:in `backup!'
/var/www/discourse/app/jobs/scheduled/create_backup.rb:10:in `execute'
/var/www/discourse/app/jobs/base.rb:153:in `block (2 levels) in perform'
============================================
Sidekiq heartbeat test failed, restarting
4 Likes
2

Hmm if this is still an issue we should look into it @zogstrip

3

I’m seeing frequent backup failures without any notifications, messages or emails to either myself as an admin or the “system” user.

Both automatic and manual backups fail.

Here is the example log from an manually executed backup (small amount removed to fit post limits):

[2016-02-16 23:32:47] 'DeanMarkTaylor' has started the backup!
...
[2016-02-16 23:33:34] pg_dump: processing data for table "versions"
[2016-02-16 23:33:34] pg_dump: dumping contents of table versions
[2016-02-16 23:33:34] pg_dump: executing SEQUENCE SET versions_id_seq
[2016-02-16 23:33:34] pg_dump: processing data for table "warnings"
[2016-02-16 23:33:34] pg_dump: dumping contents of table warnings
[2016-02-16 23:33:34] pg_dump: executing SEQUENCE SET warnings_id_seq
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT actions_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT api_keys_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT application_requests_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT badge_groupings_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT badge_types_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT badges_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT blocked_emails_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT categories_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT categories_search_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT category_custom_fields_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT category_featured_topics_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT category_featured_users_pkey
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT category_groups_pkey
[2016-02-16 23:33:34] Disabling readonly mode...
[2016-02-16 23:33:34] pg_dump: creating CONSTRAINT category_users_pkey
[2016-02-16 23:33:34] Finalizing backup...
[2016-02-16 23:33:34] Updating dump for more awesomeness...
[2016-02-16 23:33:49] Creating archive: xxx-xx-2016-02-16-233247.tar.gz
[2016-02-16 23:33:49] Making sure archive does not already exist...
[2016-02-16 23:33:49] Creating empty archive...
[2016-02-16 23:33:49] Archiving metadata...
[2016-02-16 23:33:49] Archiving data dump...
[2016-02-16 23:33:58] Archiving uploads...
[2016-02-16 23:37:15] Gzipping archive...
[2016-02-16 23:45:26] Executing the after_create_hook for the backup
[2016-02-16 23:45:26] EXCEPTION: No such file or directory - /var/www/discourse/public/backups/default/xxx-xx-2016-02-16-233247.tar.gz
[2016-02-16 23:45:26] /var/www/discourse/app/models/backup.rb:75:in `size'
/var/www/discourse/app/models/backup.rb:75:in `block in create_from_filename'
/var/www/discourse/app/models/backup.rb:72:in `tap'
/var/www/discourse/app/models/backup.rb:72:in `create_from_filename'
/var/www/discourse/lib/backup_restore/backuper.rb:266:in `after_create_hook'
/var/www/discourse/lib/backup_restore/backuper.rb:49:in `run'
/var/www/discourse/lib/backup_restore/backup_restore.rb:161:in `block in start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `fork'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:13:in `backup!'
/var/www/discourse/app/controllers/admin/backups_controller.rb:31:in `create'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/abstract_controller/base.rb:198:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/rendering.rb:10:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:117:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:117:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/abstract_controller/callbacks.rb:19:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/rescue.rb:29:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `block in instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.5.1/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/abstract_controller/base.rb:137:in `process'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionview-4.2.5.1/lib/action_view/rendering.rb:30:in `process'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.8/lib/mini_profiler/profiling_methods.rb:106:in `block in profile_method'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_controller/metal.rb:196:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_controller/metal.rb:237:in `block in action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:74:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:74:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:43:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/mapper.rb:49:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/router.rb:43:in `block in serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/router.rb:30:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/router.rb:30:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:815:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/builder.rb:63:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/conditionalget.rb:38:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
/var/www/discourse/lib/middleware/anonymous_cache.rb:129:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/flash.rb:260:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:225:in `context'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:220:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/cookies.rb:560:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.5.1/lib/active_record/query_cache.rb:36:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-1.0.1/lib/logster/middleware/reporter.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.1/lib/rails/rack/logger.rb:38:in `call_app'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.1/lib/rails/rack/logger.rb:22:in `call'
/var/www/discourse/config/initializers/100-quiet_logger.rb:10:in `call_with_quiet_assets'
/var/www/discourse/config/initializers/100-silence_logger.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/request_id.rb:21:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/methodoverride.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/runtime.rb:18:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/sendfile.rb:113:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.8/lib/mini_profiler/profiler.rb:282:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/message_bus-2.0.0.beta.2/lib/message_bus/rack/middleware.rb:60:in `call'
/var/www/discourse/lib/middleware/request_tracker.rb:73:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.1/lib/rails/engine.rb:518:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.1/lib/rails/application.rb:165:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.1/lib/rails/railtie.rb:194:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.1/lib/rails/railtie.rb:194:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:562:in `process_client'
/var/www/discourse/lib/scheduler/defer.rb:85:in `process_client'
/var/www/discourse/lib/middleware/unicorn_oobgc.rb:95:in `process_client'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:658:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:132:in `start'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/bin/unicorn:126:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `<main>'
[2016-02-16 23:45:26] Notifying 'DeanMarkTaylor' of the end of the backup...
[2016-02-16 23:45:26] Removing old backups...
[2016-02-16 23:45:26] Cleaning stuff up...
[2016-02-16 23:45:26] Removing '.tar' leftovers...
[2016-02-16 23:45:27] Removing tmp '/var/www/discourse/tmp/backups/default/2016-02-16-233247' directory...
[2016-02-16 23:45:27] Unpausing sidekiq...
[2016-02-16 23:45:27] Marking backup as finished...
[2016-02-16 23:45:27] Finished!

The backups when created are created they are 8.6GB in size, so in this case it’s likely a space issue. However my main worry is that there are no notifications of failure.

root@forum:/var/discourse/shared/standalone/backups/default# df -h
Filesystem                 Size  Used Avail Use% Mounted on
udev                       2.0G  4.0K  2.0G   1% /dev
tmpfs                      396M  340K  396M   1% /run
/dev/disk/by-label/DOROOT   59G   38G   18G  68% /
none                       4.0K     0  4.0K   0% /sys/fs/cgroup
none                       5.0M     0  5.0M   0% /run/lock
none                       2.0G  1.1M  2.0G   1% /run/shm
none                       100M     0  100M   0% /run/user
none                        59G   38G   18G  68% /var/lib/docker/aufs/mnt/ff086f43e49a8dd91ab098f8a4db6c9b222b1c838a573d464ff0db7eee29c7e3
shm                         64M     0   64M   0% /var/lib/docker/containers/354e2015381ef26aaf4057f838969d54c96718c3c7491c32f01c3db2b4bef66a/shm
root@forum:/var/discourse/shared/standalone/backups/default# ls -l -h
total 8.6G
-rw-r--r-- 1 1000 www-data 8.6G Feb 14 22:38 xxx-xx-2016-02-15-033149.tar.gz

My current version (v1.5.0.beta10 +86):

Discourse 1.5.0.beta10 - https://github.com/discourse/discourse version 8e9a8472f4714a7e2e63e669aaa7b58c9c8cd0e8
1 Like
4

Are you still seeing this?

5

Yes, certainly with a manual backup.

Steps to reproduce

  1. I filled the disk leaving a couple of GB free using:
dd if=/dev/zero of=deleteme-temp-test-disk-fill.dat  bs=1GB  count=30
  1. I then executed manual backup.

Expected

To receive a notification either as myself “DeanMarkTaylor” or “system” that the backup success / failure.

Actual

No notification or message to any admin users.

Additional Notes

Deleting the temp disk filling file then executing "no-files) backup:

  • successfully completed
  • I “DeanMarkTaylor” received a notification / PM indicating success

#Error log

[2016-05-03 16:51:05] Disabling readonly mode...
[2016-05-03 16:51:05] Finalizing backup...
[2016-05-03 16:51:05] Updating dump for more awesomeness...
[2016-05-03 16:51:32] Creating archive: example-uk-2016-05-03-165000.tar.gz
[2016-05-03 16:51:32] Making sure archive does not already exist...
[2016-05-03 16:51:32] Creating empty archive...
[2016-05-03 16:51:32] Archiving metadata...
[2016-05-03 16:51:32] Archiving data dump...
[2016-05-03 16:51:55] Archiving uploads...
[2016-05-03 16:52:48] Gzipping archive...
[2016-05-03 16:52:48] Executing the after_create_hook for the backup
[2016-05-03 16:52:48] EXCEPTION: No such file or directory - /var/www/discourse/public/backups/default/example-uk-2016-05-03-165000.tar.gz
[2016-05-03 16:52:48] /var/www/discourse/app/models/backup.rb:75:in `size'
/var/www/discourse/app/models/backup.rb:75:in `block in create_from_filename'
/var/www/discourse/app/models/backup.rb:72:in `tap'
/var/www/discourse/app/models/backup.rb:72:in `create_from_filename'
/var/www/discourse/lib/backup_restore/backuper.rb:266:in `after_create_hook'
/var/www/discourse/lib/backup_restore/backuper.rb:49:in `run'
/var/www/discourse/lib/backup_restore/backup_restore.rb:161:in `block in start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `fork'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:13:in `backup!'
/var/www/discourse/app/controllers/admin/backups_controller.rb:31:in `create'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/abstract_controller/base.rb:198:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_controller/metal/rendering.rb:10:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:117:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:117:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:505:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:505:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/abstract_controller/callbacks.rb:19:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_controller/metal/rescue.rb:29:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/notifications.rb:164:in `block in instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/notifications.rb:164:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.6/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/abstract_controller/base.rb:137:in `process'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionview-4.2.6/lib/action_view/rendering.rb:30:in `process'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.9.2/lib/mini_profiler/profiling_methods.rb:102:in `block in profile_method'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_controller/metal.rb:196:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_controller/metal.rb:237:in `block in action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_dispatch/routing/route_set.rb:74:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_dispatch/routing/route_set.rb:74:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_dispatch/routing/route_set.rb:43:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_dispatch/routing/mapper.rb:49:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_dispatch/journey/router.rb:43:in `block in serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_dispatch/journey/router.rb:30:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_dispatch/journey/router.rb:30:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_dispatch/routing/route_set.rb:817:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/builder.rb:63:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/conditionalget.rb:38:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
/var/www/discourse/lib/middleware/anonymous_cache.rb:129:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/flash.rb:260:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:225:in `context'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:220:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/cookies.rb:560:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.6/lib/active_record/query_cache.rb:36:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-1.2.2/lib/logster/middleware/reporter.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/rack/logger.rb:38:in `call_app'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/rack/logger.rb:22:in `call'
/var/www/discourse/config/initializers/100-quiet_logger.rb:13:in `call_with_quiet_assets'
/var/www/discourse/config/initializers/100-silence_logger.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/request_id.rb:21:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/methodoverride.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/runtime.rb:18:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/sendfile.rb:113:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.9.2/lib/mini_profiler/profiler.rb:281:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/message_bus-2.0.0.beta.5/lib/message_bus/rack/middleware.rb:60:in `call'
/var/www/discourse/lib/middleware/request_tracker.rb:73:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/engine.rb:518:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/application.rb:165:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/railtie.rb:194:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.6/lib/rails/railtie.rb:194:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:562:in `process_client'
/var/www/discourse/lib/scheduler/defer.rb:85:in `process_client'
/var/www/discourse/lib/middleware/unicorn_oobgc.rb:95:in `process_client'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:658:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:283:in `join'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/bin/unicorn:126:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:22:in `load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:22:in `<main>'
[2016-05-03 16:52:48] Notifying 'DeanMarkTaylor' of the end of the backup...
[2016-05-03 16:52:48] Removing old backups...
[2016-05-03 16:52:48] Cleaning stuff up...
[2016-05-03 16:52:48] Removing '.tar' leftovers...
[2016-05-03 16:52:49] Removing tmp '/var/www/discourse/tmp/backups/default/2016-05-03-165000' directory...
[2016-05-03 16:52:49] Unpausing sidekiq...
[2016-05-03 16:52:49] Marking backup as finished...
[2016-05-03 16:52:49] Finished!

Discourse 1.6.0.beta2 - https://github.com/discourse/discourse version 0b6d12f95dce480977101cec617056dec0660c9a
3 Likes
6

Noted backups weren’t running again because the nightly uploads to Amazon S3 didn’t appear on the Bandwidth graph:

Pasted image782×297 30.2 KB

So in the case of this Discourse / Digital Ocean instance, confirmed failure notifications didn’t occur and backups were failing due to space issues.

7

@zogstrip does the above exception call stack let us handle this more gracefully, somehow?

8

Are you still seeing this problem @DeanMarkTaylor?

9

By the way I’m not ignoring this - it’s on my list to check again - I’m 90% sure it’s still a problem - but will check again.

1 Like
10

We should get a repro as no backups is very Bad Mojo.

2 Likes
11

I can confirm this still occurs…

And I believe I have a solution…

Background

Basically logging is unable to complete because the disk is full.

It is possible to allow logging to complete by deleting the intermediate files (*.tar) immediately after the error before attempting to log / send notifications for the event rather than leaving it until after.

Initial change (didn’t quite work)

Changing the following should delete the *.tar file before attempting to log / send notifications.

From

https://github.com/discourse/discourse/blob/a4deb0e47db9368936d5755a2088999ad48bacf1/lib/backup_restore/backuper.rb#L54-L63

To

ensure
  clean_up

  begin
    notify_user
    remove_old
  rescue => ex
    Rails.logger.error("#{ex}\n" + ex.backtrace.join("\n"))
  end

  @success ? log("[SUCCESS]") : log("[FAILED]")

Further changes required

I tested the above on a production installation - it didn’t work…

… I tracked down the problem to the function remove_tar_leftovers:
https://github.com/discourse/discourse/blob/a4deb0e47db9368936d5755a2088999ad48bacf1/lib/backup_restore/backuper.rb#L276-L279

Whilst system('rm', '-f', "#{@archive_directory}/*.tar") looks like it should delete the tar file it doesn’t actually delete the file in this case.

I also tested the following, none of which deleted the *.tar file:

#1
system('rm', '--verbose', '-f', "#{@archive_directory}/*.tar")

#2
stdout, stderr, status = Open3.capture3('rm', '--verbose', '-f', "#{@archive_directory}/*.tar")
log stdout
log stderr
log status

#3
FileUtils.rm_rf("#{@archive_directory}/*.tar")

Finally using the following actually deleted the file as expected when coupled with moving clean_up call earlier:

log %x[rm --verbose -f #{@archive_directory}/*.tar 2>&1].inspect

This actually allowed the notifications to be delivered.

Hope this helps nail down complete solution.

5 Likes
12

I think this just happened to me, although I can’t document it as well as @DeanMarkTaylor. I have daily backups to S3. Got low on disk space, due to several rebuilds, and recovered using ./launcher cleanup, so no harm done. But I noticed that the most recent backup at S3 is from 5 days ago. There is an unzipped tar file from today sitting in my local backups directory.

When backups were failing, I had 3.7GB free. Backups are about 5.4GB zipped.

I’d rather not have to “notice” backups failing.