Backup Failure after Upgrade to 2.3.0.beta9

I recently upgraded a number of Discourse installs to 2.3.0.beta9, and three of them now throw “Backup failed” errors every time their (daily) scheduled backups run. When I manually run the backups, they appear to succeed, only to fail after saying they have finished. Here is part of an example log:

[2019-05-23 12:31:27] [STARTED]
[2019-05-23 12:31:27] 'mahler' has started the backup!
[2019-05-23 12:31:27] Marking backup as running...
[2019-05-23 12:31:27] Making sure '/var/www/discourse/tmp/backups/default/2019-05-23-123127' exists...
[2019-05-23 12:31:27] Making sure '/var/www/discourse/public/backups/default' exists...
[2019-05-23 12:31:27] Enabling readonly mode...
[2019-05-23 12:31:27] Pausing sidekiq...
[2019-05-23 12:31:27] Waiting for sidekiq to finish running jobs...
[2019-05-23 12:31:27] Dumping the public schema of the database...

…

[2019-05-23 12:31:44] Unpausing sidekiq...
[2019-05-23 12:31:44] Disabling readonly mode...
[2019-05-23 12:31:44] Finalizing backup...
[2019-05-23 12:31:44] Finalizing database dump file: freefora-2019-05-23-123127-v20190514055014.sql.gz
[2019-05-23 12:31:44] Removing tmp '/var/www/discourse/tmp/backups/default/2019-05-23-123127' directory...
[2019-05-23 12:31:44] Uploading archive...
[2019-05-23 12:31:46] EXCEPTION: Compliance for object does not allow deletion
[2019-05-23 12:31:46] /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/plugins/sse_cpk.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/plugins/dualstack.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/plugins/accelerate.rb:35:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/seahorse/client/plugins/response_target.rb:23:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/seahorse/client/request.rb:70:in `send_request'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/client.rb:471:in `complete_multipart_upload'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/multipart_file_uploader.rb:56:in `complete_upload'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/multipart_file_uploader.rb:45:in `upload'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/file_uploader.rb:32:in `upload'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/customizations/object.rb:308:in `upload_file'
/var/www/discourse/lib/backup_restore/s3_backup_store.rb:44:in `upload_file'
/var/www/discourse/lib/backup_restore/backuper.rb:268:in `upload_archive'
/var/www/discourse/lib/backup_restore/backuper.rb:51:in `run'
/var/www/discourse/lib/backup_restore/backup_restore.rb:169:in `block in start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:166:in `fork'
/var/www/discourse/lib/backup_restore/backup_restore.rb:166:in `start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:20:in `backup!'
/var/www/discourse/app/controllers/admin/backups_controller.rb:40:in `create'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/abstract_controller/base.rb:194:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/rendering.rb:30:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:132:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/abstract_controller/callbacks.rb:41:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/rescue.rb:22:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `block in instrument'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/abstract_controller/base.rb:134:in `process'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionview-5.2.3/lib/action_view/rendering.rb:32:in `process'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.0.2/lib/mini_profiler/profiling_methods.rb:78:in `block in profile_method'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal.rb:191:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal.rb:252:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:840:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-protection-2.0.5/lib/rack/protection/frame_options.rb:31:in `call'
/var/www/discourse/lib/middleware/omniauth_bypass_middleware.rb:32:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/tempfile_reaper.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/conditional_get.rb:38:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call'
/var/www/discourse/lib/content_security_policy/middleware.rb:12:in `call'
/var/www/discourse/lib/middleware/anonymous_cache.rb:220:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:232:in `context'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:226:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:98:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/logster-2.3.0/lib/logster/middleware/reporter.rb:30:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:38:in `call_app'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:28:in `call'
/var/www/discourse/config/initializers/100-quiet_logger.rb:18:in `call'
/var/www/discourse/config/initializers/100-silence_logger.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/var/www/discourse/lib/middleware/enforce_hostname.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/method_override.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.0.2/lib/mini_profiler/profiler.rb:171:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-2.2.0/lib/message_bus/rack/middleware.rb:57:in `call'
/var/www/discourse/lib/middleware/request_tracker.rb:163:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:524:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/railtie.rb:190:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/railtie.rb:190:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:68:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/lib/unicorn/http_server.rb:605:in `process_client'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/lib/unicorn/http_server.rb:700:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/lib/unicorn/http_server.rb:548:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/lib/unicorn/http_server.rb:144:in `start'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/bin/unicorn:128:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `<main>'
[2019-05-23 12:31:46] Deleting old backups...
[2019-05-23 12:31:46] Cleaning stuff up...
[2019-05-23 12:31:46] Removing archive from local storage...
[2019-05-23 12:31:46] Removing '.tar' leftovers...
[2019-05-23 12:31:46] Marking backup as finished...
[2019-05-23 12:31:46] Refreshing disk stats...
[2019-05-23 12:31:46] Notifying 'mahler' of the end of the backup...

As best I can tell, there is no difference between these installs and the others that are not having this issue. I believe this may be similar to an issue previously reported here, but that seems to have more or less resolved itself.

Everything is fully up to date and I have rebuilt the app a number of times, but the issue persists.

Are you uploading your backups to S3 (or S3-like provider)? The error which appears gives the indication that it’s trying to delete your oldest backup and it cannot.

4 Likes

Yes, I am uploading backups to S3 (to an S3-compatible storage provider, anyway).

That error indicates you may have incorrect bucket permissions set. The user should have permissions to delete from the bucket in order to keep in line with your maximum backups setting.

3 Likes

There is a retention policy set on the backups (at the S3 end), but I just bumped the “maximum backups” setting to 1000 and received the same error when I manually ran a backup. (n.b., there are definitely nowhere near one thousand backups in any of the relevant buckets.)

The important part of the error is this:

EXCEPTION: Compliance for object does not allow deletion

aws-sdk-s3/client.rb:471:in `complete_multipart_upload'
aws-sdk-s3/multipart_file_uploader.rb:56:in `complete_upload'
aws-sdk-s3/multipart_file_uploader.rb:45:in `upload'

It fails to finish the multipart upload because there’s probably some sort of policy or object lock that prevents deletion of uploaded parts.

5 Likes

It’s definitely related to the compliance policy. I disabled the compliance policy on one of the buckets and re-ran the backup and it completed successfully. So, I guess this is (now) a request to enable compatibility with compliance policies.

(Is the multipart upload new? This error just started occurring after upgrading to 2.3.0.beta9.)

1 Like

There weren’t any changes to S3 backups since the beginning of the year.

2 Likes

I have no idea why this error just started occurring, then. The retention policies on those buckets were put in place over a year ago.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.