Problem with Backblaze for backup- Failed to list backups from S3: Signature validation failed

JulianR · April 28, 2021, 7:55pm

Hi-

trying to set up a backblaze bucket for backup use only.

Followed: REQ: Support S3 backup to a service like Backblaze - #4 by frold
And referenced the linked post “using-object-storage-for-uploads-s3-clones”.

In app.yml I’ve added:

  DISCOURSE_S3_REGION: “s3.us-west-002”
  DISCOURSE_S3_INSTALL_CORS_RULE: false
  DISCOURSE_S3_ENDPOINT: https://s3.us-west-002.backblazeb2.com
  DISCOURSE_S3_ACCESS_KEY_ID: xxxxxxxxxxxxxxxxxxxx
  DISCOURSE_S3_SECRET_ACCESS_KEY: xxxxxxxxxxxxxxxxxxxxxxx
  DISCOURSE_S3_BACKUP_BUCKET: bucket/backups
  DISCOURSE_BACKUP_LOCATION: s3

I did recreate a new bucket set to public and all lowercased name after reading Setting up file and image uploads to S3.

In backblaze I did create a new App key, which only has access to the discourse bucket; it does have permission to list all bucket names.

On the backup page in discourse I get a sad face/broken page. The logs show:

Failed to list backups from S3: Signature validation failed

Backtrace:

/var/www/discourse/lib/backup_restore/s3_backup_store.rb:83:in `rescue in unsorted_files'

/var/www/discourse/lib/backup_restore/s3_backup_store.rb:72:in `unsorted_files'

/var/www/discourse/lib/backup_restore/backup_store.rb:23:in `files'

/var/www/discourse/app/controllers/admin/backups_controller.rb:22:in `block (2 levels) in index'

actionpack-6.1.3.1/lib/action_controller/metal/mime_responds.rb:214:in `respond_to'

/var/www/discourse/app/controllers/admin/backups_controller.rb:11:in `index'

actionpack-6.1.3.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'

actionpack-6.1.3.1/lib/abstract_controller/base.rb:228:in `process_action'

actionpack-6.1.3.1/lib/action_controller/metal/rendering.rb:30:in `process_action'

actionpack-6.1.3.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'

activesupport-6.1.3.1/lib/active_support/callbacks.rb:117:in `block in run_callbacks'

/var/www/discourse/app/controllers/application_controller.rb:383:in `block in with_resolved_locale'

i18n-1.8.10/lib/i18n.rb:314:in `with_locale'

/var/www/discourse/app/controllers/application_controller.rb:383:in `with_resolved_locale'

activesupport-6.1.3.1/lib/active_support/callbacks.rb:126:in `block in run_callbacks'

activesupport-6.1.3.1/lib/active_support/callbacks.rb:137:in `run_callbacks'

actionpack-6.1.3.1/lib/abstract_controller/callbacks.rb:41:in `process_action'

actionpack-6.1.3.1/lib/action_controller/metal/rescue.rb:22:in `process_action'

actionpack-6.1.3.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'

activesupport-6.1.3.1/lib/active_support/notifications.rb:203:in `block in instrument'

activesupport-6.1.3.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'

activesupport-6.1.3.1/lib/active_support/notifications.rb:203:in `instrument'

actionpack-6.1.3.1/lib/action_controller/metal/instrumentation.rb:33:in `process_action'

actionpack-6.1.3.1/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'

activerecord-6.1.3.1/lib/active_record/railties/controller_runtime.rb:27:in `process_action'

actionpack-6.1.3.1/lib/abstract_controller/base.rb:165:in `process'

actionview-6.1.3.1/lib/action_view/rendering.rb:39:in `process'

rack-mini-profiler-2.3.1/lib/mini_profiler/profiling_methods.rb:111:in `block in profile_method'

actionpack-6.1.3.1/lib/action_controller/metal.rb:190:in `dispatch'

actionpack-6.1.3.1/lib/action_controller/metal.rb:254:in `dispatch'

actionpack-6.1.3.1/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'

actionpack-6.1.3.1/lib/action_dispatch/routing/route_set.rb:33:in `serve'

actionpack-6.1.3.1/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'

actionpack-6.1.3.1/lib/action_dispatch/routing/mapper.rb:49:in `serve'

actionpack-6.1.3.1/lib/action_dispatch/journey/router.rb:50:in `block in serve'

actionpack-6.1.3.1/lib/action_dispatch/journey/router.rb:32:in `each'

actionpack-6.1.3.1/lib/action_dispatch/journey/router.rb:32:in `serve'

actionpack-6.1.3.1/lib/action_dispatch/routing/route_set.rb:842:in `call'

/var/www/discourse/lib/middleware/omniauth_bypass_middleware.rb:71:in `call'

rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'

rack-2.2.3/lib/rack/conditional_get.rb:27:in `call'

rack-2.2.3/lib/rack/head.rb:12:in `call'

actionpack-6.1.3.1/lib/action_dispatch/http/permissions_policy.rb:22:in `call'

/var/www/discourse/lib/content_security_policy/middleware.rb:12:in `call'

/var/www/discourse/lib/middleware/anonymous_cache.rb:355:in `call'

rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'

rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'

actionpack-6.1.3.1/lib/action_dispatch/middleware/cookies.rb:689:in `call'

actionpack-6.1.3.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'

activesupport-6.1.3.1/lib/active_support/callbacks.rb:98:in `run_callbacks'

actionpack-6.1.3.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'

actionpack-6.1.3.1/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'

actionpack-6.1.3.1/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'

actionpack-6.1.3.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'

logster-2.9.6/lib/logster/middleware/reporter.rb:43:in `call'

railties-6.1.3.1/lib/rails/rack/logger.rb:37:in `call_app'

railties-6.1.3.1/lib/rails/rack/logger.rb:28:in `call'

/var/www/discourse/config/initializers/100-quiet_logger.rb:23:in `call'

/var/www/discourse/config/initializers/100-silence_logger.rb:31:in `call'

actionpack-6.1.3.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'

actionpack-6.1.3.1/lib/action_dispatch/middleware/request_id.rb:26:in `call'

/var/www/discourse/lib/middleware/enforce_hostname.rb:23:in `call'

rack-2.2.3/lib/rack/method_override.rb:24:in `call'

actionpack-6.1.3.1/lib/action_dispatch/middleware/executor.rb:14:in `call'

rack-2.2.3/lib/rack/sendfile.rb:110:in `call'

actionpack-6.1.3.1/lib/action_dispatch/middleware/host_authorization.rb:92:in `call'

rack-mini-profiler-2.3.1/lib/mini_profiler/profiler.rb:373:in `call'

message_bus-3.3.4/lib/message_bus/rack/middleware.rb:61:in `call'

/var/www/discourse/lib/middleware/request_tracker.rb:177:in `call'

railties-6.1.3.1/lib/rails/engine.rb:539:in `call'

railties-6.1.3.1/lib/rails/railtie.rb:207:in `public_send'

railties-6.1.3.1/lib/rails/railtie.rb:207:in `method_missing'

rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'

rack-2.2.3/lib/rack/urlmap.rb:58:in `each'

rack-2.2.3/lib/rack/urlmap.rb:58:in `call'

unicorn-6.0.0/lib/unicorn/http_server.rb:634:in `process_client'

unicorn-6.0.0/lib/unicorn/http_server.rb:732:in `worker_loop'

unicorn-6.0.0/lib/unicorn/http_server.rb:547:in `spawn_missing_workers'

unicorn-6.0.0/lib/unicorn/http_server.rb:143:in `start'

unicorn-6.0.0/bin/unicorn:128:in `<top (required)>'

/var/www/discourse/vendor/bundle/ruby/2.7.0/bin/unicorn:23:in `load'

/var/www/discourse/vendor/bundle/ruby/2.7.0/bin/unicorn:23:in `<main>'

I think I understand each of the building blocks and steps, but no idea what the next steps are to debug this.

The key id and key are alphanumeric, no “+” or other symbols… but even so, should they be quoted?

Any suggestions appreciated!

Falco · April 28, 2021, 8:06pm

Can you try following Using Object Storage for Uploads (S3 & Clones)?

JulianR · April 28, 2021, 9:41pm

Thanks for the suggestion @Falco !

The appropriate section of my app.yml now looks like:

  ## DISCOURSE_USE_S3: true  -- for CDN usage?!
  DISCOURSE_S3_REGION: “us-west-002”
  DISCOURSE_S3_INSTALL_CORS_RULE: false
  DISCOURSE_S3_CONFIGURE_TOMBSTONE_POLICY: false
  DISCOURSE_S3_ENDPOINT: https://s3.us-west-002.backblazeb2.com
  DISCOURSE_S3_ACCESS_KEY_ID: XXXXXXXXXXXXXXXXXX
  DISCOURSE_S3_SECRET_ACCESS_KEY: XXXXXXXXXXXXXXXXXX
  ## DISCOURSE_S3_CDN_URL: https://
  DISCOURSE_S3_BUCKET: discourse
  DISCOURSE_S3_BACKUP_BUCKET: discourse/backups
  DISCOURSE_BACKUP_LOCATION: s3

I also noticed the note re CORS, so I’ve also tried setting the bucket’s CORS policy to allow access from all domains for both S3 and B2 protocols (will lock down once I’ve got it working.)

No luck with that.

And I’ve created another app key that has full access to all buckets and rebuilt with that. Also no luck.

Further suggestions?

JulianR · April 28, 2021, 9:56pm

I’ve also just tried with the above config but with

DISCOURSE_USE_S3: true

I have other buckets successfully being used to back up other services (note- that backup tool uses the B2 API.)

Small update: removed the “all buckets” API key and now get an invalid key error, so my server is connecting to Backblaze. No idea why the sig is off. Just double checked and the time is correct (TZ is UTC.)

JulianR · April 29, 2021, 8:00pm

It “just worked” the next day.