Backups fail to write to S3 when IMDSv1 is disabled and the setting “s3 use iam profile” is enabled.
As an internal security requirement, we disabled the use of IMDSv1 on the EC2 instance hosting our Discourse installation. This causes an error during the backup process while uploading the archive to S3.
[2024-08-08 03:38:47] Uploading archive…
[2024-08-08 03:38:49] EXCEPTION: unable to sign request without credentials set
Original report here.
Full Log
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/sign.rb:120:in rescue in initialize' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/sign.rb:109:in initialize’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/sign.rb:34:in new' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/sign.rb:34:in signer_for’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/sign.rb:46:in call' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/transfer_encoding.rb:26:in call’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12:in call' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/s3_signer.rb:48:in call’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/redirects.rb:20:in call' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/retry_errors.rb:362:in call’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/user_agent.rb:37:in call' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/http_checksum.rb:20:in call’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/endpoint_pattern.rb:30:in call' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/checksum_algorithm.rb:137:in call’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/request_compression.rb:94:in call' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/express_session_auth.rb:50:in call’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/expect_100_continue.rb:23:in call' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/bucket_name_restrictions.rb:21:in call’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/rest/handler.rb:10:in call' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/recursion_detection.rb:18:in call’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/endpoints.rb:49:in call' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/endpoint_discovery.rb:84:in call’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/endpoint.rb:47:in call' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/param_validator.rb:26:in call’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in call' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/dualstack.rb:21:in call’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/accelerate.rb:43:in call' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in call’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in call' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in call’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/param_converter.rb:26:in call' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/request_callback.rb:89:in call’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/response_paging.rb:12:in call' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/response_target.rb:24:in call’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/request.rb:72:in send_request' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/poller.rb:66:in block in send_request’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/user_agent.rb:28:in feature' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/poller.rb:65:in send_request’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/poller.rb:51:in call' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/waiter.rb:107:in block in poll’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/waiter.rb:104:in loop' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/waiter.rb:104:in poll’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/waiter.rb:94:in block (2 levels) in wait' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/waiter.rb:93:in catch’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/waiter.rb:93:in block in wait' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/waiter.rb:92:in catch’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/waiter.rb:92:in wait' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/waiters.rb:123:in wait’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/bucket.rb:99:in block in wait_until_exists' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/user_agent.rb:28:in feature’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/bucket.rb:98:in wait_until_exists' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/bucket.rb:79:in exists?’
/var/www/discourse/lib/s3_helper.rb:429:in s3_bucket' /var/www/discourse/lib/s3_helper.rb:259:in object’
/var/www/discourse/lib/backup_restore/s3_backup_store.rb:45:in upload_file' /var/www/discourse/lib/backup_restore/backuper.rb:351:in upload_archive’
/var/www/discourse/lib/backup_restore/backuper.rb:41:in run' /var/www/discourse/lib/backup_restore.rb:13:in backup!’
/var/www/discourse/app/jobs/regular/create_backup.rb:10:in execute' /var/www/discourse/app/jobs/base.rb:305:in block (2 levels) in perform’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-5.0.0/lib/rails_multisite/connection_management.rb:82:in with_connection' /var/www/discourse/app/jobs/base.rb:292:in block in perform’
/var/www/discourse/app/jobs/base.rb:288:in each' /var/www/discourse/app/jobs/base.rb:288:in perform’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:202:in execute_job' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:170:in block (2 levels) in process’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:177:in block in invoke' /var/www/discourse/lib/sidekiq/pausable.rb:132:in call’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:179:in block in invoke' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:182:in invoke’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:169:in block in process' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:136:in block (6 levels) in dispatch’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:113:in local' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:135:in block (5 levels) in dispatch’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq.rb:44:in block in <module:Sidekiq>' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:131:in block (4 levels) in dispatch’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:263:in stats' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:126:in block (3 levels) in dispatch’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:13:in call' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:125:in block (2 levels) in dispatch’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:80:in global' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:124:in block in dispatch’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:39:in prepare' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:123:in dispatch’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:168:in process' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:78:in process_one’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:68:in run' /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:8:in watchdog’
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:17:in `block in safe_thread’