Backups fail to write to S3 when IMDSv1 is disabled and the setting “s3 use iam profile” is enabled.
As an internal security requirement, we disabled the use of IMDSv1 on the EC2 instance hosting our Discourse installation. This causes an error during the backup process while uploading the archive to S3.
[2024-08-08 03:38:47] Uploading archive...
[2024-08-08 03:38:49] EXCEPTION: unable to sign request without credentials set
Original report here.
Full Log
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/sign.rb:120:in `rescue in initialize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/sign.rb:109:in `initialize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/sign.rb:34:in `new'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/sign.rb:34:in `signer_for'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/sign.rb:46:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/transfer_encoding.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/s3_signer.rb:48:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/redirects.rb:20:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/retry_errors.rb:362:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/user_agent.rb:37:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/http_checksum.rb:20:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/endpoint_pattern.rb:30:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/checksum_algorithm.rb:137:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/request_compression.rb:94:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/express_session_auth.rb:50:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/expect_100_continue.rb:23:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/bucket_name_restrictions.rb:21:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/rest/handler.rb:10:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/recursion_detection.rb:18:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/endpoints.rb:49:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/endpoint_discovery.rb:84:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/endpoint.rb:47:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/param_validator.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/dualstack.rb:21:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/plugins/accelerate.rb:43:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/request_callback.rb:89:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/response_target.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/request.rb:72:in `send_request'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/poller.rb:66:in `block in send_request'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/user_agent.rb:28:in `feature'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/poller.rb:65:in `send_request'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/poller.rb:51:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/waiter.rb:107:in `block in poll'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/waiter.rb:104:in `loop'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/waiter.rb:104:in `poll'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/waiter.rb:94:in `block (2 levels) in wait'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/waiter.rb:93:in `catch'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/waiter.rb:93:in `block in wait'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/waiter.rb:92:in `catch'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/waiters/waiter.rb:92:in `wait'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/waiters.rb:123:in `wait'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/bucket.rb:99:in `block in wait_until_exists'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/user_agent.rb:28:in `feature'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/bucket.rb:98:in `wait_until_exists'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.143.0/lib/aws-sdk-s3/bucket.rb:79:in `exists?'
/var/www/discourse/lib/s3_helper.rb:429:in `s3_bucket'
/var/www/discourse/lib/s3_helper.rb:259:in `object'
/var/www/discourse/lib/backup_restore/s3_backup_store.rb:45:in `upload_file'
/var/www/discourse/lib/backup_restore/backuper.rb:351:in `upload_archive'
/var/www/discourse/lib/backup_restore/backuper.rb:41:in `run'
/var/www/discourse/lib/backup_restore.rb:13:in `backup!'
/var/www/discourse/app/jobs/regular/create_backup.rb:10:in `execute'
/var/www/discourse/app/jobs/base.rb:305:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-5.0.0/lib/rails_multisite/connection_management.rb:82:in `with_connection'
/var/www/discourse/app/jobs/base.rb:292:in `block in perform'
/var/www/discourse/app/jobs/base.rb:288:in `each'
/var/www/discourse/app/jobs/base.rb:288:in `perform'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:202:in `execute_job'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:170:in `block (2 levels) in process'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:177:in `block in invoke'
/var/www/discourse/lib/sidekiq/pausable.rb:132:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:179:in `block in invoke'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/middleware/chain.rb:182:in `invoke'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:169:in `block in process'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:113:in `local'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq.rb:44:in `block in <module:Sidekiq>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:263:in `stats'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:13:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/job_retry.rb:80:in `global'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:124:in `block in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/job_logger.rb:39:in `prepare'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:123:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:168:in `process'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:78:in `process_one'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/processor.rb:68:in `run'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:8:in `watchdog'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/sidekiq-6.5.12/lib/sidekiq/component.rb:17:in `block in safe_thread'