# Backups fail to upload to S3 several times- works eventually
**URL:** https://meta.discourse.org/t/backups-fail-to-upload-to-s3-several-times-works-eventually/366874
**Category:** Support
**Tags:** backups, s3
**Created:** [May 20, 2025, 4:26pm UTC](https://meta.discourse.org/t/backups-fail-to-upload-to-s3-several-times-works-eventually/366874 "2025-05-20T16:26:08Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@markschmucker](https://meta.discourse.org/u/markschmucker)
#### Post date: [May 20, 2025, 4:26pm UTC](https://meta.discourse.org/t/backups-fail-to-upload-to-s3-several-times-works-eventually/366874/1 "2025-05-20T16:26:08Z")
Backups to S3 have been working for several years. Starting a month ago, I often get several notifications that the backup failed. Then it retries, and about an hour later fails again, so I get a second notification, and often up to six notifications in one day before it finally succeeds.
The log from the notification says it’s failing to upload the zipped tar file to S3. I can’t find any errors in my S3 account.
> **Log**
>
> The first part of the log looks normal, then:
>
> [2025-05-20 07:11:38] Finalizing backup…
> [2025-05-20 07:11:38] Creating archive: 506-investor-group-2025-05-20-070428-v20250513161753.tar.gz
> [2025-05-20 07:11:38] Making sure archive does not already exist…
> [2025-05-20 07:11:38] Creating empty archive…
> [2025-05-20 07:11:38] Archiving data dump…
> [2025-05-20 07:12:17] Archiving uploads…
> [2025-05-20 07:15:48] Removing tmp ‘/var/www/discourse/tmp/backups/default/2025-05-20-070428’ directory…
> [2025-05-20 07:15:48] Gzipping archive, this may take a while…
> [2025-05-20 07:32:51] Uploading archive…
> [2025-05-20 07:34:28] EXCEPTION: Sidekiq::Shutdown
> [2025-05-20 07:34:28] /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/multipart\_file\_uploader.rb:199:in `value' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/multipart_file_uploader.rb:199:in `map’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/multipart\_file\_uploader.rb:199:in `upload_in_threads' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/multipart_file_uploader.rb:82:in `upload\_parts’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/multipart\_file\_uploader.rb:59:in `upload' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/file_uploader.rb:42:in `block in upload’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/user\_agent.rb:69:in `metric' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/file_uploader.rb:40:in `upload’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/customizations/object.rb:477:in `block in upload_file' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/user_agent.rb:69:in `metric’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/customizations/object.rb:476:in `upload_file' /var/www/discourse/lib/backup_restore/s3_backup_store.rb:48:in `upload\_file’
> /var/www/discourse/lib/backup\_restore/backuper.rb:351:in `upload_archive' /var/www/discourse/lib/backup_restore/backuper.rb:41:in `run’
> /var/www/discourse/lib/backup\_restore.rb:13:in `backup!' /var/www/discourse/app/jobs/regular/create_backup.rb:10:in `execute’
> /var/www/discourse/app/jobs/base.rb:316:in `block (2 levels) in perform' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with\_connection’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails\_multisite-6.1.0/lib/rails\_multisite/connection\_management.rb:21:in `with_connection' /var/www/discourse/app/jobs/base.rb:303:in `block in perform’
> /var/www/discourse/app/jobs/base.rb:299:in `each' /var/www/discourse/app/jobs/base.rb:299:in `perform’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:220:in `execute_job' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:185:in `block (4 levels) in process’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:180:in `traverse' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in `block in traverse’
> /var/www/discourse/lib/sidekiq/discourse\_event.rb:6:in `call' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in `traverse’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in `block in traverse' /var/www/discourse/lib/sidekiq/pausable.rb:131:in `call’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in `traverse' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in `block in traverse’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job/interrupt\_handler.rb:9:in `call' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in `traverse’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in `block in traverse' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/metrics/tracking.rb:26:in `track’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/metrics/tracking.rb:134:in `call' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in `traverse’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:173:in `invoke' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:184:in `block (3 levels) in process’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:145:in `block (6 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job_retry.rb:118:in `local’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:144:in `block (5 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/config.rb:39:in `block in class:Config’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:139:in `block (4 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:281:in `stats’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:134:in `block (3 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job_logger.rb:15:in `call’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:133:in `block (2 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job_retry.rb:85:in `global’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:132:in `block in dispatch' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job_logger.rb:40:in `prepare’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:131:in `dispatch' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:183:in `block (2 levels) in process’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:182:in `handle_interrupt' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:182:in `block in process’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:181:in `handle_interrupt' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:181:in `process’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:86:in `process_one' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:76:in `run’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/component.rb:10:in `watchdog' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/component.rb:19:in `block in safe\_thread’
> [2025-05-20 07:34:28] Deleting old backups…
> [2025-05-20 07:34:28] Cleaning stuff up…
> [2025-05-20 07:34:28] Removing archive from local storage…
> [2025-05-20 07:34:28] Removing ‘.tar’ leftovers…
> [2025-05-20 07:34:28] Marking backup as finished…
> [2025-05-20 07:34:28] Notifying ‘system’ of the end of the backup…
I have plenty of disk space (6X as much free space as the backup size).
CPU usage from today- the backup should take under an hour but when you have to try 9 times it takes four hours:
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [May 21, 2025, 2:42pm UTC](https://meta.discourse.org/t/backups-fail-to-upload-to-s3-several-times-works-eventually/366874/2 "2025-05-21T14:42:41Z")
> [@markschmucker](#):
>
> [2025-05-20 07:34:28] EXCEPTION: Sidekiq::Shutdown
Could you be running out of memory? That exception makes me think that Sidekiq (which runs the automatic backups) is either killed by the OS or crashes for some other reason.
Did you try rebuilding the Docker container (`./launcher rebuild app`) to see if that fixes it?
---
### Author:  [@markschmucker](https://meta.discourse.org/u/markschmucker)
#### Post date: [May 22, 2025, 11:35am UTC](https://meta.discourse.org/t/backups-fail-to-upload-to-s3-several-times-works-eventually/366874/3 "2025-05-22T11:35:19Z")
> [@gerhard](#):
>
> Could you be running out of memory?
I don’t think so. The server is overkill for our community with 16GB RAM. `top` shows 11GB free memory, and that hardly changes if I manually trigger a backup.
I’ll check `/var/log/syslog` tomorrow for anything about memory, kill, or OOM. (Can’t check today because I had some unrelated verbose logging and the backup event scrolls out of the syslog buffer.)
> [@gerhard](#):
>
> Did you try rebuilding the Docker container
Yes, I updated to 3.5.0.beta5-dev a few days ago and the issue persists.
---
### Author:  [@markschmucker](https://meta.discourse.org/u/markschmucker)
#### Post date: [May 22, 2025, 10:25pm UTC](https://meta.discourse.org/t/backups-fail-to-upload-to-s3-several-times-works-eventually/366874/4 "2025-05-22T22:25:27Z")
I do have this in /logs. This particular warning did not coincide with the backup- I’ll check for that in the morning (backups are done at night). But I didn’t realize Discourse its own memory check- I thought you were referring to the OOM killer. Can I increase the allowed memory size for Sidekiq?
> Message
>
> Sidekiq is consuming too much memory (using: 547.87M) for ‘ip-172-26-9-xxx-app’, restarting
>
> Backtrace
>
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active\_support/broadcast\_logger.rb:130:in `block in warn' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/broadcast_logger.rb:231:in `block in dispatch’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active\_support/broadcast\_logger.rb:231:in `each' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/broadcast_logger.rb:231:in `dispatch’
> /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active\_support/broadcast\_logger.rb:130:in `warn' /var/www/discourse/lib/demon/sidekiq.rb:59:in `block in rss\_memory\_check’
> /var/www/discourse/lib/demon/sidekiq.rb:53:in `each' /var/www/discourse/lib/demon/sidekiq.rb:53:in `rss\_memory\_check’
> config/unicorn.conf.rb:132:in `block (2 levels) in reload’
EDIT: I see this:
> [@Sidekiq is consuming too much memory, restarting](https://meta.discourse.org/t/sidekiq-is-consuming-too-much-memory-restarting/48395/38):
>
> Also, if your server has enough total RAM, you can increase the memory allocation for Sidekiq. Place this in the env: section of app.yml: UNICORN\_SIDEKIQ\_MAX\_RSS: 1000 The default maximum is, I believe, 500mb.
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [May 26, 2025, 5:50pm UTC](https://meta.discourse.org/t/backups-fail-to-upload-to-s3-several-times-works-eventually/366874/5 "2025-05-26T17:50:18Z")
Indeed, I was referring to the OOM killer. I completely forgot that there’s a memory limit for Sidekiq. Did increasing memory for Sidekiq help?
---
### Author:  [@markschmucker](https://meta.discourse.org/u/markschmucker)
#### Post date: [May 26, 2025, 8:26pm UTC](https://meta.discourse.org/t/backups-fail-to-upload-to-s3-several-times-works-eventually/366874/6 "2025-05-26T20:26:01Z")
I believe that fixed it- want to see a couple more clean nightly backups to be sure. Will post here.
---
### Author:  [@markschmucker](https://meta.discourse.org/u/markschmucker)
#### Post date: [May 27, 2025, 12:19pm UTC](https://meta.discourse.org/t/backups-fail-to-upload-to-s3-several-times-works-eventually/366874/7 "2025-05-27T12:19:55Z")
Yes, it’s fixed. You can close this topic.
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [May 27, 2025, 12:20pm UTC](https://meta.discourse.org/t/backups-fail-to-upload-to-s3-several-times-works-eventually/366874/8 "2025-05-27T12:20:26Z")