Backups to S3 have been working for several years. Starting a month ago, I often get several notifications that the backup failed. Then it retries, and about an hour later fails again, so I get a second notification, and often up to six notifications in one day before it finally succeeds. The log from the notification says it’s failing to upload the zipped tar file to S3. I can’t find any errors in my S3 account. Log The first part of the log looks normal, then: [2025-05-20 07:11:38] Finalizing backup… [2025-05-20 07:11:38] Creating archive: 506-investor-group-2025-05-20-070428-v20250513161753.tar.gz [2025-05-20 07:11:38] Making sure archive does not already exist… [2025-05-20 07:11:38] Creating empty archive… [2025-05-20 07:11:38] Archiving data dump… [2025-05-20 07:12:17] Archiving uploads… [2025-05-20 07:15:48] Removing tmp ‘/var/www/discourse/tmp/backups/default/2025-05-20-070428’ directory… [2025-05-20 07:15:48] Gzipping archive, this may take a while… [2025-05-20 07:32:51] Uploading archive… [2025-05-20 07:34:28] EXCEPTION: Sidekiq::Shutdown [2025-05-20 07:34:28] /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/multipart_file_uploader.rb:199:in value' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/multipart_file_uploader.rb:199:in map’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/multipart_file_uploader.rb:199:in upload_in_threads' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/multipart_file_uploader.rb:82:in upload_parts’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/multipart_file_uploader.rb:59:in upload' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/file_uploader.rb:42:in block in upload’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/user_agent.rb:69:in metric' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/file_uploader.rb:40:in upload’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/customizations/object.rb:477:in block in upload_file' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/user_agent.rb:69:in metric’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/customizations/object.rb:476:in upload_file' /var/www/discourse/lib/backup_restore/s3_backup_store.rb:48:in upload_file’ /var/www/discourse/lib/backup_restore/backuper.rb:351:in upload_archive' /var/www/discourse/lib/backup_restore/backuper.rb:41:in run’ /var/www/discourse/lib/backup_restore.rb:13:in backup!' /var/www/discourse/app/jobs/regular/create_backup.rb:10:in execute’ /var/www/discourse/app/jobs/base.rb:316:in block (2 levels) in perform' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:49:in with_connection’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management.rb:21:in with_connection' /var/www/discourse/app/jobs/base.rb:303:in block in perform’ /var/www/discourse/app/jobs/base.rb:299:in each' /var/www/discourse/app/jobs/base.rb:299:in perform’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:220:in execute_job' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:185:in block (4 levels) in process’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:180:in traverse' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in block in traverse’ /var/www/discourse/lib/sidekiq/discourse_event.rb:6:in call' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in traverse’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in block in traverse' /var/www/discourse/lib/sidekiq/pausable.rb:131:in call’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in traverse' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in block in traverse’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job/interrupt_handler.rb:9:in call' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in traverse’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:183:in block in traverse' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/metrics/tracking.rb:26:in track’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/metrics/tracking.rb:134:in call' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:182:in traverse’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/middleware/chain.rb:173:in invoke' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:184:in block (3 levels) in process’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:145:in block (6 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job_retry.rb:118:in local’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:144:in block (5 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/config.rb:39:in block in class:Config’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:139:in block (4 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:281:in stats’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:134:in block (3 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job_logger.rb:15:in call’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:133:in block (2 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job_retry.rb:85:in global’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:132:in block in dispatch' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/job_logger.rb:40:in prepare’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:131:in dispatch' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:183:in block (2 levels) in process’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:182:in handle_interrupt' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:182:in block in process’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:181:in handle_interrupt' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:181:in process’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:86:in process_one' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/processor.rb:76:in run’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/component.rb:10:in watchdog' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sidekiq-7.3.9/lib/sidekiq/component.rb:19:in block in safe_thread’ [2025-05-20 07:34:28] Deleting old backups… [2025-05-20 07:34:28] Cleaning stuff up… [2025-05-20 07:34:28] Removing archive from local storage… [2025-05-20 07:34:28] Removing ‘.tar’ leftovers… [2025-05-20 07:34:28] Marking backup as finished… [2025-05-20 07:34:28] Notifying ‘system’ of the end of the backup… I have plenty of disk space (6X as much free space as the backup size). CPU usage from today- the backup should take under an hour but when you have to try 9 times it takes four hours: [image]

[图片] markschmucker: [2025-05-20 07:34:28] EXCEPTION: Sidekiq::Shutdown Could you be running out of memory? That exception makes me think that Sidekiq (which runs the automatic backups) is either killed by the OS or crashes for some other reason. Did you try rebuilding the Docker container (./launcher rebuild app) to see if that fixes it?

[图片] gerhard: Could you be running out of memory? I don’t think so. The server is overkill for our community with 16GB RAM. top shows 11GB free memory, and that hardly changes if I manually trigger a backup. I’ll check /var/log/syslog tomorrow for anything about memory, kill, or OOM. (Can’t check today because I had some unrelated verbose logging and the backup event scrolls out of the syslog buffer.) [图片] gerhard: Did you try rebuilding the Docker container Yes, I updated to 3.5.0.beta5-dev a few days ago and the issue persists.

I do have this in /logs. This particular warning did not coincide with the backup- I’ll check for that in the morning (backups are done at night). But I didn’t realize Discourse its own memory check- I thought you were referring to the OOM killer. Can I increase the allowed memory size for Sidekiq? Message Sidekiq is consuming too much memory (using: 547.87M) for ‘ip-172-26-9-xxx-app’, restarting Backtrace /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/broadcast_logger.rb:130:in block in warn' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/broadcast_logger.rb:231:in block in dispatch’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/broadcast_logger.rb:231:in each' /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/broadcast_logger.rb:231:in dispatch’ /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/broadcast_logger.rb:130:in warn' /var/www/discourse/lib/demon/sidekiq.rb:59:in block in rss_memory_check’ /var/www/discourse/lib/demon/sidekiq.rb:53:in each' /var/www/discourse/lib/demon/sidekiq.rb:53:in rss_memory_check’ config/unicorn.conf.rb:132:in `block (2 levels) in reload’ EDIT: I see this: [图片] Sidekiq is consuming too much memory, restarting Self-hosting Also, if your server has enough total RAM, you can increase the memory allocation for Sidekiq. Place this in the env: section of app.yml: UNICORN_SIDEKIQ_MAX_RSS: 1000 The default maximum is, I believe, 500mb.

Indeed, I was referring to the OOM killer. I completely forgot that there’s a memory limit for Sidekiq. Did increasing memory for Sidekiq help?

I believe that fixed it- want to see a couple more clean nightly backups to be sure. Will post here.

Yes, it’s fixed. You can close this topic.

备份多次未能上传到S3，最终成功

产品支持

gerhard (Gerhard Schlager) 2025 年5 月 26 日 17:50 5

事实上，我指的是OOM杀手。我完全忘记了Sidekiq是有内存限制的。增加Sidekiq的内存有帮助吗？

话题		回复	浏览量
Backup notifies failure on sidekiq restart, but backup succeeds Product Support	0	72	2025 年9 月 23 日
Automatic backups are a hit or miss Product Support	11	1589	2018 年7 月 8 日
Scheduled backup fails when Sidekiq is forcefully restarted Product Support	4	976	2018 年4 月 3 日
Sidekiq is being paused, how can I discover why? Product Support	17	3495	2018 年9 月 19 日
Sidekiq is consuming too much memory, restarting Self-hosting	38	9537	2020 年10 月 13 日

备份多次未能上传到S3，最终成功

相关话题