This morning around 5:35am, my forums suddenly spiked on disk usage and crashed, going completely offline. I had to resize the Digital Ocean image to get them back on their feet. Oof. Here’s my disk usage over the last 24 hours: [image] Question: what kind of logs/post-mortem analysis can I look…

Yeah, backups are a common source of this kind of failure - what’s the disk usage look like over a 7 day window? Also note that local uploads are included in these backups, so if you had a significant increase in uploads around 18:00, that would also increase the backup archive size.

Hmm. I have been transitioning files off of S3 and back to my local server , but the process to do that seems to run on the fly, and only a few hundred images (all around ~300k each) at a time = ~0.1 GB per batch. Over the last week, I might have run the script 20 times, so 20 batches = around 2 GB t…

Hmmm. My backups are set to go to S3, every 2 days, but nothing since the 9th? [image] Discourse ‘Backups’ view [image] Amazon S3 view [image] Incidentally, after seeing the above, I clicked the button in Discourse to trigger a backup. It took 28 minutes and seemed to work fine; I now see tha…

[图片] bartv: I’d start with figuring out which directory is blowing up. My standard approach is to enter /var/discourse and then run du -h -d 1 . Take the largest directory, enter it and repeat until you find the suspect. Once you have it, that might give you a clue to what’s going on. I’m mys…

You have spikes to almost 100% every night before — looks like this one tipped you over the edge. I expect the previous backups failed due to out-of-space while creating the local backup file to send to S3, but merely failed, and didn’t break your forum. You finally noticed when the out-of-space mad…

Aha, so those little spikes are Discourse trying to make a backup! That sheds some light on things. So here is my trailing 7-day chart again. [image] Maybe what we’re seeing is: Several times over this past week, Discourse attempted to make a backup. This process temporary eats a lot of disk …

It didn’t necessarily get further along last night; you just “lost a race” — what happens when you run out of space depends on what component first is affected by the problem. If it fails to make a backup, I rather expect it might try to send a message, but if it’s out of disk space, it might not s…

[图片] Michael K Johnson: I don’t know how you are judging “inordinate” — it doesn’t strike me as inordinate. Fair enough; as you point out, there are many variables in play. [图片] Michael K Johnson: A fixed percentage doesn’t really tell you much; the database might be tiny compared to upl…

Estimating space required for backup is one of the hard problems in computer science… It’s a distant relative of the progress bar. :wink: In all seriousness, part of it is a database dump, and I don’t know how you would estimate that ahead of time. If you have enough images that space becomes an is…

备份期间磁盘使用率飙升，Discourse 严重崩溃 :-(

支持安装

pnoeric (Eric) 2020 年7 月 23 日 12:51 22

谢谢 Bryan（还有 @Stephen），我已经按照大家的建议设置好了新的存储桶。现在我正在调整合适的设置。

话题		回复	浏览量
Configure automatic backups for Discourse Self-Hosting how-to , backups	42	303217	2024 年1 月 25 日
I can't get a good backup - it fails after after_create_hook Bug	17	2843	2016 年9 月 21 日
Set up BackBlaze S3 with BunnyCDN Sysadmins how-to	34	6695	2025 年4 月 17 日
How to backup Discourse when The backup has failed? Self-hosting	8	993	2020 年8 月 5 日
Backups are duplicating and not respecting number to keep on disk Self-hosting	68	2558	2019 年2 月 15 日

备份期间磁盘使用率飙升，Discourse 严重崩溃 :-(

相关话题