This morning around 5:35am, my forums suddenly spiked on disk usage and crashed, going completely offline. I had to resize the Digital Ocean image to get them back on their feet. Oof. Here’s my disk usage over the last 24 hours: [image] Question: what kind of logs/post-mortem analysis can I look…

Yeah, backups are a common source of this kind of failure - what’s the disk usage look like over a 7 day window? Also note that local uploads are included in these backups, so if you had a significant increase in uploads around 18:00, that would also increase the backup archive size.

Hmm. I have been transitioning files off of S3 and back to my local server , but the process to do that seems to run on the fly, and only a few hundred images (all around ~300k each) at a time = ~0.1 GB per batch. Over the last week, I might have run the script 20 times, so 20 batches = around 2 GB t…

Hmmm. My backups are set to go to S3, every 2 days, but nothing since the 9th? [image] Discourse ‘Backups’ view [image] Amazon S3 view [image] Incidentally, after seeing the above, I clicked the button in Discourse to trigger a backup. It took 28 minutes and seemed to work fine; I now see tha…

[immagine] bartv: I’d start with figuring out which directory is blowing up. My standard approach is to enter /var/discourse and then run du -h -d 1 . Take the largest directory, enter it and repeat until you find the suspect. Once you have it, that might give you a clue to what’s going on. I…

You have spikes to almost 100% every night before — looks like this one tipped you over the edge. I expect the previous backups failed due to out-of-space while creating the local backup file to send to S3, but merely failed, and didn’t break your forum. You finally noticed when the out-of-space mad…

Aha, so those little spikes are Discourse trying to make a backup! That sheds some light on things. So here is my trailing 7-day chart again. [image] Maybe what we’re seeing is: Several times over this past week, Discourse attempted to make a backup. This process temporary eats a lot of disk …

It didn’t necessarily get further along last night; you just “lost a race” — what happens when you run out of space depends on what component first is affected by the problem. If it fails to make a backup, I rather expect it might try to send a message, but if it’s out of disk space, it might not s…

[immagine] Michael K Johnson: I don’t know how you are judging “inordinate” — it doesn’t strike me as inordinate. Fair enough; as you point out, there are many variables in play. [immagine] Michael K Johnson: A fixed percentage doesn’t really tell you much; the database might be tiny com…

Estimating space required for backup is one of the hard problems in computer science… It’s a distant relative of the progress bar. :wink: In all seriousness, part of it is a database dump, and I don’t know how you would estimate that ahead of time. If you have enough images that space becomes an is…

Picco di utilizzo del disco durante il backup, Discourse è crashato duramente :-(

Supporto Installazione

pnoeric (Eric) 23 Luglio 2020, 12:51pm 22

Grazie Bryan (e @Stephen), ho configurato il nuovo bucket esattamente come avete consigliato. Ora sto solo definendo le impostazioni corrette.

Argomento		Risposte	Visualizzazioni
Configure automatic backups for Discourse Self-Hosting how-to , backups	42	303217	Gennaio 25, 2024
I can't get a good backup - it fails after after_create_hook Bug	17	2843	Settembre 21, 2016
Set up BackBlaze S3 with BunnyCDN Sysadmins how-to	34	6695	Aprile 17, 2025
How to backup Discourse when The backup has failed? Self-hosting	8	993	Agosto 5, 2020
Backups are duplicating and not respecting number to keep on disk Self-hosting	68	2558	Febbraio 15, 2019

Picco di utilizzo del disco durante il backup, Discourse è crashato duramente :-(

Argomenti correlati