Disk usage spike during backup, Discourse crashed hard :-(

[riking (Kane York)]

@pnoeric

Here’s my 7-day disk usage. It was climbing steadily from the images being imported, but you can see where it slammed up to 100% at 5:30am:

Yep, that’s obviously a backup.

@DBHacker

Not to rain on this parade, but actually, from reading the posts, there is no solid confirmation that the Discourse backup process is causing the problem.

Why not confirm 100% that this problem is caused by a daily backup process? There are more than one processes running daily crontab files on hosts.

Nah, there’s plenty of confirmation: spikes are on a 2 day interval with one exception, and the backup frequency is set to 2 days. Past experience on Meta has shown exactly this failure mode, too.

@mcdanlj

(With nearly 100GB of images on my server, I do Discourse scheduled backups without uploads, but with thumbnails. Then I do an offsite file-based offset backup of the backups directory first and the uploads directory second. I have tested this for recovery; it was the basis of a site migration I did last year. Storing 100GB tarballs every night would be crazy.)

Yup, this is a solid plan for moving forwards. The first recommendation for people who start to hit the disk space limit on their VPS is off-machine upload storage with the S3 mechanism, though.