Disk usage spike during backup, Discourse crashed hard :-(

You have spikes to almost 100% every night before — looks like this one tipped you over the edge. I expect the previous backups failed due to out-of-space while creating the local backup file to send to S3, but merely failed, and didn’t break your forum. You finally noticed when the out-of-space made postgresql unhappy (or redis, or whatever, it’s not really important) at just the right time to bring your forum down.

(With nearly 100GB of images on my server, I do Discourse scheduled backups without uploads, but with thumbnails. Then I do an offsite file-based offset backup of the backups directory first and the uploads directory second. I have tested this for recovery; it was the basis of a site migration I did last year. Storing 100GB tarballs every night would be crazy.)