peternlewis reported that automatic Discourse backups stopped working around early March, despite no configuration changes — though a Discourse upgrade likely occurred. Manual backups succeed but produce .sql.gz files (expected only if uploads are on S3, which they aren’t — they’re local). Automatic backups fail silently, with no logs or alerts, though manual backups also disappear — later revealed to be due to an external cleanup script. After checking system messages, peternlewis found backup failure logs indicating “No space left on device” during gzip compression. Running ./launcher cleanup freed 15GB, resolving the space issue. Key unresolved questions included: why no alerts were received (explained by gerhard as a bug fixed in recent betas), and why manual backups appeared as .sql.gz — clarified as likely due to accidentally omitting uploads during manual backup creation. peternlewis confirmed understanding and planned to verify backup settings. The issue was ultimately resolved by freeing disk space and understanding the backup behavior and alerting system.
Sometime around the start of March my Discourse backups have failed to happen.
The settings weren’t changed, though there may have been (probably was) a Discourse upgrade.
The backups are done to local storage, and other than settings the maximum backups to 3 and the backup frequency to 3, the remainder of the settings are at defaults.
I can do a manual backup, and that works, although I note when doing that that the backup is now an sql.gz, where previously it was a tar.gz file.
But the automatic backups do not run, appear to run, and I get no reports of errors. Backups shows “No backup available.” (including the manual backup I did a few days ago, which is now gone), and Backups ➤ Logs says “No logs yet…”.
I tried restarting the server (the entire digital ocean Droplet) and that did not result in it restarting backups.
Version says I am up to date with 2.6.3 (I run the stable branch, not the latest branch).
That’s to be expected as long as all your uploads are stored on S3. In that case the backup file is only a compressed database dump.
Please take a look at /logs. Are there any errors that might be related to backups?
Also, please check /u/system/messages/sent if the system sent any PMs about failed backups. There was a bug that prevented admins from getting notifications about those PMs under some circumstances.
It’s a little bit concerning that the manual backup is gone as well. I’m not sure how that could happen since you set maximum_backups to 3. Did you set up some kind of cronjob or script that moves/deletes backups or doesn’t work with the new file extension (sql.gz instead of tar.gz)?
Are they backup files really not there? What does ls /var/discourse/shared/standalone/backups/default say?
That’s normal. Logs only shows the output of manual backups.
I just checked, there weren’t any relevant changes to the backup code since the release of 2.6.
None of my uploads are stored on S3, they are all stored locally.
Not that I can see. I get the usual mess of TypeErrors, but I’ve always seen them. Searching for “backup” does not find anything except TypeErrors related to web pages for forum topics that happen to have the word “backup” in them.
Oh look, there are a bunch of “Backup failed” messages - why am I not being alerted to these?
OK, well, I don’t know why I never get them as a report, but that gives the error:
[2021-04-08 03:35:35] Removing tmp ‘/var/www/discourse/tmp/backups/default/2021-04-08-033313’ directory…
[2021-04-08 03:35:35] Gzipping archive, this may take a while…
[2021-04-08 03:38:49] EXCEPTION: lib/discourse.rb:92:in `exec’: Failed to gzip archive.
gzip: /var/www/discourse/public/backups/default/keyboard-maestro-discourse-2021-04-08-033313-v20201116132948.tar.gz: No space left on device
Actually, that is an external script - it copies the backup off to another machine for permanent storage and deletes it to leave space on the server (though apparently not enough space).
So remaining questions:
Why don’t I get any alert to these error messages?
Why is the manual backup a .sql.gz file when the actual backup is a .tar.gz file (both should have been including uploads, which are local)?
Why does the manual backup work but the automatic backup run out of space (probably related, maybe the manual backup is not actually including the uploads, and thus is smaller, though it looks to be about the right size)?
and the obvious one - I wonder how I am going to create some extra room on the server? [Edit: ok, running ./launcher cleanup released 15GB of space which should make the difference].
OK, at least now I know what the issue is, I’d really like to know the answer to the first one since I should obviously be hearing about this. Who are they emailed to or who is alerted to these messages? How can I make sure I learn about them in the future?