Sometime around the start of March my Discourse backups have failed to happen.
The settings weren’t changed, though there may have been (probably was) a Discourse upgrade.
The backups are done to local storage, and other than settings the maximum backups to 3 and the backup frequency to 3, the remainder of the settings are at defaults.
I can do a manual backup, and that works, although I note when doing that that the backup is now an sql.gz, where previously it was a tar.gz file.
But the automatic backups do not run, appear to run, and I get no reports of errors. Backups shows “No backup available.” (including the manual backup I did a few days ago, which is now gone), and Backups ➤ Logs says “No logs yet…”.
I tried restarting the server (the entire digital ocean Droplet) and that did not result in it restarting backups.
Version says I am up to date with 2.6.3 (I run the stable branch, not the latest branch).
That’s to be expected as long as all your uploads are stored on S3. In that case the backup file is only a compressed database dump.
Please take a look at /logs. Are there any errors that might be related to backups?
Also, please check /u/system/messages/sent if the system sent any PMs about failed backups. There was a bug that prevented admins from getting notifications about those PMs under some circumstances.
It’s a little bit concerning that the manual backup is gone as well. I’m not sure how that could happen since you set maximum_backups to 3. Did you set up some kind of cronjob or script that moves/deletes backups or doesn’t work with the new file extension (sql.gz instead of tar.gz)?
Are they backup files really not there? What does ls /var/discourse/shared/standalone/backups/default say?
That’s normal. Logs only shows the output of manual backups.
I just checked, there weren’t any relevant changes to the backup code since the release of 2.6.
None of my uploads are stored on S3, they are all stored locally.
Not that I can see. I get the usual mess of TypeErrors, but I’ve always seen them. Searching for “backup” does not find anything except TypeErrors related to web pages for forum topics that happen to have the word “backup” in them.
Oh look, there are a bunch of “Backup failed” messages - why am I not being alerted to these?
OK, well, I don’t know why I never get them as a report, but that gives the error:
[2021-04-08 03:35:35] Removing tmp ‘/var/www/discourse/tmp/backups/default/2021-04-08-033313’ directory…
[2021-04-08 03:35:35] Gzipping archive, this may take a while…
[2021-04-08 03:38:49] EXCEPTION: lib/discourse.rb:92:in `exec’: Failed to gzip archive.
gzip: /var/www/discourse/public/backups/default/keyboard-maestro-discourse-2021-04-08-033313-v20201116132948.tar.gz: No space left on device
Actually, that is an external script - it copies the backup off to another machine for permanent storage and deletes it to leave space on the server (though apparently not enough space).
So remaining questions:
Why don’t I get any alert to these error messages?
Why is the manual backup a .sql.gz file when the actual backup is a .tar.gz file (both should have been including uploads, which are local)?
Why does the manual backup work but the automatic backup run out of space (probably related, maybe the manual backup is not actually including the uploads, and thus is smaller, though it looks to be about the right size)?
and the obvious one - I wonder how I am going to create some extra room on the server? [Edit: ok, running ./launcher cleanup released 15GB of space which should make the difference].
OK, at least now I know what the issue is, I’d really like to know the answer to the first one since I should obviously be hearing about this. Who are they emailed to or who is alerted to these messages? How can I make sure I learn about them in the future?
