A backup issue has been reported where backups to S3 fail and the backup is stored locally, ignoring the backup limit. This issue has been experienced with Scaleway S3 (markersocial) and Wasabi S3 (pfaffman). The problem is that the local backup is not deleted after a failed upload, causing the disk to fill up indefinitely.
gerhard suggests using a different S3 provider or setting backup_location to “local” as a solution. However, markersocial explains that switching providers would require a full migration of the main storage, and they are waiting for Scaleway to roll out a fix.
It is also noted that the backup log shows the message “Removing archive from local storage…” even though the local backup is not deleted. This issue has been reported on Discourse 2.6.4 stable.
Multipart upload backups to Scaleway S3 aren’t working for me still. Backup Upload to S3 Fails on scaleway (multipart upload)
The problem is, if the upload fails, the backup remains stored locally like when doing normal local backups. Which is good. However the backup limit is ignored, so it will keep filling up the local disk indefinitely with backups until it causes downtime.
This is on 2.6.4 stable.
I’ve had a problem with one site using wasabi S3 that kept filling the disk. Neither discourse nor wasabi showed any errors in the logs.
I can’t remember now if I moved to a different S3 provider or if it’s been working. This is is up to date.
I don’t get it. Why are you trying to store backups on Scaleway S3 when you know that it doesn’t work? I think using a different S3 provider or setting the backup_location to “local” would the the best solution.
Anyway, Discourse should delete the local backup if the upload fails. You should see the message “Removing archive from local storage…” near the end of the backup log.
This wasn’t known to me when migrating from local to S3 storage. It isn’t possible to use a different S3 provider for backups than the main storage, meaning a full migration of the main storage would need to be done to use an alternative provider for backups.
Scaleway may be rolling out a fix (can see in the post I linked to), so rather than needing to rebuild several times with downtime just to check if it is working, I’ve been letting it try to succeed on a weekly basis, seeing that the backups that don’t successfully upload are kept in the local backups folder anyway.
It doesn’t, as I stated in my post. IIRC the message saying it will be removed still appears in the log, but that is not what happens in reality. If anyone is interested in understanding this edge case bug, I can check/confirm.