Notification when uploading backup to cloud storage fails

Contribute Feature
,

AI-generated summary

The discussion revolves around a feature request to notify administrators when a backup upload to cloud storage fails, as the current system may fail silently. markschmucker highlights that failed uploads leave tar files on the local disk, potentially leading to disk space exhaustion and system downtime. Canapin confirms experiencing similar issues. gerhard explains that the backup process should clean up after itself, but failures can occur due to system interruptions like restarts, container crashes, or memory issues. The proposed solution is a background job that detects aborted backups, cleans up temporary files, and alerts admins, improving system reliability and user experience.

1

As discussed at the link below, if Discourse is configured to upload backups to cloud storage, the upload can fail silently, leaving the tar file on local disk. After a few such failures the forum goes down because there is no disk space. Request a notification when an upload fails, like there is when a backup fails.

Configure an S3 compatible object storage provider for uploads - documentation / sysadmin - Discourse Meta

2 Likes
2

I think I have encountered this issue a couple of times on my forum as well.

3

The backup process is supposed to clean up when it’s finished, even when there was a failure. But it’s possible that the OS kills the backup process before it’s able to do that cleanup. Reasons I can think of right now:

  • the host restarts (maybe because of OS upgrade)
  • the Docker container restarts (maybe because the Docker daemon is updated or crashed)
  • the system runs out of memory and kills processes
  • the process crashes for whatever reason and isn’t able to exit cleanly

I guess the feature request here is: A background job should check if a backup was aborted. In that case it should cleanup temporary artifacts of the backup and send a warning to the admin.

2 Likes