Backup fails if someone uploads during backup

AI-generated summary

A backup failure issue occurs when a user uploads a file during the backup process. The error message indicates that the tar command encounters a file that has changed while being read. A possible solution is to add the --warning=no-file-changed switch to the tar command, as suggested on Stack Overflow.

The issue arises because the site is only in read-only mode while the database is being dumped, and not during the compression of the backup, which captures the uploads. Although the window for an upload to occur during this process is small, the site’s high traffic makes it a possibility. A pull request has been created to implement the suggested solution here, and is awaiting review from @mpalmer and @tgxworld. codinghorror has expressed approval for merging the pull request.

[2017-06-07 18:12:36] Creating archive: asdf-2017-06-07-181158-v20170524182846.tar.gz
[2017-06-07 18:12:36] Making sure archive does not already exist...
[2017-06-07 18:12:36] Creating empty archive...
[2017-06-07 18:12:36] Archiving data dump...
[2017-06-07 18:12:36] Archiving uploads...
[2017-06-07 18:13:18] EXCEPTION: Failed to archive uploads.
tar: uploads/default/original/2X/c/c5df928d1883cc5f15d1dac6d95c0c0ae55f073e.jpeg: file changed as we read it
tar: uploads/default/original/2X/c: file changed as we read it
tar: uploads/default/original/2X: file changed as we read it
tar: uploads/default/original: file changed as we read it
tar: uploads/default: file changed as we read it

A site called “stack overflow” suggests
adding --warning=no-file-changed to the tar command.

Is it as simple as adding that switch here?

4 « J'aime »

How can a user upload a file while the site is RO during the backup?

1 « J'aime »

I just know that it happened. :slight_smile:

I think read only mode only kicks in while the database is being dumped and is disabled before the backup is compressed (which captures the uploads).

I can’t do squat from the web interface while our backup is running. If it were a sidekiq job doing something then surely someone would have seen it by now.

That’s right. Read-only is only on while the database is backing up. Creating the tar file could take a long while.

It’s fairly improbable that an upload would happen in this window, but this site got 2.6M page views in the past 30 days, with 6.9K posts yesterday.

3 « J'aime »

It looks like it to me.

https://github.com/discourse/discourse/pull/4907

5 « J'aime »

If this looks good to you @mpalmer and @tgxworld please merge

2 « J'aime »