Disk space and backup failure notifications

I can confirm this still occurs…

And I believe I have a solution…

Background

Basically logging is unable to complete because the disk is full.

It is possible to allow logging to complete by deleting the intermediate files (*.tar) immediately after the error before attempting to log / send notifications for the event rather than leaving it until after.

Initial change (didn’t quite work)

Changing the following should delete the *.tar file before attempting to log / send notifications.

From

To

ensure
  clean_up

  begin
    notify_user
    remove_old
  rescue => ex
    Rails.logger.error("#{ex}\n" + ex.backtrace.join("\n"))
  end

  @success ? log("[SUCCESS]") : log("[FAILED]")

Further changes required

I tested the above on a production installation - it didn’t work…

… I tracked down the problem to the function remove_tar_leftovers:

Whilst system('rm', '-f', "#{@archive_directory}/*.tar") looks like it should delete the tar file it doesn’t actually delete the file in this case.

I also tested the following, none of which deleted the *.tar file:

#1
system('rm', '--verbose', '-f', "#{@archive_directory}/*.tar")

#2
stdout, stderr, status = Open3.capture3('rm', '--verbose', '-f', "#{@archive_directory}/*.tar")
log stdout
log stderr
log status

#3
FileUtils.rm_rf("#{@archive_directory}/*.tar")

Finally using the following actually deleted the file as expected when coupled with moving clean_up call earlier:

log %x[rm --verbose -f #{@archive_directory}/*.tar 2>&1].inspect

This actually allowed the notifications to be delivered.

Hope this helps nail down complete solution.

5 лайков