I can confirm this still occurs…
And I believe I have a solution…
Background
Basically logging is unable to complete because the disk is full.
It is possible to allow logging to complete by deleting the intermediate files (*.tar) immediately after the error before attempting to log / send notifications for the event rather than leaving it until after.
Initial change (didn’t quite work)
Changing the following should delete the *.tar file before attempting to log / send notifications.
From
To
ensure
clean_up
begin
notify_user
remove_old
rescue => ex
Rails.logger.error("#{ex}\n" + ex.backtrace.join("\n"))
end
@success ? log("[SUCCESS]") : log("[FAILED]")
Further changes required
I tested the above on a production installation - it didn’t work…
… I tracked down the problem to the function remove_tar_leftovers:
Whilst system('rm', '-f', "#{@archive_directory}/*.tar") looks like it should delete the tar file it doesn’t actually delete the file in this case.
I also tested the following, none of which deleted the *.tar file:
#1
system('rm', '--verbose', '-f', "#{@archive_directory}/*.tar")
#2
stdout, stderr, status = Open3.capture3('rm', '--verbose', '-f', "#{@archive_directory}/*.tar")
log stdout
log stderr
log status
#3
FileUtils.rm_rf("#{@archive_directory}/*.tar")
Finally using the following actually deleted the file as expected when coupled with moving clean_up call earlier:
log %x[rm --verbose -f #{@archive_directory}/*.tar 2>&1].inspect
This actually allowed the notifications to be delivered.
Hope this helps nail down complete solution.