It actually didn’t delete the temporary backup file in all cases and it didn’t log an error when the deletion failed. I made some changes so that it always tries to delete the backup file and if it fails you will see it in /logs.
https://github.com/discourse/discourse/commit/c94a2bc69b9fcbc39435066a35f0263390a8257c