Amazon S3 large backups not uploading


(Dean Taylor) #1

My backups are quite large at 2.1GB.

Currently I have receiving the error EXCEPTION: failed to allocate memory

Smaller DB only backups at ~250MB upload and complete successfully.

##Backup Log
[2014-09-25 00:33:00] Executing the after_create_hook for the backup
[2014-09-25 00:33:00] EXCEPTION: failed to allocate memory
[2014-09-25 00:33:00] /var/www/discourse/app/models/backup.rb:56:in read' /var/www/discourse/app/models/backup.rb:56:inupload_to_s3’
/var/www/discourse/app/models/backup.rb:36:in after_create_hook' /var/www/discourse/lib/export/exporter.rb:267:inafter_create_hook’
/var/www/discourse/lib/export/exporter.rb:48:in `run’

##Possible Issue
To me it looks like this code loads the entire 2GB file into memory - the cause of the failed to allocate memory exception?


(Sam Saffron) #2

This looks really bad, @eviltrout I think an urgentish fix is in place here, it can cause some bad crashes.


(Régis Hanol) #3

I will fix it tomorrow morning since I refactored that code today :slight_smile:


(Régis Hanol) #4

Ok, this is now fixed :dog2:

https://github.com/discourse/discourse/commit/b16e6f828949b2175cf69eb75768228bf31d23ef


(Dean Taylor) #5

Thanks @zogstrip !!

I can confirm “Works For Me” status - successfully uploaded 2+GB file.

:thumbsup:


(Régis Hanol) #6