Hot off the presses, automated backup support!


(Robin Ward) #1

If you are on the latest version of Discourse, you can now enable automated daily backups, with support for uploading them offsite to S3!

I’m super proud of this feature because in my opinion not enough software (and certainly no forum software I am aware of) encourages users to create offsite backups of their data. The last thing you want in the event of a server failure is to lose all the awesome content your forum users have created.

Enabling this feature is easy! First go to the admin section and find backups under site settings. You’ll see the following options:

Note that for the uploading to S3 support you’ll need to have defined your s3 credentials as well. If you don’t upload to S3 the backups will be left on the server filesystem.

Once you enable this, you can go to the backups tab in admin to view a list of backups that were performed with links to download or restore any images.

Let us know what you think of the feature and please help us try it out by enabling backups for your forums!


Where do the local backups go? (when S3 backups aren't enabled)
(Johan Jatko) #2

Might already be in the works, but will you extend this to, lets say, FTP backup? =)


Automated Backups to other Destinations Besides Amazon S3
(Robin Ward) #3

We have no plans to implement FTP but I purposely created a function called after_create_hook in Backup so that a plugin could easily be added to handle custom behaviour like FTP. I imagine a FTP plugin would not be very complicated.


(Claus Strasburger) #4

Wow! That’s a great feature.

Currently testing this, but somehow the upload is going wrong:

[2014-03-18 20:49:04] Executing the after_create_hook for the backup
[2014-03-18 20:49:14] EXCEPTION: Connection reset by peer (Errno::ECONNRESET)
[2014-03-18 20:49:14] /home/discourse/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/openssl/buffering.rb:375:in `syswrite_nonblock'
/home/discourse/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/openssl/buffering.rb:375:in `write_nonblock'
/home/discourse/discourse/vendor/bundle/ruby/2.0.0/gems/excon-0.28.0/lib/excon/socket.rb:91:in `write'
/home/discourse/discourse/vendor/bundle/ruby/2.0.0/gems/excon-0.28.0/lib/excon/ssl_socket.rb:98:in `write'
[... stack trace ...]
/home/discourse/discourse/app/models/backup.rb:41:in `upload_to_s3'
/home/discourse/discourse/app/models/backup.rb:32:in `after_create_hook'
/home/discourse/discourse/lib/export/exporter.rb:244:in `after_create_hook'
[... more trace ...]
[2014-03-18 20:49:14] Cleaning stuff up...
[2014-03-18 20:49:14] Removing tmp '/home/discourse/discourse/tmp/backups/default/2014-03-18-204859' directory...
[2014-03-18 20:49:14] Unpausing sidekiq...
[2014-03-18 20:49:14] Disabling readonly mode...
[2014-03-18 20:49:14] Marking backup as finished...
[2014-03-18 20:49:14] Finished!

Before this, I got 403 Forbidden, but that was due to wrong permissions in my bucket…

any Ideas?


(Robin Ward) #5

Hmmm that’s quite unusual. I actually have no idea what would cause that.

What region is your bucket in?


(Claus Strasburger) #6

I set it to eu-west (ireland).


(Antonin Hildebrand) #7

I hit another minor issue. I wanted to name my bucket discuss-backup, but because S3 buckets have global namespace this name was already taken. So I decided to append my domain name to it and created bucket named discuss-backup-discuss.binaryage.com.

And this is a mistake! after_create_hook fails with some SSL error when do you use a bucket name with dots in it.

[2014-03-18 22:00:25] Executing the after_create_hook for the backup
[2014-03-18 22:00:25] EXCEPTION: hostname "discuss-backup-discuss.binaryage.com.s3.amazonaws.com" does not match the server certificate (OpenSSL::SSL::SSLError)
[2014-03-18 22:00:25] /usr/local/lib/ruby/2.0.0/openssl/ssl.rb:139:in `post_connection_check'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/excon-0.28.0/lib/excon/ssl_socket.rb:85:in `initialize'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/excon-0.28.0/lib/excon/connection.rb:373:in `new'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/excon-0.28.0/lib/excon/connection.rb:373:in `socket'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/excon-0.28.0/lib/excon/connection.rb:122:in `request_call'
...

(Antonin Hildebrand) #8

Also one more note. When automated backup finishes successfully I get system message “Export completed successfully”. But when it fails I don’t get anything. Of course during manual backup execution, I get popup that backup failed and I should go and see the logs. But in case of automated backup I would appreciate to get system message with full log embedded, so I can inspect it right away without trying to reproduce the issue with manual backup (the issue may be no longer reproducible).


(Jeff Atwood) #9

I believe @zogstrip has run into bucket name with dots issues before, anything to add here? Agree that failed backups should include some sort of log attachment @eviltrout.


(Antonin Hildebrand) #10

@codinghorror I would just restrict the configuration field in settings to safe characters maybe with some explanation. There is no need for dots in names. AFAIK Amazon has some naming convention and requires them only if you want to host buckets on open web and map bucket name to your domain name (which should not be the case here).


(Régis Hanol) #11

Definitely related to bucket with dots in their name. They are not compatible with https…

Added that to my list.

While I agree this is nice to have, you can already see the last backup logs when you directly navigate to the /admin/backup/logs page.


(Antonin Hildebrand) #12

This does not work for me. I can see there only “No logs yet…”. This works for me only immediately after running a manual backup.


(Régis Hanol) #13

@darwin I just pushed a fix that will send the logs to the admin user when the backup/restore operation fails.

https://github.com/discourse/discourse/commit/0def4747da9e086e0dc0e0a4d1759fdac4bd9f1e


(William Gross) #14

Do the new automated backups include images/attachments or just the database? My Discourse forum is still using local file storage, not S3. I am trying to decide if I should start using S3 for images/attachments.


(Robin Ward) #15

The backup images include all uploaded files, right @zogstrip?

So if you enable the option your content will be created with your backup files.


(Régis Hanol) #16

Exactly. But only if you’re using the local storage.


(Jeff Atwood) #17

When you say “using the local storage” you mean “not using S3” right?


(Régis Hanol) #18

Right. RIght now, there’s 2 file stores: local store and S3 store.


(James Gifford) #19

So I have a question… would it be possible to have a hourly option? We’ve had a few situations where it would have been nice to have a hourly backup instead of a nightly backup, since we are on the master discourse instead of the releases.

Of course, I’d also accept “you’re being dumb by staying on the development branch” as a acceptable answer. :smile:


Hourly backup, only if something has changed
(probus) #20

I would like to see more frequent options as well, in the future. On another topic I suggested using WAL-E from Heroku for database backups. That would provide a simple way to set up continuous backups. WALs use little disk space and the system works on the background with no downtime taking advantage of the features of PostgreSQL database.