Configure automatic backups for Discourse


(Régis Hanol) #37

There you go :hamburger:


(Sam) #38

@zogstrip ~ could you provide some guidance on how the backup_time_of_day setting works? Leaving it at the default of 3:30 seems to keep backups running at a little after 10:00 PM EST, which is a bit difficult to interpret.

Just a quick clarification of setting to time zone or some such would work. I’ll figure it out through trial and error if necessary, but a little explanatory text might be helpful for the setting.

Actually - is this correct: the ScheduleBackup job runs at about 7PM EST for me, so the time_of_day setting is just offset from this, so +3.5hrs goes to 10:30PMish? Just noticed that, thought it would be a weird coincidence if that’s not the case, but still could be I suppose.


(Jeff Atwood) #39

It’s most likely UTC. The description should say that, I will add.


(Sam) #40

That sounds more likely!

Thanks. :smile:


(Chris Frederick) #41

I followed these instructions to automatically upload backups to Amazon S3 a few days ago, but nothing is appearing in the bucket I specified. Where should I start looking to debug the problem? I don’t see anything in the error logs.


(Dan Dascalescu) #42

I’ve followed the instructions here and in the Setting up file and image uploads to S3 (with the addendum that that post needs to be updated to mention inline policies vs. the old Attach Policy -> Custom Policy flow) post but I get this error when performing a backup:

EXCEPTION: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

On the S3 side, I had bucket logs enabled, and the log entries show this:

9d9… qsforum [17/Jan/2016:02:40:41 +0000] 54.239.6.71 9d9… D22… REST.GET.NOTIFICATION - “GET /?notification HTTP/1.1” 200 - 115 - 19 - “-” “aws-internal/3” -
9d9… qsforum [17/Jan/2016:02:40:42 +0000] 54.239.6.71 9d9… D2F… REST.GET.TAGGING - “GET /qsforum?tagging HTTP/1.1” 404 NoSuchTagSet 287 - 114 - “-” “S3Console/0.4” -

9d9… qsforum [17/Jan/2016:02:40:41 +0000] 54.239.6.36 9d9… 59E… REST.GET.REPLICATION - “GET /?replication HTTP/1.1” 404 ReplicationConfigurationNotFoundError 330 - 16 - “-” “aws-internal/3” -
9d9… qsforum [17/Jan/2016:02:40:42 +0000] 54.239.6.9 9d9… 09C… REST.GET.CORS - “GET /qsforum?cors HTTP/1.1” 404 NoSuchCORSConfiguration 310 - 5 - “-” “S3Console/0.4” -
9d9… qsforum [17/Jan/2016:02:40:42 +0000] 54.239.6.9 9d9… 899… REST.GET.BUCKETPOLICY - “GET /qsforum?policy HTTP/1.1” 404 NoSuchBucketPolicy 300 - 20 - “-” “S3Console/0.4” -

Permissions look like this:

My forum stores images locally, so I haven’t gone through the setup to store images to S3. Maybe the instructions for backing up to S3 assume image uploads go to S3 as well?

I’m also confused by the s3 region setting in /admin/site_settings/category/files - the setting’s description is The Amazon S3 region name that will be used to upload images., but does it also apply to backups, not just images? On Amazon, I’ve set the region to Frankfurt. I couldn’t find a precise matching region in the Discourse settings (there’s eu-west-1).

Anyway, why would the region be specified in Discourse, when it’s already specified on Amazon for the S3 bucket? I’m new to S3 and find the setup oddly complicated.

After resetting the region to default (us-east-1), I get a blank EXCEPTION

[2016-01-17 04:08:59] EXCEPTION: 
[2016-01-17 04:08:59] /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/aws-sdk-resources-2.2.9/lib/aws-sdk-resources/resource.rb:134:in `rescue in exists?'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/aws-sdk-resources-2.2.9/lib/aws-sdk-resources/resource.rb:131:in `exists?'
/var/www/discourse/lib/s3_helper.rb:64:in `s3_bucket'
/var/www/discourse/lib/s3_helper.rb:15:in `upload'
/var/www/discourse/app/models/backup.rb:54:in `block in upload_to_s3'
/var/www/discourse/app/models/backup.rb:53:in `open'
/var/www/discourse/app/models/backup.rb:53:in `upload_to_s3'
/var/www/discourse/app/models/backup.rb:33:in `after_create_hook'
/var/www/discourse/lib/backup_restore/backuper.rb:267:in `after_create_hook'
/var/www/discourse/lib/backup_restore/backuper.rb:49:in `run'
/var/www/discourse/lib/backup_restore/backup_restore.rb:161:in `block in start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `fork'
/var/www/discourse/lib/backup_restore/backup_restore.rb:158:in `start!'
/var/www/discourse/lib/backup_restore/backup_restore.rb:13:in `backup!'
/var/www/discourse/app/controllers/admin/backups_controller.rb:31:in `create'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/abstract_controller/base.rb:198:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_controller/metal/rendering.rb:10:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5/lib/active_support/callbacks.rb:117:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5/lib/active_support/callbacks.rb:117:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5/lib/active_support/callbacks.rb:505:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5/lib/active_support/callbacks.rb:505:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/abstract_controller/callbacks.rb:19:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_controller/metal/rescue.rb:29:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5/lib/active_support/notifications.rb:164:in `block in instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5/lib/active_support/notifications.rb:164:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.5/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/abstract_controller/base.rb:137:in `process'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionview-4.2.5/lib/action_view/rendering.rb:30:in `process'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.8/lib/mini_profiler/profiling_methods.rb:77:in `block in profile_method'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_controller/metal.rb:196:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_controller/metal.rb:237:in `block in action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_dispatch/routing/route_set.rb:76:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_dispatch/routing/route_set.rb:76:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_dispatch/routing/route_set.rb:45:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_dispatch/routing/mapper.rb:49:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_dispatch/journey/router.rb:43:in `block in serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_dispatch/journey/router.rb:30:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_dispatch/journey/router.rb:30:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_dispatch/routing/route_set.rb:817:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/builder.rb:63:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/conditionalget.rb:38:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
/var/www/discourse/lib/middleware/anonymous_cache.rb:129:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_dispatch/middleware/flash.rb:260:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:225:in `context'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:220:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_dispatch/middleware/cookies.rb:560:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.5/lib/active_record/query_cache.rb:36:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5/lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-1.0.1/lib/logster/middleware/reporter.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5/lib/rails/rack/logger.rb:38:in `call_app'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5/lib/rails/rack/logger.rb:22:in `call'
/var/www/discourse/config/initializers/100-quiet_logger.rb:10:in `call_with_quiet_assets'
/var/www/discourse/config/initializers/100-silence_logger.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5/lib/action_dispatch/middleware/request_id.rb:21:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/methodoverride.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/runtime.rb:18:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/sendfile.rb:113:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.8/lib/mini_profiler/profiler.rb:170:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/message_bus-2.0.0.beta.2/lib/message_bus/rack/middleware.rb:60:in `call'
/var/www/discourse/lib/middleware/request_tracker.rb:73:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5/lib/rails/engine.rb:518:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5/lib/rails/application.rb:165:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5/lib/rails/railtie.rb:194:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5/lib/rails/railtie.rb:194:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:562:in `process_client'
/var/www/discourse/lib/scheduler/defer.rb:85:in `process_client'
/var/www/discourse/lib/middleware/unicorn_oobgc.rb:95:in `process_client'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:658:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:132:in `start'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/bin/unicorn:126:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `<main>'

Looks like a region mismatch issue?


S3 region vs. Discourse region
Setting up file and image uploads to S3
(Dean Taylor) #43

You can find a list of Amazon’s regions here:
http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

EU (Frankfurt) is listed as eu-central-1.


(FOSS dev/hacker) #44

Setting this up can be rather confusing. Here’s a simple guide to help you out.

  • Log into your Discourse admin panel
  • Configure daily backups
  • Set maximum backups to 7
  • Log into your Amazon Web Services account
  • Go in the S3 Dashboard
  • Open the bucket containing the backups
  • Click on the properties tab
  • Activate versioning
  • Open the Lifecycle menu
  • Add a rule for the whole bucket
  • Set current version to expire after 15 days
  • Set previous version to
  • Archive to Glacier after 1 days
  • expire after 91 days
  • Save and logout

How it works

Versioning will keep backups automaticly deleted by Discourse. One day after beeing deleted it will be moved to the Glacier storage. After 91 days it will be delete from the Glacier storage.

Warning

Amazon charge you for item stored in Glacier for 90 days even if you delete them before. Make sure your Glacier Lyfecicle keep your file at least 90 days.


Install Discourse on Amazon WS with Cloudflare
(Steve) #45

Worth noting that Glacier storage costs are the cheap bit.


(Marco Jakobs) #46

@codinghorror: Did you ever think about adding the option to upload the automatic backup via FTP? Personally my server in the datacenter are sending their backup to my server at home via FTP, so I have anything on site. But no talking about me, I think FTP is the most common way to transfer (backup) files :grin:


(Jim DeLaHunt) #47

Sometimes there will be upgrades to the Discourse software. What guarantees does the software make about forward compatibility of backups when restored to newer versions of Discourse?

That is, if I backup on Monday, upgrade Discourse on Tuesday, and attempt to restore Monday’s backup on Wednesday, what does Discourse guarantee about whether this works?


(Chris Saenz) #48

What do you want, your money back?


(Jim DeLaHunt) #49

Yeah! I’ll be rich!

No, I meant “guarantee” in the software engineering sense, rather than the commercial sense. It might be that the Discourse software is designed so that there is a format for backed-up data which is deliberately kept stable across Discourse software versions; that the backup function guarantees that what it writes will conform to this format, across versions; and that the restore function guarantees that if data is in this format, it will be restored correctly, across versions.

Or it might be that each version of Discourse software has a restore function that guarantees it can correctly restore data only from that same version of backup function, with no guarantees across versions.


(Jeff Atwood) #50

I think the way it works is that migrations are performed on older DB versions to bring them up to date. Since every db change is a migration, tracked in source control, etc.


(Kane York) #51

Yep: The behavior is, you can restore any older backup. Not backups created on a newer version of the database, though.

@jdlh


(Erlend Sogge Heggen) #52

I’ve always been a bit curious about this, since one of our migrations guides is pretty adamant about Discourse instances needing to be on the exact same version:

@techAPJ Should we ease down on the language here a little bit? Sounds like the most important thing is that the new instance is on a more recent or same version as the old instance.


(Jeff Atwood) #53

But there’s no reason for them not to be at the same version. If you control updates on both, and are using a Docker-based install, and can update easily, the upgrade is far easier if you keep them at the same version.


(Régis Hanol) #54

You can definitely replace the must with a should :wink: But I agree with @codinghorror, there’s no harm in upgrading your old instance first to the latest version (it’s just two clicks and some waiting) before doing a backup.


(Steven Greco) #55

I guess the senario would be that the current environment cannot be updated to the lastest because it has failed either via hardware or software issues. Now the lastest backup you have is on a previous version than that of the lastest.


(Michael Wransky) #56

@codinghorror

I am trying to configure automatic backups for our Discourse account (we are under the Enterprise payment option). When I go to Admin -> Settings -> Backups all I see is this:

Why is it that I see no option to configure automatic backups? Is it because we have Discourse acting as our server provider?

We seriously need access to configuring this so we can set up an automated data pipeline to Chartio for data analysis.