Setting up backup and image uploads to DigitalOcean Spaces


(Rishabh Nambiar) #1

do

Introducing DigitalOcean Spaces for Discourse

  • Spaces is available for a simple $5 per month price and includes 250GB of storage and 1TB of outbound bandwidth. There are no costs per request and additional storage is priced at the lowest rate available: $0.01 per GB transferred and $0.02 per GB stored. Uploads are free. Spaces provides cost savings of up to 10x along with predictable pricing and no surprises on your monthly bill. Read more here: Introducing Spaces Object Storage

  • If you’re already running Discourse on a DigitalOcean VPS, you could move your uploads/backups to Spaces to keep everything under a single platform and billing mechanism.

Step 1: Configuring DigitalOcean Spaces

Creating an Account

Go to Spaces on DigitalOcean - Beautifully simple object storage and click on Try Free for 2 months. If you already have a DigitalOcean account, log in and visit https://cloud.digitalocean.com/spaces.

Generating API Keys

Visit https://cloud.digitalocean.com/settings/api/ and generate a new key in the Spaces Access keys section. Store this key pair safely as you will not be able to view it again. This key pair is the Access Key ID and your Secret Access Key for accessing all your Spaces (buckets) on DigitalOcean.

Step 2: Configuring Discourse

After setting up DigitalOcean keys, the final step is to configure your Discourse instance. Make sure you’re logged in with an administrator account and go the Settings section in the admin panel.

Type in “s3” in the textbox on the top-left to display only the relevant settings:

You will need to:

  • Check the “enable s3 backups” checkbox if you want to activate manual or automated backups
    • Enter the desired Space name (bucket) in “s3 backup bucket” if enable s3 backups is checked
  • Check the “enable s3 uploads” checkbox if you want to allow images to be uploaded and served by DigitalOcean Spaces
    • Enter the desired Space name (bucket) in “s3 upload bucket” if enable s3 uploads is checked
  • Paste in both “Access Key ID” and “Secret Access Key” in their respective text fields
  • In s3_endpoint, paste in one of the below endpoints to select your preferred region:
    • https://nyc3.digitaloceanspaces.com [New York]
    • https://ams3.digitaloceanspaces.com [Amsterdam]
    • https://sgp1.digitaloceanspaces.com [Singapore]
  • In s3_cdn_url paste the equivalente Digital Spaces CDN, eg: https://sfo2.cdn.digitaloceanspaces.com

Discourse will automatically create the Spaces (buckets) required for uploads/backups.

Note: If you’d like to configure the Space yourself using the website instead of Discourse, create a Space at https://cloud.digitalocean.com/spaces and copy the endpoint from the Settings pane of your newly created Space into s3 endpoint.

In the case of DigitalOcean, The “region” setting is overridden by the region provided in the endpoint you pasted earlier, for example: https://nyc3.digitaloceanspaces.com

What your settings should look like for DigitalOcean:
(Admin -> Settings -> Type ‘s3’ in filter)

Note: You can enable only backups or only uploads or both.

Enjoy

That’s it. From now on, your images or backups will be uploaded to and served from DigitalOcean :tada:


Which one use at DO for discourse backup?
Minio instead of S3?
Extend S3 configuration for other S3 API compatible services
(@SenpaiMass) #2

Thanks for this awesome guide, i just wanted to know if there is a way to move all the previous image uploads to spaces?


(Rishabh Nambiar) #3

As of now, it isn’t possible.
Currently, there is support for migrating only local images to an external provider using the migrate_to_s3 task from uploads.rake.
I’ll check and let you know if I find a good way to migrate uploads from one external provider to another. :+1:


#4

Will images uploaded for a theme be stored on spaces if uploads are enabled? It seems like those might be better stored on the local server to remove the latency of retrieving the files on a regular basis.


(Jay Pfaffman) #5

I would like to be able to configure S3 backups to Digital Ocean using environment variables.

I’m surprised that s3_access_key_id and s3_secret_access_key aren’t already shadowed, as I thought that was one of the main uses–to be able to put some settings where an admin (who doesn’t have access to the server) can’t see them.

@sam, Would a PR that enables shadow access for the following be acceptable? Is there some other way I should do that, a plugin, maybe? If so, is there a convenient example of a plugin that removes a system setting from the web interface?

s3_access_key_id
s3_secret_access_key
s3_upload_bucket
s3_endpoint
enable_s3_backups
s3_backup_bucket

(Sam Saffron) #6

There are globals for them … it stronger than a simple “shadow by global” as it integrates into a bunch of other spots:

I guess we can add:

I don’t mind if there is a shadow for:

enable_s3_backups
s3_backup_bucket

(Jay Pfaffman) #7

I’d never noticed that before. Is the recommended way to write to discourse.conf to use an after_code hook that appends it? (Well, it appears not; either there or in run: has no effect on discourse.conf, but putting them in environment variables does. . .)

I’ve put this in app.yml

  DISCOURSE_S3_ACCESS_KEY_ID: 'key'
  DISCOURSE_S3_SECRET_ACCESS_KEY: 'secret'
  DISCOURSE_S3_UPLOAD_BUCKET: 'my-bucket'
  DISCOURSE_S3_ENDPOINT: 'https://nyc3.digitaloceanspaces.com'
  DISCOURSE_ENABLE_S3_BACKUPS: true
  DISCOURSE_S3_BACKUP_BUCKET: 'lc-backups

And discourse.conf has

smtp_user_name = 'postmaster@s3tester.ltcmp.net'
s3_endpoint = 'https://nyc3.digitaloceanspaces.com'
db_host = ''
developer_emails = 'jay@literatecomputing.com'
smtp_port = '2525'
smtp_password = 'redacted'
smtp_address = 'smtp.mailgun.org'
hostname = 's3tester.ltcmp.net'
s3_secret_access_key = 'my secret'
s3_upload_bucket = 'lc-backups'
db_port = ''
db_socket = '/var/run/postgresql'
enable_s3_backups = 'true'
s3_backup_bucket = 'lc-backups'
s3_access_key_id = 'my id'

enable s3 backups does not appear in the system settings, which is surprising, because I don’t see it in discourse_defaults.conf, but the rest of the S3 stuff is in system settings.

When I add shadowed_by_global: true to s3_backup_bucket that does what I expect, but adding shadowed to s3_access_key_id` makes Discourse not crank up.

There’s clearly something that I just don’t understand here.


(Jay Pfaffman) #8

Hey @sam, here’s a PR that allows S3 backups to Digital Ocean with shadow variables. I couldn’t figure out how to make the globals work.


(Jay Pfaffman) #9

@sam, per your comment in gitub, I’m pulling in Add shadowed_by_global for S3 uploads by pfaffman · Pull Request #6346 · discourse/discourse · GitHub here.

What I’m trying to do is put stuff (env variables, modification to config/discourse.conf) in a .yml file so that I can push backups (and uploads) to Digital Ocean Spaces and admins on the site can’t see the keys and/or disable pushing backups/uploads to Digital Ocean Spaces (S3).

My reading of discourse_defaults.conf suggests that sticking those values in DISCOURSE_VAR_NAME results in those values ending up in discourse.conf and hence overrides those values such that those variables don’t appear in the UI.

I feel like I must be missing something really obvious/trivial, as I’m pretty sure that you guys are doing this (unless you’re doing it in a plugin?)

What am I missing?


(Sam Saffron) #10

We do this via a plugin

I am not to keen to play with these settings now especially since many of these settings are somewhat unique and live in global settings as well with attached special behavior


(David Kingham) #11

A couple questions before I move forward with this:

  1. I’m currently using s3, I assume all old images will still be served from s3 because the urls are written in each post?
  2. If I copy all old images to spaces using rclone following this guide, can I then rebake all the old posts so they will be served from spaces?

(David Kingham) #12

I tried switching over to DO Spaces today and get this error when uploading an image:

SSL_connect returned=1 errno=0 state=error: certificate verify failed (ok)

These are my discourse settings:
s3 access key id - REDACTED
s3 secret access key - REDACTED
s3 upload bucket - npn
s3 endpoint - https://npn.sfo2.cdn.digitaloceanspaces.com
s3 cdn - BLANK
s3 backup bucket - npn-discourse-backups

and a screenshot of my Space settings:

The only thing I can see that is different from the original guide is that DO now allows you to use a CDN which I have turned on. I tried using these settings with the same result:
s3 endpoint - https://npn.sfo2.digitaloceanspaces.com
s3 cdn - https://npn.sfo2.cdn.digitaloceanspaces.com

I know the connection with access keys is working as I just used it with rsync without any problems. Any ideas?


(Jay Pfaffman) #13

I think that you want this to be https://sfo2.digitaloceanspaces.com/ (I’m not sure about the npn` part, and I’m not sure about the https). The CDN part is just for the, uh, CDN, it’s not the endpoint for putting stuff in spaces.

If you want to use the CDN, that goes in app.yml.

EDIT: This worked for me for the endpoint: https://nyc3.digitaloceanspaces.com


(David Kingham) #14

Ahhh! Taking the space name out of the endpoint was the key, thanks Jay!

For others trying to use CDN with DO Spaces for images, here is the setup:

Copy the origin endpoint from DO, remove the space name from the url, put this in s3 endpoint.
Copy the ‘Edge’ url from DO and put this in the s3 cdn url, but do not remove the space name.

image


(David Kingham) #15

I have copied all my files from s3 to DO using rsync, is there any way to update the urls on the old posts to point to DO instead of s3?


(Jay Pfaffman) #16

I don’t know, but since you’ve but gotten an answer, I am pretty sure that all you need to do is rebake.


(David Kingham) #17

Thanks Jay, I tried rebuilding the html on a couple posts and all it did was change from the previous CDN link to the direct link to s3 rather than changing to the new DO link. Does anyone know if rebake would produce different results than the rebuild html function? I don’t want to switch all the old posts to non CDN so I’m a little leary to run the rebake.


(Jay Pfaffman) #18

Did you stick the new CDN address in app.yml?


(David Kingham) #19

No, I’m not using the CDN for the entire site, just for the images.


#20

There seems to be no generic documentation on how to migrate from S3 to DO Spaces. Even though our instance is not yet heavy in file size, I find the flat-rate model more appealing and predictable.

Is the easiest way (in terms of sysadmin wizardry) to do the migration:

  1. Migrate from S3 to local
  2. Configure Spaces bucket credentials etc.
  3. Migrate to S3 (with Spaces configuration)
  4. Rebake everything

Would this do the trick? @rishabhn