Configure an S3 compatible object storage provider for uploads

[Jerome_Hordies]

I will try to summarize the answers to my questions:

• Do the Web UI and ENV variable collide ?

@pfaffman

If you use the ENV variables (which you should) those settings are no longer visible in the Web UI.

• When are the assets supposed to be uploaded to the bucket ?
  By adding this snippet to the app.yml in the hook section, it will be uploaded after_assets_precompile (during rebuild app).

@pfaffman

@Falco

 after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets

• How can I debug this ? I don’t see any error in the logs
  By running :

cd /var/discourse
sudo ./launcher enter app
sudo -E -u discourse bundle exec rake s3:upload_assets --trace

• Is it possible to set a subfolder of a bucket in the config ?

@zogstrip

Do I really need to use separate buckets for uploads and backups?

No, you don’t, but it’s usually the easiest way to set-up. Essentially you need to either use two different buckets or a prefix for the backup bucket. For example, the following combinations will work:

1. Different buckets

• s3_upload_bucket: your-uploads-bucket
• s3_backup_bucket: your-backups-bucket

2. Different prefixes

• s3_upload_bucket: your-uploads-bucket/uploads
• s3_backup_bucket: your-uploads-bucket/backups

You can use prefixes to organize the data that you store in Amazon S3 buckets. A prefix is a string of characters at the beginning of the object key name. A prefix can be any length, subject to the maximum length of the object key name (1,024 bytes). You can think of prefixes as a way to organize your data in a similar way to directories. However, prefixes are not directories.

• Once this works, Are the previously uploaded images transferred to the bucket ? If I rebake, what will the url of the previously uploaded images look like ?

@zogstrip

I’ve enabled S3 uploads in my Discourse instance (which has been going for a while); what do I do with the existing local uploads?

To migrate your existing uploads to S3, you can do a couple of rake tasks. To perform this, you need SSH access, root permissions, and have entered the discourse app (as per Administrative Bulk Operations ). Oh, and you have to set some environmental variables in app.yml. Not for the faint-hearted.

Once you have done all that you are ready for the rake tasks:

rake uploads:migrate_to_s3
rake posts:rebake

Once these are done (and the uploads are working well) you no longer need to include uploads in your backups. And as a bonus, you will be able to Restore a backup from command line in the event of catastrophe (just keep a copy of app.yml somewhere).