Setting up file and image uploads to S3

(Joel Natividad) #42

Thanks @zogstrip!
Do you mind giving a quick howto on using it?

I SSHed into my discourse, CDed into /var/www/discourse and tried running
sudo -E u discourse bundle exec uploads:migrate_from_s3
and no joy

Unable to upload profile picture
(Sander Datema) #43

Bit of a late reply, but it’s a rake task, so you should use

rake uploads:migrate_from_s3


Just out of curiosity, the OP states that S3 for files/images will be deprecated, but backups will still be supported?

Are you suggesting to bypass the S3 backup functionality altogether and go with another solution? Or is it still a viable option to send the discourse generated backups to S3 using the existing functionality?

(Régis Hanol) #45

Sending backups to S3 will not be deprecated. Only support for storing images/attachments on S3.

(dobon) #52

I found another trap for young players: your bucket name can’t be the same as the name of the user that creates the bucket.

ie: a user named “dominic” cannot then make a bucket also called “dominic”.

There is a pop-up to tell you about the namespace conflict when you try manually creating a bucket with an already-used name in the AWS command center, but if you want Discourse to make the bucket automatically, the process will silently (I think? maybe it is buried in a log that I didn’t catch) fail.

(Dan Dascalescu) #53

The S3 bucket creation interface already asks to select a region:

What does the Discourse region setting do? I ran into a blocking issue with creating backups because of a region mismatch apparently.

S3 region vs. Discourse region
(Régis Hanol) #54

When you create a bucket on S3 you have to select a region in which it’ll be stored. You won’t be able to access that bucket from another region.

(ljpp) #55

So, we’ve been running our forum on Discourse for a few months and I can see the /Uploads folder gaining size rather rapidly. Looking ahead for the years to come and the amount of storage I will need for the images, the S3 option is starting to feel appealing.

But if I enable it now, on my already active site, what will happen to the images that are currently stored on the local server? Can I somehow transfer the existing image catalog to S3, or is Discourse brilliant enough to do it if S3 is enabled?

(Jeff Atwood) #56

Not sure, that is a question for @zogstrip

(Zinda Xyz) #57

not working s3

(Régis Hanol) #58

Was that file already uploaded before moving to S3? Can you try with an image that was not already uploaded on your Discourse?

(Pat David) #59

I’m also considering offloading image/uploads to S3 and have the same question as you. @zogstrip, if I implement S3 storage right now, will existing uploads be migrated?

(Régis Hanol) #60

No they won’t. You’ll have to run the “uploads:migrate_to_s3” rake task.

(Philip Colmer) #62

At some point, Discourse was extended to support the use of IAM profiles - kudos for that!

However, the downside is that our server’s dashboard is now complaining because it sees that I haven’t set s3_access_key_id or s3_secret_access_key. I guess I could put dummy values in :slight_smile:

I know that S3 is working because the server has created the bucket and migrated the user images.

(Régis Hanol) #63

Looking at the code we are properly taking into account the s3_use_iam_profile site setting.

Are all the mentioned site settings properly filled?

(Philip Colmer) #64

Are all the mentioned site settings properly filled?

They are … I’m wondering if the warning occurred while I was still setting things up. I didn’t think to click on Refresh and, now that I have, the message has disappeared.

Sorry for the noise.

(Philip Colmer) #65

Hmm … looks like I spoke too soon :frowning:

The problem checker is flagging up the problem again and clicking Refresh definitely brings the message back.

I’ve read and re-read the code that you linked to, @zogstrip, and it certainly looks as if it should work :frowning:

There are only three settings that the message complains about. One of the fields definitely has a value and, given that I’ve got “s3 use iam profile” checked, the other two fields shouldn’t be a problem:

I can put dummy values into the access fields to clear the message but that isn’t really the point :slight_smile:

(Anton) #66

What is the today’s state of the official Discourse advice re S3 for images? Should we still avoid using it?

Also, does using S3 mean the following:

  1. The images will be stored in S3 only and not locally
  2. The images won’t be included in backups

(Jeff Atwood) #67

It is 110% supported and safe.

(Scott Smith) #68

I switched to S3 several months ago and everything is working well. I would not recommend migrating the old images over however, the migration script is buggy - it didn’t move many images and also broke a few. I don’t know which ones got moved so I am keeping all the old images on my server, and so the overall disk usage (or Discourse backup size) has not reduced any. But at least its not growing rapidly like it was before the switch.