Setting up file and image uploads to S3

(Rex) #37

S3 file still don’t work. Should I make the “s3 use iam profile” setting check or uncheck?

When I make it uncheck, I get NoMethodError: undefined methodget_or_create_directory’ for #FileStore::LocalStore:0x007fae3b8ccff8` using these commands you provide.

But when I make it check, I get [fog][WARNING] Unable to fetch credentials: Connection refused - connect(2) (Errno::ECONNREFUSED)

Is it something wrong?

(Sean Cronin) #38

Is S3 for uploads still not recommended? I’m setting up my discourse site in AWS right now, and it’d be nice to know if I should avoid this option.

(Sam Saffron) #39

Yes, I would recommend using origin pull and a simple built-in backup system. Simpler and better tested solution.

(Joel Natividad) #40

I wish I found this posting before we turned on S3 image hosting. :worried:

We’d like to go back and just leave the file/images on the DigitalOcean Discourse Docker image.

Is there an automated way to do that? If not, what are the steps to move the files back with minimal disruption to the users?

Also, we’d still like an offsite backup on S3, is that OK? If S3 is not really recommended for image/files, shouldn’t that feature be deprecated and eventually retired?

Thanks in advance!

(Régis Hanol) #41

There’s the uploads:migrate_from_s3 rake task. Hasn’t seen much usage though…

(Joel Natividad) #42

Thanks @zogstrip!
Do you mind giving a quick howto on using it?

I SSHed into my discourse, CDed into /var/www/discourse and tried running
sudo -E u discourse bundle exec uploads:migrate_from_s3
and no joy

Unable to upload profile picture
(Sander Datema) #43

Bit of a late reply, but it’s a rake task, so you should use

rake uploads:migrate_from_s3


Just out of curiosity, the OP states that S3 for files/images will be deprecated, but backups will still be supported?

Are you suggesting to bypass the S3 backup functionality altogether and go with another solution? Or is it still a viable option to send the discourse generated backups to S3 using the existing functionality?

(Régis Hanol) #45

Sending backups to S3 will not be deprecated. Only support for storing images/attachments on S3.

(dobon) #52

I found another trap for young players: your bucket name can’t be the same as the name of the user that creates the bucket.

ie: a user named “dominic” cannot then make a bucket also called “dominic”.

There is a pop-up to tell you about the namespace conflict when you try manually creating a bucket with an already-used name in the AWS command center, but if you want Discourse to make the bucket automatically, the process will silently (I think? maybe it is buried in a log that I didn’t catch) fail.

(Dan Dascalescu) #53

The S3 bucket creation interface already asks to select a region:

What does the Discourse region setting do? I ran into a blocking issue with creating backups because of a region mismatch apparently.

S3 region vs. Discourse region
(Régis Hanol) #54

When you create a bucket on S3 you have to select a region in which it’ll be stored. You won’t be able to access that bucket from another region.

(ljpp) #55

So, we’ve been running our forum on Discourse for a few months and I can see the /Uploads folder gaining size rather rapidly. Looking ahead for the years to come and the amount of storage I will need for the images, the S3 option is starting to feel appealing.

But if I enable it now, on my already active site, what will happen to the images that are currently stored on the local server? Can I somehow transfer the existing image catalog to S3, or is Discourse brilliant enough to do it if S3 is enabled?

(Jeff Atwood) #56

Not sure, that is a question for @zogstrip

(Zinda Xyz) #57

not working s3

(Régis Hanol) #58

Was that file already uploaded before moving to S3? Can you try with an image that was not already uploaded on your Discourse?

(Pat David) #59

I’m also considering offloading image/uploads to S3 and have the same question as you. @zogstrip, if I implement S3 storage right now, will existing uploads be migrated?

(Régis Hanol) #60

No they won’t. You’ll have to run the “uploads:migrate_to_s3” rake task.

(Philip Colmer) #62

At some point, Discourse was extended to support the use of IAM profiles - kudos for that!

However, the downside is that our server’s dashboard is now complaining because it sees that I haven’t set s3_access_key_id or s3_secret_access_key. I guess I could put dummy values in :slight_smile:

I know that S3 is working because the server has created the bucket and migrated the user images.

(Régis Hanol) #63

Looking at the code we are properly taking into account the s3_use_iam_profile site setting.

Are all the mentioned site settings properly filled?