Setting up file and image uploads to S3


(Gary Windham) #31

Hi folks,

I’m getting upload errors in Discourse after configuring S3 for file uploads, along with “The AWS Access Key Id you provided does not exist in our records.” errors in production.log.

I installed Discourse on an EC2 instance, using the Docker image. The EC2 instance I launched has an IAM role, “discourse-myorg-upload”, which gives full access to an S3 bucket called “discourse-myorg-files”:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::discourse-myorg-files",
        "arn:aws:s3:::discourse-myorg-files/*"
      ]
    }
  ]
}

This IAM role was applied to the EC2 instance when it was launched, and I get a valid response (with Access Key, Secret Key, etc) when I do a curl http://169.254.169.254/latest/meta-data/iam/security-credentials/discourse-myorg-upload from that instance. I plugged these values into the correct S3 fields under Settings->Files and have checked them many times for accuracy), even re-entered them a couple of times…no dice.

Any ideas as to what I missed?


Allow use of AWS EC2 IAM roles with S3 file/image uploads
(Gary Windham) #32

Sorry to reply to my own post, but I figured it out…lib/file_store/s3_store.rb, which uses the “fog” gem, doesn’t have any provision for using fog’s “use_iam_profile” option. I will submit a feature request to address this.


(Jeff Widman) #33

I really like having the option to easily decouple the app server from the asset storage.

If you pull out these S3 connectors, how hard will it be to build it back in using a plugin?

I do understand that for the vast majority of Discourse installs it’s simpler to just stick everything together on a single box.

But as I think about migrating RockClimbing.com off our current setup, we’ve got 80+ gb of images, videos, etc across multiple apps, and these apps sometimes access the same files. For example, our climbing routes app, our photos app, and our forum currently can all easily access the same photo file.

My current plan is to break out each app into it’s own Docker container, alongside our Discourse Docker instance, and use a single S3 bucket as the central file storage for our non-code assets across all apps, rather than try to get various apps to reach inside the Discourse Docker container on another server.

Or am I misunderstanding and pulling out the S3 connectors wouldn’t affect this scenario? I don’t understand what @sam means about when he talks about supporting origin pull cdn is better than supporting S3…

Or maybe there’s some reason I’m unaware of why each app/docker container should have a separate file store? (more of a general web app architecture question than a discourse-specific question)


(node) #34

You may experience the error Statement is missing required element - Statement "NO_ID-0" is missing "Principal" element

To fix this use the following code:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "",
			"Effect": "Allow",
			"Principal": "*",
			"Action": "s3:*",
			"Resource": [
				"arn:aws:s3:::name-of-your-bucket",
				"arn:aws:s3:::name-of-your-bucket/*"
			]
		}
	]
}

(Music Existence) #35

This does not work for me :frowning: It wont create the bucks.


(Eric Schleicher) #36

Agreed. I’ll create a separate topic since its a bit organonal to this one.


Migration of system stored images to S3 after configuration change
(Rex) #37

S3 file still don’t work. Should I make the “s3 use iam profile” setting check or uncheck?

When I make it uncheck, I get NoMethodError: undefined methodget_or_create_directory’ for #FileStore::LocalStore:0x007fae3b8ccff8` using these commands you provide.

But when I make it check, I get [fog][WARNING] Unable to fetch credentials: Connection refused - connect(2) (Errno::ECONNREFUSED)

Is it something wrong?


(Sean Cronin) #38

Is S3 for uploads still not recommended? I’m setting up my discourse site in AWS right now, and it’d be nice to know if I should avoid this option.


(Sam Saffron) #39

Yes, I would recommend using origin pull and a simple built-in backup system. Simpler and better tested solution.


(Joel Natividad) #40

I wish I found this posting before we turned on S3 image hosting. :worried:

We’d like to go back and just leave the file/images on the DigitalOcean Discourse Docker image.

Is there an automated way to do that? If not, what are the steps to move the files back with minimal disruption to the users?

Also, we’d still like an offsite backup on S3, is that OK? If S3 is not really recommended for image/files, shouldn’t that feature be deprecated and eventually retired?

Thanks in advance!


(Régis Hanol) #41

There’s the uploads:migrate_from_s3 rake task. Hasn’t seen much usage though…


(Joel Natividad) #42

Thanks @zogstrip!
Do you mind giving a quick howto on using it?

I SSHed into my discourse, CDed into /var/www/discourse and tried running
sudo -E u discourse bundle exec uploads:migrate_from_s3
and no joy


Unable to upload profile picture
(Sander Datema) #43

Bit of a late reply, but it’s a rake task, so you should use

rake uploads:migrate_from_s3

#44

Just out of curiosity, the OP states that S3 for files/images will be deprecated, but backups will still be supported?

Are you suggesting to bypass the S3 backup functionality altogether and go with another solution? Or is it still a viable option to send the discourse generated backups to S3 using the existing functionality?


(Régis Hanol) #45

Sending backups to S3 will not be deprecated. Only support for storing images/attachments on S3.


(dobon) #52

I found another trap for young players: your bucket name can’t be the same as the name of the user that creates the bucket.

ie: a user named “dominic” cannot then make a bucket also called “dominic”.

There is a pop-up to tell you about the namespace conflict when you try manually creating a bucket with an already-used name in the AWS command center, but if you want Discourse to make the bucket automatically, the process will silently (I think? maybe it is buried in a log that I didn’t catch) fail.


(Dan Dascalescu) #53

The S3 bucket creation interface already asks to select a region:

What does the Discourse region setting do? I ran into a blocking issue with creating backups because of a region mismatch apparently.


S3 region vs. Discourse region
(Régis Hanol) #54

When you create a bucket on S3 you have to select a region in which it’ll be stored. You won’t be able to access that bucket from another region.


(ljpp) #55

So, we’ve been running our forum on Discourse for a few months and I can see the /Uploads folder gaining size rather rapidly. Looking ahead for the years to come and the amount of storage I will need for the images, the S3 option is starting to feel appealing.

But if I enable it now, on my already active site, what will happen to the images that are currently stored on the local server? Can I somehow transfer the existing image catalog to S3, or is Discourse brilliant enough to do it if S3 is enabled?


(Jeff Atwood) #56

Not sure, that is a question for @zogstrip