Setting up file and image uploads to S3


(Régis Hanol) #14

Did you figure it out?

The highlighting engine is unfortunately not smart enought to detect that your code block is javascript. You can force it using GitHub’s fenced code blocks.


(Jonathan Allard) #15

I have the same “signature” problem.

Edit: I just changed the keys, and it went away. Maybe I had a trailing space? @haiku would you check yours?


(Lgkh) #17

Faced a problem below…

Installed latest discourse today. Setup S3 using the guide above but when creating new topic with image upload, it still loads from local /uploads folder.

So I checked Amazon S3 console, there was no bucket created.
Tried creating bucket manually, but same issue.
Checked keys such that there is no trailing space.
Checked logs, there was no errors relating to S3.

How can I debug this?

Discourse Version: 0.9.8.11
Git Version: a1b501c3fba126a3bc1705bea69a6397196b396e

UPDATED:
It is working now, guess it was some sort of delay/cache?


(Joegoggins) #18

Does anyone have any suggestions about how to migrate file system stored images into S3?

If I need to write a script to make it happen that’s cool (and I’d be happy to share it), just figured I’d ask here before diving into it.


(Phil Nelson) #19

I would also like some kind of howto for this.


(Geoff Bowers) #20

Might want to add to the howto a recommendation to not use dots in the bucket name. This is allowed by AWS but prevents you from referencing the bucket under SSL.

I came a cropper over this one earlier:


(Jeff Atwood) #21

Very good point, done in the code and here too. We’ve had 5-6 reports of bucket problems due to periods in the name, something we definitely want to avoid in the future.


(jon r) #22

Did someone already try a “region hack” to use custom S3-API providers like Welcome to Ceph — Ceph Documentation or GitHub - basho/riak_cs: Riak CS is simple, available cloud storage built on Riak.?


(Justin Gordon) #23

Is this (using S3 for images) something that needs to be decided up front? Or is it easy to migrate later?

If I’m trying to build a large global community forum, should I be starting with S3?

Is there a migration for moving to S3 or can a few images stay stored locally?

If I use S3, can I allow 24 MP images (as my Nikon 7100 takes them)? I’d rather not hassle users with having to convert images before posting. @sam, it wasn’t totally clear to me that we’re resizing images as part of the upload process. Clearly, we would not want to serve a 24 mp image to a mobile phone browser.


(Sam Saffron) #24

Personally, I would strongly recommend to avoid s3 and instead use origin pull with your CDN of choice.

We have zero systems in production that use the magic S3 switches and it is quite likely to have bugs going forward.

To be honest I would prefer to rip the S3 image stuff out and just support origin pull CDNs which amazon does as well.


(Justin Gordon) #25

So just storing images in the default configuration on Digital Ocean is the way to go? How do backups handle images? I’m guessing they are just included in the backups.


(Adam Loving) #26

I followed the instructions at the top of the post, and S3 didn’t immediately work for me, so I thought I’d share what I did to debug. I am using heroku.

I think the issue for me was due to file size. I was testing with a 2MB file that wasn’t big enough to trigger the “file too big” Discourse error message, but didn’t make it to S3 either.

You can run this from heroku run console to test if everything is configured correctly.

bucket_name = Discourse.store.send :s3_bucket
bucket = Discourse.store.send :get_or_create_directory, bucket_name

f = File.open('README.md')
bucket = Discourse.store.send :upload, f, 'test.txt'

(Kane York) #27

And for regular installations, just do those commands from this:

cd /var/docker
./launcher ssh app
rails c

(Lee) #28

Hey Adam,

Thanks for the great idea. I’m able to push the readme from heroku -> s3 but I’m not able to upload avatars. The files I’ve tried are 48kb and 78kb jpegs respectively. Any thoughts as to what else I could test? I’m new to Discourse and kinda stumped.


(Dave McClure) #29

What is the recommended practice for DO installs? Leave images and uploads on the DO box?

Are images included in the backups?


(Régis Hanol) #30

Yes. 20-30GB should be enough for a medium-sized forum.

Yes they are. (all the uploads are included in the backups).


(Gary Windham) #31

Hi folks,

I’m getting upload errors in Discourse after configuring S3 for file uploads, along with “The AWS Access Key Id you provided does not exist in our records.” errors in production.log.

I installed Discourse on an EC2 instance, using the Docker image. The EC2 instance I launched has an IAM role, “discourse-myorg-upload”, which gives full access to an S3 bucket called “discourse-myorg-files”:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::discourse-myorg-files",
        "arn:aws:s3:::discourse-myorg-files/*"
      ]
    }
  ]
}

This IAM role was applied to the EC2 instance when it was launched, and I get a valid response (with Access Key, Secret Key, etc) when I do a curl http://169.254.169.254/latest/meta-data/iam/security-credentials/discourse-myorg-upload from that instance. I plugged these values into the correct S3 fields under Settings->Files and have checked them many times for accuracy), even re-entered them a couple of times…no dice.

Any ideas as to what I missed?


Allow use of AWS EC2 IAM roles with S3 file/image uploads
(Gary Windham) #32

Sorry to reply to my own post, but I figured it out…lib/file_store/s3_store.rb, which uses the “fog” gem, doesn’t have any provision for using fog’s “use_iam_profile” option. I will submit a feature request to address this.


(Jeff Widman) #33

I really like having the option to easily decouple the app server from the asset storage.

If you pull out these S3 connectors, how hard will it be to build it back in using a plugin?

I do understand that for the vast majority of Discourse installs it’s simpler to just stick everything together on a single box.

But as I think about migrating RockClimbing.com off our current setup, we’ve got 80+ gb of images, videos, etc across multiple apps, and these apps sometimes access the same files. For example, our climbing routes app, our photos app, and our forum currently can all easily access the same photo file.

My current plan is to break out each app into it’s own Docker container, alongside our Discourse Docker instance, and use a single S3 bucket as the central file storage for our non-code assets across all apps, rather than try to get various apps to reach inside the Discourse Docker container on another server.

Or am I misunderstanding and pulling out the S3 connectors wouldn’t affect this scenario? I don’t understand what @sam means about when he talks about supporting origin pull cdn is better than supporting S3…

Or maybe there’s some reason I’m unaware of why each app/docker container should have a separate file store? (more of a general web app architecture question than a discourse-specific question)


(node) #34

You may experience the error Statement is missing required element - Statement "NO_ID-0" is missing "Principal" element

To fix this use the following code:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "",
			"Effect": "Allow",
			"Principal": "*",
			"Action": "s3:*",
			"Resource": [
				"arn:aws:s3:::name-of-your-bucket",
				"arn:aws:s3:::name-of-your-bucket/*"
			]
		}
	]
}