Setting up file and image uploads to S3


(jon r) #22

Did someone already try a “region hack” to use custom S3-API providers like Welcome to Ceph — Ceph Documentation or GitHub - basho/riak_cs: Riak CS is simple, available cloud storage built on Riak.?


(Justin Gordon) #23

Is this (using S3 for images) something that needs to be decided up front? Or is it easy to migrate later?

If I’m trying to build a large global community forum, should I be starting with S3?

Is there a migration for moving to S3 or can a few images stay stored locally?

If I use S3, can I allow 24 MP images (as my Nikon 7100 takes them)? I’d rather not hassle users with having to convert images before posting. @sam, it wasn’t totally clear to me that we’re resizing images as part of the upload process. Clearly, we would not want to serve a 24 mp image to a mobile phone browser.


(Sam Saffron) #24

Personally, I would strongly recommend to avoid s3 and instead use origin pull with your CDN of choice.

We have zero systems in production that use the magic S3 switches and it is quite likely to have bugs going forward.

To be honest I would prefer to rip the S3 image stuff out and just support origin pull CDNs which amazon does as well.


(Justin Gordon) #25

So just storing images in the default configuration on Digital Ocean is the way to go? How do backups handle images? I’m guessing they are just included in the backups.


(Adam Loving) #26

I followed the instructions at the top of the post, and S3 didn’t immediately work for me, so I thought I’d share what I did to debug. I am using heroku.

I think the issue for me was due to file size. I was testing with a 2MB file that wasn’t big enough to trigger the “file too big” Discourse error message, but didn’t make it to S3 either.

You can run this from heroku run console to test if everything is configured correctly.

bucket_name = Discourse.store.send :s3_bucket
bucket = Discourse.store.send :get_or_create_directory, bucket_name

f = File.open('README.md')
bucket = Discourse.store.send :upload, f, 'test.txt'

(Kane York) #27

And for regular installations, just do those commands from this:

cd /var/docker
./launcher ssh app
rails c

(Lee) #28

Hey Adam,

Thanks for the great idea. I’m able to push the readme from heroku -> s3 but I’m not able to upload avatars. The files I’ve tried are 48kb and 78kb jpegs respectively. Any thoughts as to what else I could test? I’m new to Discourse and kinda stumped.


(Dave McClure) #29

What is the recommended practice for DO installs? Leave images and uploads on the DO box?

Are images included in the backups?


(Régis Hanol) #30

Yes. 20-30GB should be enough for a medium-sized forum.

Yes they are. (all the uploads are included in the backups).


(Gary Windham) #31

Hi folks,

I’m getting upload errors in Discourse after configuring S3 for file uploads, along with “The AWS Access Key Id you provided does not exist in our records.” errors in production.log.

I installed Discourse on an EC2 instance, using the Docker image. The EC2 instance I launched has an IAM role, “discourse-myorg-upload”, which gives full access to an S3 bucket called “discourse-myorg-files”:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::discourse-myorg-files",
        "arn:aws:s3:::discourse-myorg-files/*"
      ]
    }
  ]
}

This IAM role was applied to the EC2 instance when it was launched, and I get a valid response (with Access Key, Secret Key, etc) when I do a curl http://169.254.169.254/latest/meta-data/iam/security-credentials/discourse-myorg-upload from that instance. I plugged these values into the correct S3 fields under Settings->Files and have checked them many times for accuracy), even re-entered them a couple of times…no dice.

Any ideas as to what I missed?


Allow use of AWS EC2 IAM roles with S3 file/image uploads
(Gary Windham) #32

Sorry to reply to my own post, but I figured it out…lib/file_store/s3_store.rb, which uses the “fog” gem, doesn’t have any provision for using fog’s “use_iam_profile” option. I will submit a feature request to address this.


(Jeff Widman) #33

I really like having the option to easily decouple the app server from the asset storage.

If you pull out these S3 connectors, how hard will it be to build it back in using a plugin?

I do understand that for the vast majority of Discourse installs it’s simpler to just stick everything together on a single box.

But as I think about migrating RockClimbing.com off our current setup, we’ve got 80+ gb of images, videos, etc across multiple apps, and these apps sometimes access the same files. For example, our climbing routes app, our photos app, and our forum currently can all easily access the same photo file.

My current plan is to break out each app into it’s own Docker container, alongside our Discourse Docker instance, and use a single S3 bucket as the central file storage for our non-code assets across all apps, rather than try to get various apps to reach inside the Discourse Docker container on another server.

Or am I misunderstanding and pulling out the S3 connectors wouldn’t affect this scenario? I don’t understand what @sam means about when he talks about supporting origin pull cdn is better than supporting S3…

Or maybe there’s some reason I’m unaware of why each app/docker container should have a separate file store? (more of a general web app architecture question than a discourse-specific question)


(node) #34

You may experience the error Statement is missing required element - Statement "NO_ID-0" is missing "Principal" element

To fix this use the following code:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "",
			"Effect": "Allow",
			"Principal": "*",
			"Action": "s3:*",
			"Resource": [
				"arn:aws:s3:::name-of-your-bucket",
				"arn:aws:s3:::name-of-your-bucket/*"
			]
		}
	]
}

(Music Existence) #35

This does not work for me :frowning: It wont create the bucks.


(Eric Schleicher) #36

Agreed. I’ll create a separate topic since its a bit organonal to this one.


Migration of system stored images to S3 after configuration change
(Rex) #37

S3 file still don’t work. Should I make the “s3 use iam profile” setting check or uncheck?

When I make it uncheck, I get NoMethodError: undefined methodget_or_create_directory’ for #FileStore::LocalStore:0x007fae3b8ccff8` using these commands you provide.

But when I make it check, I get [fog][WARNING] Unable to fetch credentials: Connection refused - connect(2) (Errno::ECONNREFUSED)

Is it something wrong?


(Sean Cronin) #38

Is S3 for uploads still not recommended? I’m setting up my discourse site in AWS right now, and it’d be nice to know if I should avoid this option.


(Sam Saffron) #39

Yes, I would recommend using origin pull and a simple built-in backup system. Simpler and better tested solution.


(Joel Natividad) #40

I wish I found this posting before we turned on S3 image hosting. :worried:

We’d like to go back and just leave the file/images on the DigitalOcean Discourse Docker image.

Is there an automated way to do that? If not, what are the steps to move the files back with minimal disruption to the users?

Also, we’d still like an offsite backup on S3, is that OK? If S3 is not really recommended for image/files, shouldn’t that feature be deprecated and eventually retired?

Thanks in advance!


(Régis Hanol) #41

There’s the uploads:migrate_from_s3 rake task. Hasn’t seen much usage though…