S3 upload bucket, cdn url and bucket name in CNAME

(Lars) #1

Continuing the discussion from S3 Upload with Vanity Domains:

I have some questions!

1: I am running out of space on my droplet because of uploads, so I want to use S3 for my uploads.

I might have missed something, but in the S3 documentation:

Because Amazon S3 sees only the original host name www.example.com and is unaware of the CNAME mapping used to resolve the request, the CNAME and the bucket name must be the same.

But in the discourse settings in “s3 upload bucket” it says “must be lowercase, no periods, no underscores.” - and it won’t let you save the bucket name. I can see that there is an “s3 cdn url” to put the vanity name - but I don’t get how I am supposed to set it up.

2: When I get it to work - will Discourse move all my existing images to S3 or do I have to do something?

Thanks in advance! :smiley:

I can't get a good backup - it fails after after_create_hook
(Jeff Atwood) #2

Hmm, not sure, @zogstrip would know best. Periods in bucket names cause major havoc with https and certificates.

(Lars) #3

Is there another way to do this? Should I just do it without a vanity domain?
I would love to offload some storage - my droplet’s running out of space…

(Régis Hanol) #4

Just wondering, why do you need a vanity domain for your uploads?

(Lars) #5

I just think it’s nicer - Just as I use sendgrid with read counters etc using my own domain…
Anyway, I just thought it was a bit confusing that there is a s3 cdn url, but we can’t really use it because amazon wants to use the domain name as the bucket name…

(fearlessfrog) #6

I was recently looking at doing this, and wanted to chime in with a reason why it is useful.

We use a free Cloudflare plan (as we’re a small forum) and it works great. We turn off rocketloader etc and just use it like a decent pull CDN by pointing the Cloudflare subdomain DNS entry (forum.) at the discourse instance and all is good. It speeds things up and protects our server nicely.

When we wanted to switch to use S3 for uploads from local files, we realized that we can’t use Cloudflare for our free CDN for these often large images. This means our S3 bill is now getting larger (lots of GETs and repeated bandwidth) and it is pretty slow generally just used as a file store in the sky (for topics with lots of images in them). We can set up AWS Cloudfront in-front of S3 and set-up Discourse like that to improve the speed but that too costs more money, but what we really wanted to do is use the free Cloudflare to save us the bandwidth and surge requests.

Cloudflare and S3 require the virtual DNS naming that Discourse prohibits:

Now @codinghorror is correct that if we were hosting our own wildcard SSL cert then this might be an issue with a files.myforum.com.s3.amazonaws.com due to the periods in the name, but we’re not. We’re using Cloudflares free SSL support. Between Cloudflare and our secure it isn’t end-to-end secure, but we’re ok with that as it’s a free feature.

So for our case, where we don’t have a local wildcard cert and don’t need that, it would be really beneficial to our running costs to be able to use AWS virtual hosting of buckets using these period names.

S3 buckets containing periods not allowed
(fearlessfrog) #7

As a follow-up to this, and I’m not sure if this is madness or not :slight_smile: but one way around that seems to work is to do the following:

  • Set up S3 uploads i.e. mybucket and then let Discourse use that.

  • In AWS create a CloudFront distribution for that S3 bucket. Set up an alternative CNAME called ‘uploads.myforum.com’.

  • In CloudFlare set up a CNAME and active cache for uploads.myforum.com and point it at the xyz.cloudfront.net AWS domain name for that distribution.

  • Tell Discourse to use the CDN URL https://uploads.myforum.com (and I guess rebake posts, although the old Cloudfront URL should resolve still fine?)

That way you get a wildcard HTTPS on your images and (I guess?) lower CloudFront and S3 GETs and bandwidth. A couple of CURL tests show Cloudflare is caching ok in-front of Cloudfront.

So, basically using CloudFront’s ability to have an alternative CNAME to get around the periods in S3 upload buckets in Discourse.

This all seems to work ok (I think?), but the the Discourse CDN setting doesn’t seem to do what I thought it would do, i.e:

If all the URLs in the uploads used the CDN value then we’d be good.

(Silver Quettier) #8

You sir, are crazy. Crazy brillant. This works and was exactly what I was looking for. :smile:

Now, to wait for 1.7 to enter stable, and get my hands on the fix for lightboxes.

(ljpp) #9

I am now approaching the same territory – the S3 bill has started bloating rapidly. We are also using CloudFlare and are very happy with it.

With this CloudFlare <- CloudFront <- S3 setup, what kind of bandwidth and financial savings have you achieved?