S3 buckets containing periods not allowed

[sam (Sam Saffron)]

aha … well then let’s just leave it as is for now.

[uppfinnarn]

This is a good point. However, I think it should be downgraded to a warning, rather than being outright forbidden.

While locking yourself out of HTTPS is a terrible idea on its own, both Amazon CloudFront and CloudFlare (off the top of my head) offer SNI-based SSL certificates that can be put in front of a bucket, even if the default certificate would no longer cover it.

CloudFront can be configured to pull from an S3 bucket of your choice (even if it’s named something different), but also costs money. Well invested money, one may argue, but money nonetheless. If one does not need the additional functionality of a full-fledged CDN, CloudFront is a slightly overkill solution to the problem of “I want my forum to load faster”.
(An incredibly convoluted pricing model doesn’t help… I think there’s an XKCD for this, but I can’t find it.)

CloudFlare is not as configurable, but also offer quite a few other features as well - as well as the very approachable price tag of free. With a properly named bucket and a DNS rule (possibly supplemented by some caching rules), you get an instant improvement to loading performance, with no downsides whatsoever.
Perhaps not as big of a difference as a full CDN would make, but hey, it’s free performance.

[gingerman]

@codinghorror @sam I am using a cloudflare CDN to access the S3 content (where discourse uploads all the files) and not directly.

s3 mandates to use bucket name like files.discourse.org to access its content through cname files

http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html#VirtualHostingCustomURLs

Is there a way to allow periods in the S3 name (may be with a warning for https issue for direct s3 use). This is a showstopper in my case to use S3 with cloudflare and discourse.

Thanks

[fearlessfrog]

We worked around that here @gingerman :

S3 upload bucket, cdn url and bucket name in CNAME

I was recently looking at doing this, and wanted to chime in with a reason why it is useful. We use a free Cloudflare plan (as we’re a small forum) and it works great. We turn off rocketloader etc and just use it like a decent pull CDN by pointing the Cloudflare subdomain DNS entry (forum.) at the discourse instance and all is good. It speeds things up and protects our server nicely. When we wanted to switch to use S3 for uploads from local files, we realized that we can’t use Cloudflare for …

[gingerman]

@fearlessfrog Thanks…I ended up doing exactly the same. I don’t know how I missed your thread earlier. Just to highlight - this approach has a limitation in terms of cache invalidation. we have to do separately invalidations in cloudfront and cloudflare (after rebaking posts). It would be great to automate them with a script at the very least.

I hit up on the same blocks eventually in terms of optimised image urls and was planning to start a thread today…Nice to know that you have already helped the team towards a fix in the below given thread. Kudos for that.

Want to understand s3_cdn_url files setting

Here’s the fix https://github.com/discourse/discourse/commit/f3905fd99ab2d26a98a9554f61be25c2922383ff

[delta (David Wilkinson)]

I just wanted to add my +1 for allowing periods in S3 bucket names within Discourse. AWS itself allows (indeed instructs) this now for CNAME-based access to S3 content, and provided the s3_cdn_url setting in Discourse is utilized so far as I can tell the previous issues encountered around HTTPS are rendered non-problematic, at least when services such as Cloudflare are used (as @uppfinnarn notes SNI-based SSL certs solve this issue).

Might it be possible to revisit this in an upcoming release?

Thank you for a great product!

[Falco (Rafael dos Santos Silva)]

Since we already support the S3_CDN_URL to be anything, what does we win by allowing this?

[delta (David Wilkinson)]

Hi Rafael, unless I’m missing something although that param allows Discourse to pull from a CDN URL, if we can’t specify an S3 bucket name with periods in, we can’t actually upload to those buckets? This necessitates hacky workarounds like setting up alternative CNAMEs in additional non-free third party services like CloudFront (which we’re not using at all at this time, and would prefer not to).

[delta (David Wilkinson)]

(And I guess ultimately I just don’t understand why you’d ever want to blanket reject something that is a valid S3 bucket name!)

[codinghorror (Jeff Atwood)]

So you don’t use https … at all? That’s the main reason per the above.

[Falco (Rafael dos Santos Silva)]

I don’t understand what is the trade-off here.

I we allow buckets with dots, we must add a setting so you can disable SSL verification on connect, or people using it must add a proxy that adds SSL between Discourse and the S3 endpoint but it will still be plain HTTP between the proxy and S3 endpoint.

All that, and what is the benefit?

[delta (David Wilkinson)]

So you don’t use https … at all? That’s the main reason per the above.

Hi Jeff, we are using HTTPS, via Cloudflare.

I we allow buckets with dots, we must add a setting so you can disable SSL verification on connect, or people using it must add a proxy that adds SSL between Discourse and the S3 endpoint but it will still be plain HTTP between the proxy and S3 endpoint.

I’m pretty sure this would be picked up and handled by Cloudflare automatically without need to add an additional setting (it’s an approach we’ve used before with other services).

I can totally appreciate wanting to warn users off using a convention you think problematic, but given valid S3 bucket names can contain periods (and AWS docs advise using them in a range of scenarios) would it be something you might consider enabling in future?

[codinghorror (Jeff Atwood)]

@delta

would it be something you might consider enabling in future?

No, we would not consider it.

[delta (David Wilkinson)]

Okay, well thanks. I guess we’ll patch or find an alternative

[riking (Kane York)]

What alternative? Just create a new S3 bucket without dots in it. There’s no reason I can think of to use an existing bucket for your Discourse uploads. It even makes accounting easier as there’s not random non-Discourse objects in there.

[delta (David Wilkinson)]

Hopefully something less suboptimal than this (but at least it’d work).

[riking (Kane York)]

@delta

Hopefully something less suboptimal than this (but at least it’d work).

The CloudFront CDN is optional. You could use any other CDN you want with a no-dots bucket name and you wouldn’t have the HTTPS problems either.

@delta

the previous issues encountered around HTTPS are rendered non-problematic, at least when services such as Cloudflare are used (as @uppfinnarn notes SNI-based SSL certs solve this issue).

@Falco

but it will still be plain HTTP between the proxy and S3 endpoint.

@delta

Hi Jeff, we are using HTTPS, via Cloudflare.

Also, this image is the only thing I can think of while reading that exchange.

(Warning: anyone holding a current US security clearance should not reveal the image here.)

[theblazehen]

So, I also wanted to put my uploads behind CloudfFlare, using Wasabi instead of S3, so no CloudFront available.

I ended up directly modifying the DB, and restarting discourse and it worked. Also, does https between CDN and S3 really matter if the bucket is public read only?

UPDATE site_settings SET value = 'uploads.domain.tld' WHERE name = 's3_upload_bucket';

[riking (Kane York)]

Set the DISCOURSE_S3_CDN variable as explained in the #howto guides.

[theblazehen]

I can set the CDN URL just fine, and point it directly to wasabi. But the only way to get it to go through cloudflare first is to have a CNAME on my domain pointing to s3.wasabisys.com, where the bucket name needs to be subdomain.your.domain.

Eg, I can’t have a bucket called xyz-uploads and use that behind cloudflare with a CNAME to s3.wasabisys.com