S3 buckets containing periods not allowed

If periods are indeed supported we should relax the restriction … @techAPJ can you have a quick look

As I understood it, periods in the bucket name cause havoc with https and certificates so it is strongly discouraged that you do this. Unless you never plan to use https.

http://shlomoswidler.com/2009/08/amazon-s3-gotcha-using-virtual-host.html

We ran into many problems with this, which is why it is disallowed. It is not random. It’s a resolution to a problem many Discourse instances faced.

5 Likes

aha … well then let’s just leave it as is for now.

1 Like

This is a good point. However, I think it should be downgraded to a warning, rather than being outright forbidden.

While locking yourself out of HTTPS is a terrible idea on its own, both Amazon CloudFront and CloudFlare (off the top of my head) offer SNI-based SSL certificates that can be put in front of a bucket, even if the default certificate would no longer cover it.

CloudFront can be configured to pull from an S3 bucket of your choice (even if it’s named something different), but also costs money. Well invested money, one may argue, but money nonetheless. If one does not need the additional functionality of a full-fledged CDN, CloudFront is a slightly overkill solution to the problem of “I want my forum to load faster”.
(An incredibly convoluted pricing model doesn’t help… I think there’s an XKCD for this, but I can’t find it.)

CloudFlare is not as configurable, but also offer quite a few other features as well - as well as the very approachable price tag of free. With a properly named bucket and a DNS rule (possibly supplemented by some caching rules), you get an instant improvement to loading performance, with no downsides whatsoever.
Perhaps not as big of a difference as a full CDN would make, but hey, it’s free performance.

2 Likes

@codinghorror @sam I am using a cloudflare CDN to access the S3 content (where discourse uploads all the files) and not directly.

s3 mandates to use bucket name like files.discourse.org to access its content through cname files

http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html#VirtualHostingCustomURLs

Is there a way to allow periods in the S3 name (may be with a warning for https issue for direct s3 use). This is a showstopper in my case to use S3 with cloudflare and discourse.

Thanks

1 Like

We worked around that here @gingerman :

5 Likes

@fearlessfrog Thanks…I ended up doing exactly the same. I don’t know how I missed your thread earlier. Just to highlight - this approach has a limitation in terms of cache invalidation. we have to do separately invalidations in cloudfront and cloudflare (after rebaking posts). It would be great to automate them with a script at the very least.

I hit up on the same blocks eventually in terms of optimised image urls and was planning to start a thread today…Nice to know that you have already helped the team towards a fix in the below given thread. Kudos for that.

2 Likes

I just wanted to add my +1 for allowing periods in S3 bucket names within Discourse. AWS itself allows (indeed instructs) this now for CNAME-based access to S3 content, and provided the s3_cdn_url setting in Discourse is utilized so far as I can tell the previous issues encountered around HTTPS are rendered non-problematic, at least when services such as Cloudflare are used (as @uppfinnarn notes SNI-based SSL certs solve this issue).

Might it be possible to revisit this in an upcoming release?

Thank you for a great product!

Since we already support the S3_CDN_URL to be anything, what does we win by allowing this?

Hi Rafael, unless I’m missing something although that param allows Discourse to pull from a CDN URL, if we can’t specify an S3 bucket name with periods in, we can’t actually upload to those buckets? This necessitates hacky workarounds like setting up alternative CNAMEs in additional non-free third party services like CloudFront (which we’re not using at all at this time, and would prefer not to).

(And I guess ultimately I just don’t understand why you’d ever want to blanket reject something that is a valid S3 bucket name!)

So you don’t use https … at all? That’s the main reason per the above.

1 Like

I don’t understand what is the trade-off here.

I we allow buckets with dots, we must add a setting so you can disable SSL verification on connect, or people using it must add a proxy that adds SSL between Discourse and the S3 endpoint but it will still be plain HTTP between the proxy and S3 endpoint.

All that, and what is the benefit?

So you don’t use https … at all? That’s the main reason per the above.

Hi Jeff, we are using HTTPS, via Cloudflare.

I we allow buckets with dots, we must add a setting so you can disable SSL verification on connect, or people using it must add a proxy that adds SSL between Discourse and the S3 endpoint but it will still be plain HTTP between the proxy and S3 endpoint.

I’m pretty sure this would be picked up and handled by Cloudflare automatically without need to add an additional setting (it’s an approach we’ve used before with other services).

I can totally appreciate wanting to warn users off using a convention you think problematic, but given valid S3 bucket names can contain periods (and AWS docs advise using them in a range of scenarios) would it be something you might consider enabling in future?

No, we would not consider it.

Okay, well thanks. I guess we’ll patch or find an alternative :slight_smile:

What alternative? Just create a new S3 bucket without dots in it. There’s no reason I can think of to use an existing bucket for your Discourse uploads. It even makes accounting easier as there’s not random non-Discourse objects in there.

Hopefully something less suboptimal than this (but at least it’d work).

The CloudFront CDN is optional. You could use any other CDN you want with a no-dots bucket name and you wouldn’t have the HTTPS problems either.

Also, this image is the only thing I can think of while reading that exchange.

(Warning: anyone holding a current US security clearance should not reveal the image here.)

image

2 Likes

So, I also wanted to put my uploads behind CloudfFlare, using Wasabi instead of S3, so no CloudFront available.

I ended up directly modifying the DB, and restarting discourse and it worked. Also, does https between CDN and S3 really matter if the bucket is public read only?

UPDATE site_settings SET value = 'uploads.domain.tld' WHERE name = 's3_upload_bucket';

2 Likes