# Migrating uploads to s3 does not work with private bucket

**URL:** https://meta.discourse.org/t/migrating-uploads-to-s3-does-not-work-with-private-bucket/94591
**Category:** Feature
**Created:** [August 13, 2018, 10:41am UTC](https://meta.discourse.org/t/migrating-uploads-to-s3-does-not-work-with-private-bucket/94591 "2018-08-13T10:41:15Z")
**Posts on this page:** 11
**Page:** 1

<div class="post-metadata">

### Author: ![ash-kadem](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/ash-kadem/32/106585_2.png) [@ash-kadem](https://meta.discourse.org/u/ash-kadem)
#### Post date: [August 13, 2018, 10:41am UTC](https://meta.discourse.org/t/migrating-uploads-to-s3-does-not-work-with-private-bucket/94591/1 "2018-08-13T10:41:15Z")

</div>

Hi all,

We want to use a CDN with Discourse using Cloudfront and S3. The S3 bucket is not public and can only be read by and written to using the Discourse IAM role or Cloudfront. However, if we migrate the existing uploads or try to upload new files we get Access Denied. This is because the bucket does not allow public-read files. I believe that the following piece of code prevents us from using the bucket.

> <https://github.com/discourse/discourse/blob/93dfc87b99eaee9e2bb64abf31ad54f8f1384a0b/lib/file_store/s3_store.rb#L35>

If I remove the piece of code `acl: "public-read"`, then the task `rake uploads:migrate_to_s3` works fine. 🙂

---

<div class="post-metadata">

### Author: ![schleifer](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schleifer/32/86416_2.png) [@schleifer](https://meta.discourse.org/u/schleifer)
#### Post date: [August 13, 2018, 6:47pm UTC](https://meta.discourse.org/t/migrating-uploads-to-s3-does-not-work-with-private-bucket/94591/2 "2018-08-13T18:47:43Z")

</div>

IAM roles and CloudFront are what we call “Advanced Clouds and Discourses”. The majority of installers aren’t going to be able to figure it out. It’s a reasonable setup, but out-of-scope for the intended purpose of that rake task. So I’d call this a feature request, not a bug.

> [@ash-kadem](#):
>
> remove the piece of code `acl: "public-read"`

If the bucket _is_ supposed to be public, then changing this will copy over a bunch of private objects and the posts will show broken links.

---

<div class="post-metadata">

### Author: ![ash-kadem](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/ash-kadem/32/106585_2.png) [@ash-kadem](https://meta.discourse.org/u/ash-kadem)
#### Post date: [August 13, 2018, 7:33pm UTC](https://meta.discourse.org/t/migrating-uploads-to-s3-does-not-work-with-private-bucket/94591/3 "2018-08-13T19:33:24Z")

</div>

Hi @schleifer, I thought about it after finishing the post and indeed it would be a nice feature if this was configurable. The bucket we have is only accessible by Cloudfront and the CDN that I wanted to use is the Cloudfront endpoint. Thus the links should work fine if we use the Cloudfront endpoint.

Would it be preferable to have this as another option to configure with `public-read` as default? I wouldn’t mind spending some time to whip up a pull request. 🙂

---

<div class="post-metadata">

### Author: ![schleifer](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/schleifer/32/86416_2.png) [@schleifer](https://meta.discourse.org/u/schleifer)
#### Post date: [August 13, 2018, 7:45pm UTC](https://meta.discourse.org/t/migrating-uploads-to-s3-does-not-work-with-private-bucket/94591/4 "2018-08-13T19:45:42Z")

</div>

Yeah, an argument to specify the acl – so you could call `rake uploads:migrate_to_s3[private]` – would work nicely and could keep the default as `public-read`.

---

<div class="post-metadata">

### Author: ![ash-kadem](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/ash-kadem/32/106585_2.png) [@ash-kadem](https://meta.discourse.org/u/ash-kadem)
#### Post date: [August 15, 2018, 7:42am UTC](https://meta.discourse.org/t/migrating-uploads-to-s3-does-not-work-with-private-bucket/94591/5 "2018-08-15T07:42:46Z")

</div>

@schleifer this should also work when users upload files, so I decided to implement it as a site setting 🙂

edit: by “should”, I mean I tested both scenarios + rake task on my s3 bucket and it works

[https://github.com/discourse/discourse/pull/6271](https://github.com/discourse/discourse/pull/6271)

---

<div class="post-metadata">

### Author: ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)
#### Post date: [August 22, 2018, 6:08am UTC](https://meta.discourse.org/t/migrating-uploads-to-s3-does-not-work-with-private-bucket/94591/6 "2018-08-22T06:08:30Z")

</div>

I am against a site setting here, an extensibility point so you can create a plugin to inject this is OK. But I do not want to complicate our already extremely complicated S3 site settings with yet another very extremely specific mega enterprisey kind of setting that even I am struggling really hard to get my head around.

---

<div class="post-metadata">

### Author: ![ash-kadem](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/ash-kadem/32/106585_2.png) [@ash-kadem](https://meta.discourse.org/u/ash-kadem)
#### Post date: [August 23, 2018, 5:26pm UTC](https://meta.discourse.org/t/migrating-uploads-to-s3-does-not-work-with-private-bucket/94591/7 "2018-08-23T17:26:32Z")

</div>

I haven’t had a look at developing a Discourse plugin yet, but I can work on a plugin that uses [Signed Cookies](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-cookies.html) to ensure that content can only be viewed by users of the forum. I’ll see what I can do.

---

<div class="post-metadata">

### Author: ![timkelty](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/timkelty/32/158090_2.png) [@timkelty](https://meta.discourse.org/u/timkelty)
#### Post date: [October 9, 2019, 4:23pm UTC](https://meta.discourse.org/t/migrating-uploads-to-s3-does-not-work-with-private-bucket/94591/8 "2019-10-09T16:23:18Z")

</div>

Also looking for a solution here.

A fairly standard (and recommended by AWS) configuration is to never have a public s3 bucket/objects, and if you need to serve objects publically, do it through a Cloudfront distribution.

The current s3 implementation forces public s3 bucket access (through this issue, as well as [S3 CDN URL ignored when uploading into posts](https://meta.discourse.org/t/s3-cdn-url-ignored-when-uploading-into-posts/54898)), which is recommended against - for security and costs.

---

<div class="post-metadata">

### Author: ![mahcr](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mahcr/32/206943_2.png) [@mahcr](https://meta.discourse.org/u/mahcr)
#### Post date: [March 1, 2021, 9:57pm UTC](https://meta.discourse.org/t/migrating-uploads-to-s3-does-not-work-with-private-bucket/94591/9 "2021-03-01T21:57:55Z")

</div>

Did you find a solution? I’m running into the same issue. Only uploading images is not working for and I think is related to this issue

---

<div class="post-metadata">

### Author: ![catchthewavecoke](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/catchthewavecoke/32/423120_2.png) [@catchthewavecoke](https://meta.discourse.org/u/catchthewavecoke)
#### Post date: [July 8, 2024, 12:37am UTC](https://meta.discourse.org/t/migrating-uploads-to-s3-does-not-work-with-private-bucket/94591/10 "2024-07-08T00:37:58Z")

</div>

I am grateful that this thread remains open, as historical context over time is important. (Often I find that relevant older posts are locked by moderators 🙂 )

Can Discourse support private S3 buckets? Feature request please 🙂

---

<div class="post-metadata">

### Author: ![mpalmer](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mpalmer/32/45740_2.png) [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [July 8, 2024, 1:02am UTC](https://meta.discourse.org/t/migrating-uploads-to-s3-does-not-work-with-private-bucket/94591/11 "2024-07-08T01:02:44Z")

</div>

Yes, Discourse can support private S3 buckets.
