Migrating uploads to s3 does not work with private bucket

ash-kadem · 8월 13, 2018, 10:41오전

Hi all,

We want to use a CDN with Discourse using Cloudfront and S3. The S3 bucket is not public and can only be read by and written to using the Discourse IAM role or Cloudfront. However, if we migrate the existing uploads or try to upload new files we get Access Denied. This is because the bucket does not allow public-read files. I believe that the following piece of code prevents us from using the bucket.

github.com/discourse/discourse

lib/file_store/s3_store.rb

93dfc87b9


      
          # options
          #   - filename
          #   - content_type
          #   - cache_locally
          def store_file(file, path, opts={})
            filename = opts[:filename].presence || File.basename(path)
            # cache file locally when needed
            cache_file(file, File.basename(path)) if opts[:cache_locally]
            # stored uploaded are public by default
            options = {
              acl: "public-read",
              content_type: opts[:content_type].presence || Rack::Mime.mime_type(File.extname(filename))
            }
            # add a "content disposition" header for "attachments"
            options[:content_disposition] = "attachment; filename=\"#{filename}\"" unless FileHelper.is_image?(filename)
            # if this fails, it will throw an exception
            path = @s3_helper.upload(file, path, options)
            # return the upload url
            "#{absolute_base_url}/#{path}"
          end

If I remove the piece of code acl: "public-read", then the task rake uploads:migrate_to_s3 works fine.

schleifer · 8월 13, 2018, 6:47오후

IAM roles and CloudFront are what we call “Advanced Clouds and Discourses”. The majority of installers aren’t going to be able to figure it out. It’s a reasonable setup, but out-of-scope for the intended purpose of that rake task. So I’d call this a feature request, not a bug.

If the bucket is supposed to be public, then changing this will copy over a bunch of private objects and the posts will show broken links.

ash-kadem · 8월 13, 2018, 7:33오후

Hi @schleifer, I thought about it after finishing the post and indeed it would be a nice feature if this was configurable. The bucket we have is only accessible by Cloudfront and the CDN that I wanted to use is the Cloudfront endpoint. Thus the links should work fine if we use the Cloudfront endpoint.

Would it be preferable to have this as another option to configure with public-read as default? I wouldn’t mind spending some time to whip up a pull request.

schleifer · 8월 13, 2018, 7:45오후

Yeah, an argument to specify the acl – so you could call rake uploads:migrate_to_s3[private] – would work nicely and could keep the default as public-read.

ash-kadem · 8월 15, 2018, 7:42오전

@schleifer this should also work when users upload files, so I decided to implement it as a site setting

edit: by “should”, I mean I tested both scenarios + rake task on my s3 bucket and it works

https://github.com/discourse/discourse/pull/6271

sam · 8월 22, 2018, 6:08오전

I am against a site setting here, an extensibility point so you can create a plugin to inject this is OK. But I do not want to complicate our already extremely complicated S3 site settings with yet another very extremely specific mega enterprisey kind of setting that even I am struggling really hard to get my head around.

ash-kadem · 8월 23, 2018, 5:26오후

I haven’t had a look at developing a Discourse plugin yet, but I can work on a plugin that uses Signed Cookies to ensure that content can only be viewed by users of the forum. I’ll see what I can do.

timkelty · 10월 9, 2019, 4:23오후

여기서도 해결책을 찾고 있습니다.

상당히 표준적이고 (AWS에서 권장하는) 구성 방식은 S3 버킷/오브젝트를 공개하지 않는 것이며, 공개적으로 오브젝트를 서빙해야 할 경우 CloudFront 배포를 통해 수행하는 것입니다.

현재의 S3 구현은 공개 S3 버킷 접근을 강제하고 있습니다(이 이슈뿐 아니라 S3 CDN URL ignored when uploading into posts 참조). 이는 보안과 비용 측면에서 권장되지 않는 방식입니다.

mahcr · 3월 1, 2021, 9:57오후

해결 방법을 찾았나요? 저도 동일한 문제를 겪고 있습니다. 이미지 업로드만 작동하지 않고, 이 문제와 관련이 있다고 생각합니다.

catchthewavecoke · 7월 8, 2024, 12:37오전

이 스레드가 계속 열려 있어 감사드립니다. 시간이 지나면서 과거의 맥락이 중요하기 때문입니다. (종종 관련 있는 오래된 게시글이 관리자들에 의해 잠금 처리되는 것을 발견하곤 합니다 )

Discourse가 비공개 S3 버킷을 지원할 수 있을까요? 기능 요청드립니다

mpalmer · 7월 8, 2024, 1:02오전

네, Discourse는 비공개 S3 버킷을 지원합니다.

글		댓글	조회수
S3 Storage with no Public access Feature s3	7	965	7월 8, 2025
Upload objects to private S3 is not working Product Support s3	5	2905	6월 14, 2022
Set up file and image uploads to S3 Self-Hosting configuring , how-to	35	121817	10월 25, 2025
S3 Bucket objects restricted access policy as per Discourses groups Product Support	2	947	10월 1, 2021
The bucket does not allow ACLs Product Support s3	4	2863	11월 3, 2022

Migrating uploads to s3 does not work with private bucket

Related topics