Secure Uploads

[martin (Martin Brennan)]

Added in the Discourse 2.4 release in February is the Secure Uploads feature, which provides a higher degree of security for ALL uploads (images, video, audio, text, pdfs, zips, and others) within a Discourse instance.

Prerequisites

You must have S3 uploads enabled on your site, which needs the following settings to be filled:

• S3 access key id
• S3 secret access key
• S3 region
• S3 upload bucket

You also must be using an S3 bucket that does not have a Public bucket policy, and you need to make sure that all existing uploads have a public-read S3 ACL. See the “Enabling Secure Uploads” below.

After these prerequisites are satisfied you can enable the “secure uploads” site setting.

Enabling Secure Uploads

HERE BE DRAGONS

This is an advanced feature and support outside of our Enterprise tier will be limited at best. Only enable secure uploads if you are an expert user.

---

To enable secure uploads, you need to follow these steps:

1. Ensure you have S3 uploads configured.
2. Take note whether your S3 bucket has a Public bucket policy. If it is, there is an additional step required (step 4).
3. Run the uploads:sync_s3_acls rake task. This will make sure all your uploads have the correct ACL in S3. This is important; if you do step 4 before doing this some uploads may become inaccessible on your forum.
4. Remove the Public bucket policy from your bucket if it was present in step 1.
5. Enable the “secure uploads” site setting. Optionally enable the “prevent anons from downloading files” site setting to stop anonymous users downloading attachments from public posts. Any uploads from this time on could possibly be marked as secure depending on the conditions below.
6. If you want all uploads retroactively to be analysed and possibly marked as secure, run the uploads:secure_upload_analyse_and_update rake task.

Note on S3 bucket policy

You need to make sure is that the bucket you are uploading to does not have a Public bucket policy. A public bucket policy will have something like this:

{
    "Version": "2012-10-17",
    "Id": "ComputedBucketPolicy",
    "Statement": [
        {
            "Sid": "AllowWorldRead",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::your-bucket-name/*"
        }
    ]
}

The important part here is that we are Allowing * to GetObject, which is saying let anyone download anything in the bucket. This label will also show if the policy is Public:

The settings here should not be touched. Pictured is the ideal state for the “Block public access” tab:

What it does

Once you have enabled Secure Uploads, any file uploaded via the Composer will either be marked as secure or not secure based on the following criteria:

• If you have the “login required” site setting enabled, all uploads will be marked as secure, and anonymous users will not be able to access it.
• If you are uploading something within a Personal Message, it will be marked as secure.
• If you are uploading something within a Topic that is inside a private Category, it will be marked as secure.

The upload on S3 will have a private ACL, so direct links to the file on S3 will throw a 403 access denied error. Any and all access to secure uploads will be via an S3 presigned URL. This will be hidden to your users though; if an upload is secure any reference to it will be made via the /secure-uploads/ Discourse URL.

Permissions and access control

The /secure-uploads/ URL will determine whether the current user is allowed to access the media and serve it if they are. When the upload is created, the post that it first appears in will be set to its “access control post” and all permissions will be based on that post.

• If you have the “login required” site setting enabled, anonymous users will always get a 404 error accessing the URL.
• If accessing media whose access control post is a Personal Message, the user must be a part of that Personal Message topic to access the media, otherwise the user will get a 403 error.
• If accessing media whose access control post is within a topic that is inside a private Category, the user must have access to that category to access the media, otherwise the user will get a 403 error.

Copying /secure-uploads/ URLs around between Posts and Topics is unwise, as different users will have different access levels within your Discourse forums. New uploads should always be created via the Composer. Oneboxes and hotlinked images will also respect the secure uploads rules. Site setting uploads, emojis, and theme uploads are unaffected by secure uploads, as they must be public.

If an access control post is deleted, the attached upload will no longer be accessible.

Moving posts with secure uploads

If you move an “access control post” between different security contexts then the upload attached can possibly be changed to secure or not secure. These are the situations which may change security for an upload:

• Changing a topic category. Will cycle through all posts in the topic and update upload security status accordingly.
• Changing a topic between being a public topic and personal message. Will do the same as above.
• Moving posts from a topic to a new or existing other topic. Will run the same as the above on the target topic.

Secure uploads in emails

Embedding secure images in emails is enabled by default. You can configure these site settings for further control:

• secure_uploads_allow_embed_images_in_emails : Disable this to redact secure images in emails.
• secure_uploads_max_email_embed_image_size_kb : The cap to the size of the secure image we will embed, defaulting to 1mb, so the email does not become too big. Max is 10mb. Works in tandem with email_total_attachment_size_limit_kb .

The secure images will be added as email attachments and embedded using the cid: url format because base64 URL support in email clients is still flaky.

If you don’t have secure_uploads_allow_embed_images_in_emails enabled, or if the images start to exceed the size limits, then this is what you will see in place of secure images (also secure audio and video which is not embedded):

Hosted Customers

At this time secure uploads is available to our enterprise customers only. Please contact us for more details.

[codinghorror (Jeff Atwood)]

There should probably be a lot of warnings around this feature @martin as it is an ADVANCED thing, not for the faint of , and there is a limit to how much we will support it outside our enterprise tier. Bring your own expertise…

[sam (Sam Saffron)]

The security was never intended to cover avatars, this is not a use case we planned for

[martin (Martin Brennan)]

Advisory: Setting the S3 bucket to “Block all public access” is not correct

@genachka @AntiMetaman @Hugh_Roberts @znedw @Thamer

I am going to amend the OP. After talking with our infrastructure team member @schleifer I confirmed that I was incorrect to advise that the “Block all public access” setting should be enabled. Turn this off for your S3 bucket and then run uploads:sync_s3_acls to ensure the ACLs are correct and then try again with the custom avatars.

An additional thing that everyone needs to make sure is that the bucket you are uploading to does not have a Public bucket policy. A public bucket policy will have something like this:

{
    "Version": "2012-10-17",
    "Id": "ComputedBucketPolicy",
    "Statement": [
        {
            "Sid": "AllowWorldRead",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::your-bucket-name/*"
        }
    ]
}

The important part here is that we are Allowing * to GetObject, which is saying let anyone download anything in the bucket. This label will also show if the policy is Public:

My apologies for this. @AntiMetaman neither @schleifer or I could reproduce this error:

@AntiMetaman

However, now if I try to upload anything it will give me this error:

It would be helpful if you could provide more information about your S3/AWS setup.

[AntiMetaman]

@martin Thanks. My issue wasn’t that error but anons unable to access the page. If I set the upload bucket to private, then I dont have that error anymore and I can upload. I don’t have login as required on my site.

I never had the “Block all public access” enabled for my bucket settings in the first place. If you are telling me that anons should still be able to access a topic, read it, and view secure images - then I can try this again. If adding security to media prevents anons from seeing those images, then I’d rather just secure attachments only.

If it helps, I am using BackBlaze B2 with BunnyCDN. My upload bucket settings are currently public:

[martin (Martin Brennan)]

@genachka

My concern is that having public access set on an S3 bucket is something drilled in to us as a very bad idea. So I want to double confirm that the secured uploads will be safe with this setting? So based on your current recommendation, the premise is that for the avatars to be accessible for upload by the user/admin – public access is required and the secured uploads are protected because they each get explicitly set an ACL to prevent being accessed publicly and that running uploads:sync_s3_acls will update the current ones there and any new uploads in topics will at that time have the necessary ACL set?

This is the gist of it yes. Anything that needs to be private will have a private ACL set and is inaccessible unless a presigned URL is used. Note that the bucket policy is not Public. And yes all those “Block public access” settings should be unchecked. If you are interested in how we determine whether an upload is secure all of the rules are here discourse/lib/upload_security.rb at main · discourse/discourse · GitHub and here discourse/app/models/post.rb at main · discourse/discourse · GitHub

@AntiMetaman

I never had the “Block all public access” enabled for my bucket settings in the first place. If you are telling me that anons should still be able to access a topic, read it, and view secure images - then I can try this again. If adding security to media prevents anons from seeing those images, then I’d rather just secure attachments only.

If it helps, I am using BackBlaze B2 with BunnyCDN. My upload bucket settings are currently public:

I am not familiar with BackBlaze sorry. I am not sure how the settings shown would translate to a bucket policy. Basically you do not want Public as the bucket policy, and you do not want to turn on “Block all public access”. That way we can set private and public ACLs appropriately. If you do not have login required then any upload made in a topic that is not in a PM or private category should be public and accessible by anons. The images should not be secure in a topic accessible by anons.

[riking (Kane York)]

The secure media implementation is not compatible with Backblaze. Do they support pre-signed URLs?

[AntiMetaman]

@riking

Yes, pre-signed URLs are supported: Does the B2 S3 Compatible API support Pre-Signed URLs? – Backblaze Help

@martin

Yes, changing the bucket from public to private does this. I did this and it allowed me to upload, but as an anon I couldn’t view topics.

[genachka]

@martin thanks for the clarification. And I can confirm that with leaving the setting on the S3 as in my screenshot as Public and everything unchecked (and suggested by you) that the custom avatars / profile images are finally working while the topic secured uploads remain working as well. Thank you!

[martin (Martin Brennan)]

I merged this PR this week and I am adding these details to the OP:

https://github.com/discourse/discourse/pull/10563

---

If you want to allow embedding secure images in emails you can configure these site settings:

• secure_media_allow_embed_images_in_emails : If enabled we will embed secure images in emails instead of redacting them.
• secure_media_max_email_embed_image_size_kb : The cap to the size of the secure image we will embed, defaulting to 1mb, so the email does not become too big. Max is 10mb. Works in tandem with email_total_attachment_size_limit_kb .

The secure images will be added as email attachments and embedded using the cid: url format because base64 URL support in email clients is still flaky.

If you don’t have secure_media_allow_embed_images_in_emails enabled, or if the images start to exceed the size limits, then this is what you will see in place of secure images (also secure audio and video which is not embedded):

[martin (Martin Brennan)]

Further addendum to my previous post; from this PR:

https://github.com/discourse/discourse/pull/10688

We now enable secure media image embedding by default.

[gerald]

After setting up secure media uploads (following the OP guide) everything works well (attachments, images…) except uploads that are not topic attachments (like site logo, profile avatar) those raise “Access Denied” pop up in Discourse.
Did i mess up some setting?

[michaeld (Michael - Communiteq)]

Please read the topic, your issue is addressed a few posts above this one.

[gerald]

I read everything carefully. Please let me know what exactly you mean by few posts above. I don’t see this issue.
Edit:
Trying to read between the lines here, i see that this thread used to be longer but some key replies were deleted from it, i see references to replies that don’t exist and mentions of screenshots that don’t appear in the thread.
If I understand correctly though, the recommendation is to set S3 bucket to NOT block any public access at all. I just want to confirm that, and ask if it is safe?

[michaeld (Michael - Communiteq)]

Correct.

The original post is indeed gone but the problem and solution are still there

@genachka

the premise is that for the avatars to be accessible for upload by the user/admin – public access is required

[gerald]

I noticed a bug with using secure media and Knowledge Base Plugin
Links to attachments in the knowledge base fail to open (redirect to 404 page) unless forced to be opened in a new window.
The extra weird thing is that the very same attachment can be opened without issue from the Discourse topic associated with the knowledge base item.

Edit:
The same bug happens when someone is copying a link to an attachment from one reply to another. the link is the exact same one naturally, but it only works from the original reply, unless forced to be opened in a new window.

[sbernhard (Bernhard Suttner)]

What about secure media if s3 is not used?
Currently it looks like that, if you have the direct link to the uploaded file, you can download that file without login. Which is like a security issue then…

[RGJ (Richard - Communiteq)]

@sbernhard

What about secure media if s3 is not used?

Secure media requires s3.
This entire feature was developed to make it impossible to share and access direct links.

[mcaruanagalizia (Matthew Caruana Galizia)]

This is an excellent feature. Thank you for developing it. Minio, a free drop-in replacement for S3, has no support for ACLs and never will. Their argument, which is a good one, is that ACL are not a useful feature and negatively affect performance because they require a second write operation. Secure media uploads will work with Discourse - the uploads:secure_upload_analyse_and_update task will error out on the final step but it seems you can ignore it. That said, is there any reason for Discourse to make ACL calls at all?

[martin (Martin Brennan)]

@mcaruanagalizia

That said, is there any reason for Discourse to make ACL calls at all?

Yes, because the S3 bucket is private based on the setup in the OP, any non-secure uploads need to have their ACL set to public, and secure uploads will have a private ACL so the S3 URL can be accessed directly. I do not think we have plans at this time to modify how this works; I think there would be quite a bit of work involved to avoid using ACLs altogether for S3 replacements.