Secure Uploads

I’m not crazy pants for thinking it’s weird that post text is protected but attachments are not though, right?

There’s a big difference between intentional and accidental sharing though. Obviously, there’s no real way to prevent the former. But the latter can happen right now simply because people don’t understand the underlying tech. Consider email notifications for posts that include links to attached images getting innocently forwarded to non-members. An option to redact attachments in emails that’s not tied to the secure media functionality might be a good alternative there.

Even when people do intentionally share attachment links, they might have simply forgotten that it was from a protected category. But in an ideal world, the attachment link should be worthless to anyone without access to that category.

A current or future bug in one of the S3 implementations could also potentially allow for anonymous enumeration of the resources in a bucket or the ability to guess and brute force object URLs. In an ideal world, I’d want my on-prem S3 interface to be firewalled off from the internet at large so only the Discourse server can reach it directly.

3 Likes

Not crazy, implementing secure uploads for non s3 setups is certainly not something I am against building. It is just that we have no urgency or push to build the feature and the change is non trivial.

We occasionally have interns and audition projects, this is certainly a type of project that could fit into one of those buckets.

9 Likes

Any thoughts on the feasibility of this as an easier path for secure media on not-AWS-S3 storage?

1 Like

I’ve got a quick hack that I think would provide secure uploads for login-required sites.

Basically, you set up NGINX Docs | Authentication Based on Subrequest Result for uploads. The only issue is that I can’t find a URL that returns a 403/401 when login-required is turned on, so accessing an upload when you’re not logged in gives a 500 error. This would only happen if someone had an upload URL and tried to access it when they weren’t logged in, so it doesn’t seem that bad.

It’s something like this:

# JP
    location = /auth {
        internal;
        proxy_pass http://discourse/categories;
        proxy_pass_request_body off;
        proxy_set_header Content-Length "";
        proxy_set_header X-Original-URI $request_uri;
    } 
    # END JP
    location ~ ^/uploads/ {

      auth_request /auth; #$JP
      # NOTE: it is really annoying that we can't just define headers
      # at the top level and inherit.
      #
2 Likes

We have no plans to make distinct bucket functionality for secure uploads, nor do we have plans to make secure uploads work with non-S3 setups or setups without ACLs. We may look at doing this in the future if there is sufficient demand for this work to a point where it makes sense for us to devote time and resources to this effort.

5 Likes

why its using https://hashhackersforum.s3.us-east-2.amazonaws.com/optimized/2X/3/3204e85df407adfce19e105308248aee8b3b3f57_2_690x424.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAU5WBGG5Z7FNHQEIS%2F20210415%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210415T025617Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=15c9118929ccd9c24a9594ab02a47e900269b9d921d41679a317e29d6174c2bc
instead of https://forum.cdn.hashhackers.com/optimized/2X/3/3204e85df407adfce19e105308248aee8b3b3f57_2_690x424.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAU5WBGG5Z7FNHQEIS%2F20210415%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210415T025617Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=15c9118929ccd9c24a9594ab02a47e900269b9d921d41679a317e29d6174c2bc

while i’ve entered CDN URL correctly.

It won’t use a CDN if you enable “Secure Media Uploads”.

5 Likes

One thing I’ve noticed is that the admin section warns that

The server is configured to upload files to S3, but there is no S3 CDN configured. This can lead to expensive S3 costs and slower site performance. See “Using Object Storage for Uploads” to learn more.

I don’t mind ignoring it, but we have secure media enabled which means it’s not possible to use a CDN. This message probably shouldn’t be shown on sites with secure_media enabled.

1 Like

A CDN can still be used for all the JS assets, making your site a lot faster to render for users around the world.

3 Likes

That’s cool. It wasn’t clear to me that discourse would treat the static assets differently. I think I need to read all the threads here again.

2 Likes

After I upgraded to 2.8.0.beta1 version, my badges using authenticated S3 Urls are not working. It was OK before the upgrade.

I checked the table uploads and the value for column secure is true.

Even after uploading a file using the option New Badge the image preview was kept empty.

image

No problems with other files using S3 url so far.

Any help? Thank you! :wink:

2 Likes

Ah…yet another place that is making secure uploads that shouldn’t be. Badges should not be marked secure because they are essentially public images, like avatars, category logos, and other things. I will need to do a fix to make sure that badge images are not marked secure and a migration script that fixes ones that are already secure.

I am surprised this was working for you at all, nothing in 2.8.0 should have affected this.

5 Likes

Hello @martin

Thank you for your reply. This is my first time using Ruby and I am delighted how clear this language can be. After some hours debugging I believe I found where to look. I guess I did the opposite you said :slight_smile: I put some lines in the Badge model and now it can load the images. I also see that we have a flag for_site_setting there. I believe it relies on this information to adjust the ACL for objects on S3, and set false for that column.

app/models/badge.rb

  def image_url
    if image_upload_id.present?
      return upload_cdn_path(image_upload.url) if !image_upload.url.include?(SiteSetting.Upload.absolute_base_url)
      uri = URI.parse(image_upload.url)
      Rails.application.routes.url_for(
        controller: "uploads",
        action: "show_secure",
        path: uri.path[1..-1],
        only_path: true
      )
    end
  end

I will have a look what will change in the next upgrade to learn more about that.
Could you tell me what is the best version to use in production ?

Thank you!

I hope I can contribute more in the code base in the future.

6 Likes

Hey @danilogit,

The change you made to the badge model will certainly work, and it’s great that you were able to put that together the first time you used ruby! Though I will still go ahead and fix the core issue where badges shouldn’t be marked as secure.

Our tests-passed branch is the best version to use in production because that will get all the latest fixes as soon as they are available, though some people may prefer to stay on the beta branch which gets fixes and changes every few weeks or so.

I’ll let you know when I have the fix for badges being marked as secure done.

5 Likes

I finished a fix for this issue today:

cc @danilogit

8 Likes

I just posted this,

…but wonder if the issue is related to your comment here?

Is it supported to have users upload avatars when secure media uploads is turned on? At present I’m getting an error, but wonder if this is because avatars are uploaded to the same bucket where public access is not allowed…

1 Like

After enabling secure media, I saw this:
“Page publishing cannot be enabled if secure media is enabled.” (when I tried to turn on page publishing in admin settings)

I currently need to use page publishing, so was surprised to see that secure media and page publishing seem to be incompatible.

This presents me with a dilemma:

  1. I want to be sure that links to file uploads in private categories are never indexed by search engines (and it seems that secure media is required for this), and also
  2. I want to use page publishing.

Is there a solution?

1 Like

Nothing is required for that. If the category is private search engines can’t see them. Secure uploads ensures that even with the link to the upload, someone cannot access the files.

3 Likes

Thanks. Yes I’ll have to consider if there are any other ways that a file upload could be picked up by search engines.

But is there a way to do secure media uploads while also having page publishing?

I recently renamed “secure media” and related settings to “secure uploads” since it doesn’t just apply to images/video etc., and everyone generally refers to it as secure uploads anyway. The relevant core commit is here:

This OP of this topic has now been updated to reflect this.

6 Likes