Secure Media Uploads

Added in the Discourse 2.4 release in February is the Secure Media feature, which provides a higher degree of security for ALL uploads (images, video, audio, text, pdfs, zips, and others) within a Discourse instance.

Prerequisites

You must have S3 uploads enabled on your site, which needs the following settings to be filled:

  • S3 access key id
  • S3 secret access key
  • S3 region
  • S3 upload bucket

You also must be using an S3 bucket that is NOT Public, and you need to make sure that all existing uploads have a public-read S3 ACL. See the “Enabling Secure Media” below.

After these prerequisites are satisfied you can enable the “secure media” site setting.

Enabling Secure Media

:dragon: :warning: HERE BE DRAGONS :warning: :dragon:

This is an advanced feature and support outside of our Enterprise tier will be limited at best. Only enable secure media if you are an expert user.


To enable secure media, you need to follow these steps:

  1. Ensure you have S3 uploads configured.
  2. Take note whether your S3 bucket is Public. If it is, there is an additional step required.
  3. Run the uploads:sync_s3_acls rake task. This will make sure all your uploads have the correct ACL in S3. This is important; if you do step 4 before doing this some uploads may become inaccessible on your forum.
  4. Make your S3 bucket not Public if it was Public in step 1.
  5. Enable the “secure media” site setting. Optionally enable the “prevent anons from downloading files” site setting to stop anonymous users downloading attachments from public posts. Any uploads from this time on could possibly be marked as secure depending on the conditions below.
  6. If you want all uploads retroactively to be analysed and possibly marked as secure, run the uploads:secure_upload_analyse_and_update rake task.

What it does

Once you have enabled Secure Media, any file uploaded via the Composer will either be marked as secure or not secure based on the following criteria:

  • If you have the “login required” site setting enabled, all media will be marked as secure, and anonymous users will not be able to access it.
  • If you are uploading media within a Private Message, it will be marked as secure.
  • If you are uploading media within a Topic that is inside a private Category, it will be marked as secure.

The upload on S3 will have a private ACL, so direct links to the file on S3 will throw a 403 access denied error. Any and all access to secure uploads will be via an S3 presigned URL. This will be hidden to your users though; if an upload is secure any reference to it will be made via the /secure-media-uploads/ Discourse URL.

Permissions and access control

The /secure-media-uploads/ URL will determine whether the current user is allowed to access the media and serve it if they are. When the upload is created, the post that it first appears in will be set to its “access control post” and all permissions will be based on that post.

  • If you have the “login required” site setting enabled, anonymous users will always get a 404 error accessing the URL.
  • If accessing media whose access control post is a Private Message, the user must be a part of that Private Message topic to access the media, otherwise the user will get a 403 error.
  • If accessing media whose access control post is within a topic that is inside a private Category, the user must have access to that category to access the media, otherwise the user will get a 403 error.

Copying /secure-media-uploads/ URLs around between Posts and Topics is unwise, as different users will have different access levels within your Discourse forums. New uploads should always be created via the Composer. Oneboxes and hotlinked images will also respect the secure media rules. Site setting uploads, emojis, and theme uploads are unaffected by secure media, as they must be public.

:warning: If an access control post is deleted, the attached upload will no longer be accessible. :warning:

Moving posts with secure media

If you move an “access control post” between different security contexts then the upload attached can possibly be changed to secure or not secure. These are the situations which may change security for an upload:

  • Changing a topic category. Will cycle through all posts in the topic and update upload security status accordingly.
  • Changing a topic between being a public topic and private message. Will do the same as above.
  • Moving posts from a topic to a new or existing other topic. Will run the same as the above on the target topic.

Hosted Customers

At this time secure media is available to our enterprise customers only. Please contact us for more details.

31 Likes

First of all, congratulations to the Discourse team for this feature. Reading the topic I get some questions, maybe because this can be a WIP, or maybe I just wasn’t able to get them all.

  • This will be only for S3, or for S3-like uploads?
  • If I have a Secure Media, in a post, in a public Category, will the media show to the user, but unnaccesible otherwise?

The use case I’m trying to cover in my line of thinking, is having the media shown in my public topics, but if anyone tries to hotlink the media, access it by it’s URL, get the 404/403 error.

6 Likes

We use S3 ACLs, so we can only guarantee compatibility with AWS S3. Clones who implement the full feature set may be compatible, but the many providers out there may fail in subtle ways. Changes to accommodate those may be #pr-welcome.

I believe this is covered here:

Hot linking will result in errors loading images for users who lack the privilege.

10 Likes

There’s currently a problem with backups on Google’s S3 product as described here. I didn’t write, and haven’t looked at the code, but I’d definitely recommend that you use sho’nuff AWS S3 if you’re counting on this feature.

5 Likes

I was thinking in DO’s S3-like spaces. But I’d need to take a look if it’s even possible, or is a AWS S3 launch exclusive feature.

Other thing, for the Discourse team, is there a way to add all media as secure, although the forum is not “login required”?

I don’t really understand the use case here? Public and restricted are mutually exclusive.

4 Likes

To avoid hotlinking the media, initially. Maybe, having this feature, in a later date can be added some protected categories, where the content is public, but the images are login only.

For example, my community has a few valuable original tutorials, the images are really important to understand said tutorials. Therefore:

  1. The images are valuable to try to keep, as original content to the site (avoid hotlinking)
  2. Not valuable enough to hide behind a paywall, but to get users to sign in and participate in the community.
1 Like

This is an interesting use case, I could see this as being an additional setting alongside secure media. By default enabling secure media would get you all the rules above e.g. login_required, PMs, and private categories. Then you could have a “strict” setting that marks all media as secure no matter what, which will block anon users from accessing it as well as keeping the private category and PM restrictions.

I guess the only place this falls down is that we don’t have nice placeholders for images, videos, and audio that you are not allowed to see. Users will just see a broken piece of media.

7 Likes

By the way, I think Secure Media here in meta, broke the email templates.

I can’t see the Visit Topic button, that used to be there.

6 Likes

We are disabling this on meta for now, it does not really match our internal use case of meta.discourse.org as we use it for external customer support with staged user accounts, I am surprised though that this would cause it to break, we will investigate.

11 Likes

Hello,

I’ve enabled this feature but uploads are not using the secure-media-uploads link in private messages. Is there anything else I need to do in order to enable this? Like a rebake of all posts?

Our setup has CDN with Cloudfront and uploads configured with S3

1 Like

This feature does not retroactively apply to old uploads just by turning on SiteSettings.secure_media, only new uploads are affected. Rebaking the old posts will not work, as that does not change the upload security.

There is a rake task that you can run on your forum bundle exec rake uploads:ensure_correct_acl. What this does is:

  1. Sets an “access control post” for every media upload in your database, which will control who can see the upload
  2. Sets each upload in your database to secure/not secure based on the rules outlined in the OP
  3. Updates the ACL of the upload in S3 based on whether it is secure
  4. Rebakes all posts associated with the uploads to change the URL in the post

Keep in mind updating the ACLs on S3 can be quite slow. Also if your S3 bucket is public, the private ACL will not actually make the upload private on S3. I will add that information to the OP for future reference too. Here is what an example of the output of the task looks like:

Ensuring correct ACL for uploads in default...

There are 106 upload(s) with supported media that could be marked secure.

Marking 106 upload(s) as secure because login_required is true.

Finished marking upload(s) as secure.

Determining which of 106 upload posts need to be marked secure and be rebaked.

Determination complete!                  106 / 106 (100.0%)

Rebaking 122 posts with affected uploads.

Rebaking complete!                  122 / 122 (100.0%)

Done

If you want to turn off secure media completely for your Discourse forum, you can run bundle exec uploads:disable_secure_media. This will turn off the feature, mark all uploads as not secure, update the ACLs, and rebake the posts.

10 Likes

Thanks for this information, will try it and post it here. We have the bucket public, but I’m right now changing permissions to make it private.

Thanks again @mjrbrennan

3 Likes

Hello @mjrbrennan,

I ran the command and got an error NoMethodError: undefined method 'with_secure_media?' for nil:NilClass. Here’s the full log.

Ensuring correct ACL for uploads in default...

There are 1928 upload(s) with supported media that could be marked secure.

Marking posts as secure in the next step because login_required is false.

Determining which of 1928 upload posts need to be marked secure and be rebaked.

Updating ACL for upload.......        0 / 1928 (  0.0%)rake aborted!
NoMethodError: undefined method `with_secure_media?' for nil:NilClass
/var/www/discourse/lib/upload_security.rb:63:in `access_control_post_has_secure_media?'
/var/www/discourse/lib/upload_security.rb:49:in `uploading_in_secure_context?'
/var/www/discourse/lib/upload_security.rb:43:in `secure_media?'
/var/www/discourse/lib/upload_security.rb:25:in `should_be_secure?'
/var/www/discourse/lib/tasks/uploads.rake:783:in `block in determine_upload_security_and_posts_to_rebake'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:70:in `each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:70:in `block in find_each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:238:in `block in in_batches'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:222:in `loop'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:222:in `in_batches'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:135:in `find_in_batches'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/relation/batches.rb:69:in `find_each'
/var/www/discourse/lib/tasks/uploads.rake:778:in `determine_upload_security_and_posts_to_rebake'
/var/www/discourse/lib/tasks/uploads.rake:703:in `block (3 levels) in <top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:281:in `block in transaction'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:281:in `transaction'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/transactions.rb:212:in `transaction'
/var/www/discourse/lib/tasks/uploads.rake:668:in `block (2 levels) in <top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.0.7/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rails_multisite-2.0.7/lib/rails_multisite/connection_management.rb:73:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:660:in `block in <top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:ensure_correct_acl
(See full trace by running task with --trace)

I did a rebuild and I’m running the latest version (2.4.0.beta11). Not sure what this could be.

Thanks!

1 Like

This is a massive coincidence because I just got the same error. What is happening is that if an upload gets an access_control_post that is deleted our default scoping to exclude deleted posts makes upload.access_control_post equal nil.

I just did a hotfix to master to deal with this Work around deleted upload.access_control_post scoping issue · discourse/discourse@5dc6100 · GitHub. I will come back and work around this in a better way as well.

Edit: FYI the rake task now works as intended, however I have found a number of areas of improvement that I am going to start working on. It is up to you whether you run the task as is or wait for the improvements; they are mainly around efficiency and better messaging about what is going on and fixing a few little bugs.

5 Likes

Great, thanks for the quick fix!

I’ll give it a try, if it’s only better messaging I think it’s fine since it’s a one-off task.

I’ll post back the results here.

Cheers

2 Likes

Hey @mjrbrennan,

The rake task completed without errors! Here’s the log.

Ensuring correct ACL for uploads in default...

There are 1928 upload(s) with supported media that could be marked secure.

Marking posts as secure in the next step because login_required is false.

Determining which of 1928 upload posts need to be marked secure and be rebaked.

Determination complete!                 1416 / 1928 ( 73.4%)
Marking 896 uploads as secure because UploadSecurity determined them to be secure.
Finished marking uploads as secure.

Rebaking 10758 posts with affected uploads.

Rebaking complete!                10758 / 10758 (100.0%)

Done

When I started reviewing the uploads, I saw the images are working correctly (behind a secure-media-uploads link), but other file types are not getting replaced, like PDFs in our case.

For example this file doesn’t have a secure link but the ACL was changed for it.

image

The link looks like https://example.com/uploads/short-url/oWD....KXoV.pdf and when you click it you get the S3 access denied page

Any feedback would be appreciated.

Thanks!

2 Likes

That is very strange, because the rake task only targets media uploads specifically:

uploads_with_supported_media = Upload.includes(:posts, :access_control_post, :optimized_images).where(
  "LOWER(original_filename) SIMILAR TO '%\.(jpg|jpeg|png|gif|svg|ico|mp3|ogg|wav|m4a|mov|mp4|webm|ogv)'"
).joins(:post_uploads)

I guess it is possible that your had other non-media uploads on S3 had private ACLs before you changed the bucket from Public to Not-Public. What you can do to correct this is run the following in your rails console:

non_media_uploads = Upload.where(secure: false).where.not("LOWER(original_filename) SIMILAR TO '%\.(jpg|jpeg|png|gif|svg|ico|mp3|ogg|wav|m4a|mov|mp4|webm|ogv)'")
puts "Updating #{non_media_uploads.count} non-media upload ACLs..."
non_media_uploads.each do |upload|
  Discourse.store.update_upload_ACL(upload)
end
puts "Update complete!"

This will go through every non-secure non-media upload in your database and ensure the ACL is set to public-read. Note: this may take a while depending on how many uploads you have. What you can do is run Discourse.store.update_upload_ACL(upload) first for that specific PDF upload to make sure it resolves your issue before running it for all of them.

2 Likes

Is it possible to extend the secure uploads to other file types like pdf? The files we share in private messages sometimes contain personal information, so making every upload on PMs secure is something we would like to have.

Thanks!

2 Likes

I had a look into this, attachments like PDF and txt files already get marked secure if you have the “prevent anons from downloading files” site setting enabled.

However this is not correct, as uploads should only ever be marked secure in the database if the “secure media” site setting is enabled.

I have a fix for this here that I will merge in soon, which will make it so if you have secure media and prevent anon from downloading files enabled the PDFs etc. will be marked correctly as secure:

There will be some additional fixes I need to do to the rake task now as a result of this. Also we will discuss internally whether those attachments should have the secure media URL as well. Thanks for your feedback!

9 Likes