For a completely private forum (no anonymous access), the lack of a secure media option in the default local storage configuration seems like a huge oversight to me. I wouldn’t be surprised if a lot of folks running private forums don’t realize that all of their post attachments are publicly accessible (if you know the URL). If my post’s text can’t be accessed anonymously, intuitively I would think the post’s attachments can’t be either.
Is it worth submitting an issue requesting this to see what sort of support it would get? Does one already exist? This one issue is currently a deal breaker for my small, poor group who is trying to spin up Discourse in an environment where we own the hosting hardware already and don’t want to pay for cloud storage (particularly at AWS S3 prices) just to make sure our media isn’t viewable anonymously.
Also, forgive the potential naivete of this question. But there is already support for distinct buckets for uploads and backups, right? Would it be difficult to support a third bucket for secure uploads? Public uploads go in the public bucket, secure uploads go in the secure bucket, and no individual object ACLS are needed. And if the security status of an object changes, it could just be moved between buckets?
I think it’s quite a bit of work. At least a day’s work, but maybe a week?
Cdck hosting, which pays for development, is run with uploads on S3, so it’s only login required self hosted sites without budget for S3 that’s interested.
If your users see sharing links to your assets they could just as well be downloading them to share them. It doesn’t seem like a big problem.
I’m not crazy pants for thinking it’s weird that post text is protected but attachments are not though, right?
There’s a big difference between intentional and accidental sharing though. Obviously, there’s no real way to prevent the former. But the latter can happen right now simply because people don’t understand the underlying tech. Consider email notifications for posts that include links to attached images getting innocently forwarded to non-members. An option to redact attachments in emails that’s not tied to the secure media functionality might be a good alternative there.
Even when people do intentionally share attachment links, they might have simply forgotten that it was from a protected category. But in an ideal world, the attachment link should be worthless to anyone without access to that category.
A current or future bug in one of the S3 implementations could also potentially allow for anonymous enumeration of the resources in a bucket or the ability to guess and brute force object URLs. In an ideal world, I’d want my on-prem S3 interface to be firewalled off from the internet at large so only the Discourse server can reach it directly.
Not crazy, implementing secure uploads for non s3 setups is certainly not something I am against building. It is just that we have no urgency or push to build the feature and the change is non trivial.
We occasionally have interns and audition projects, this is certainly a type of project that could fit into one of those buckets.
I’ve got a quick hack that I think would provide secure uploads for login-required sites.
Basically, you set up NGINX Docs | Authentication Based on Subrequest Result for uploads. The only issue is that I can’t find a URL that returns a 403/401 when login-required is turned on, so accessing an upload when you’re not logged in gives a 500 error. This would only happen if someone had an upload URL and tried to access it when they weren’t logged in, so it doesn’t seem that bad.
It’s something like this:
# JP
location = /auth {
internal;
proxy_pass http://discourse/categories;
proxy_pass_request_body off;
proxy_set_header Content-Length "";
proxy_set_header X-Original-URI $request_uri;
}
# END JP
location ~ ^/uploads/ {
auth_request /auth; #$JP
# NOTE: it is really annoying that we can't just define headers
# at the top level and inherit.
#
We have no plans to make distinct bucket functionality for secure uploads, nor do we have plans to make secure uploads work with non-S3 setups or setups without ACLs. We may look at doing this in the future if there is sufficient demand for this work to a point where it makes sense for us to devote time and resources to this effort.
why its using https://hashhackersforum.s3.us-east-2.amazonaws.com/optimized/2X/3/3204e85df407adfce19e105308248aee8b3b3f57_2_690x424.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAU5WBGG5Z7FNHQEIS%2F20210415%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210415T025617Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=15c9118929ccd9c24a9594ab02a47e900269b9d921d41679a317e29d6174c2bc
instead of https://forum.cdn.hashhackers.com/optimized/2X/3/3204e85df407adfce19e105308248aee8b3b3f57_2_690x424.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAU5WBGG5Z7FNHQEIS%2F20210415%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210415T025617Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=15c9118929ccd9c24a9594ab02a47e900269b9d921d41679a317e29d6174c2bc
I don’t mind ignoring it, but we have secure media enabled which means it’s not possible to use a CDN. This message probably shouldn’t be shown on sites with secure_media enabled.
Ah…yet another place that is making secure uploads that shouldn’t be. Badges should not be marked secure because they are essentially public images, like avatars, category logos, and other things. I will need to do a fix to make sure that badge images are not marked secure and a migration script that fixes ones that are already secure.
I am surprised this was working for you at all, nothing in 2.8.0 should have affected this.
Thank you for your reply. This is my first time using Ruby and I am delighted how clear this language can be. After some hours debugging I believe I found where to look. I guess I did the opposite you said I put some lines in the Badge model and now it can load the images. I also see that we have a flag for_site_setting there. I believe it relies on this information to adjust the ACL for objects on S3, and set false for that column.
app/models/badge.rb
def image_url
if image_upload_id.present?
return upload_cdn_path(image_upload.url) if !image_upload.url.include?(SiteSetting.Upload.absolute_base_url)
uri = URI.parse(image_upload.url)
Rails.application.routes.url_for(
controller: "uploads",
action: "show_secure",
path: uri.path[1..-1],
only_path: true
)
end
end
I will have a look what will change in the next upgrade to learn more about that.
Could you tell me what is the best version to use in production ?
Thank you!
I hope I can contribute more in the code base in the future.
The change you made to the badge model will certainly work, and it’s great that you were able to put that together the first time you used ruby! Though I will still go ahead and fix the core issue where badges shouldn’t be marked as secure.
Our tests-passed branch is the best version to use in production because that will get all the latest fixes as soon as they are available, though some people may prefer to stay on the beta branch which gets fixes and changes every few weeks or so.
I’ll let you know when I have the fix for badges being marked as secure done.
…but wonder if the issue is related to your comment here?
Is it supported to have users upload avatars when secure media uploads is turned on? At present I’m getting an error, but wonder if this is because avatars are uploaded to the same bucket where public access is not allowed…
1 Like
Canapin
(Coin-coin le Canapin)
Split this topic
126
I recently renamed “secure media” and related settings to “secure uploads” since it doesn’t just apply to images/video etc., and everyone generally refers to it as secure uploads anyway. The relevant core commit is here:
This OP of this topic has now been updated to reflect this.
I put some lines in the Badge model and now it can load the images. I also see that we have a flag for_site_setting there. I believe it relies on this information to adjust the ACL for objects on S3, and set false for that column.