"blocked onebox domains" setting not respected

dschaper · February 16, 2022, 9:35pm

blocked onebox domains are ignored in 2.9.0beta2 (c6265eec6b).

We have:

Those domains are behind OAuth walls and the onebox is trying to preview the domains and getting the OAuth login URL instead.

supermathie · February 16, 2022, 11:01pm

That setting only takes effect on the server — your browser will still attempt to do a onebox for the preview.

I can see how this might be unexpected.

If you submit the post, does it show the onebox after the server renders it, or does it show the bare URL as you expect?

dschaper · February 17, 2022, 12:08am

It shows as a onebox when the comment is posted:

supermathie · February 17, 2022, 12:14am

Yes, this doesn’t seem to be working properly…

nat · February 17, 2022, 1:09am

Can you try adding “auth.pi-hole.net” in your blocked onebox domains setting?

We’ve made this setting such that it follows redirects but only blocks the specified domain if it’s the final destination.

dschaper · February 17, 2022, 1:15am

Just tried and it still oneboxes it. I think because the auth.pi-hole.net endpoint is a uri-redirect from the OAuth endpoint on github.com. And we’d prefer not to block github.com from oneboxing.

Let me check and see if blocking github does anything…No, blocking github.com doesn’t preven oneboxing either. Is there a cache that onebox may be using that needs to be cleared so a new lookup is done for tricorder?

nat · February 17, 2022, 1:55am

Yes I can understand why.

There’s a cache for one day for fetched onebox URLs, so rebaking the post manually (post.rebake!) after one day would prevent the oneboxing with github.com blocked.

We have some work backlogged to be smarter about auth redirects, but that’s scoped to auth for Discourse forums only. I’m thinking perhaps expanding the setting to also allow subdirectories would help us with this case for github: https://github.com/login/oauth/authorize … Will discuss internally about this.

It’s probably also good to have the hint text be specific on what is actually blocked, perhaps an additional: “Onebox will follow redirects and block only if the final destination matches this setting”.

sam · February 17, 2022, 2:27am

Wild idea… should we support:

cache-control: no-store

The no-store response directive indicates that any caches of any kind (private or shared) should not store this response.

GitHub is specifically saying … please … don’t store me …

Maybe onebox should respect it … and not store it.

We could always make this an experimental setting to start and see what impact it has.

The preview would have to explain why we are not oneboxing.

Note… we do use no-store in quite a few places, but I think we redirect in those cases, yet there is this slight complexity.

github.com

discourse/discourse/blob/a0c65e91d91dc23f9fd99c9be1b4aa9a1c068782/spec/requests/topics_controller_spec.rb#L1868-L1874


      
          it 'correctly renders canonicals' do
            get "/t/#{topic.id}", params: { slug: topic.slug }
          
            expect(response.status).to eq(200)
            expect(css_select("link[rel=canonical]").length).to eq(1)
            expect(response.headers["Cache-Control"]).to eq("no-cache, no-store")
          end

dschaper · February 17, 2022, 8:15pm

We do link to a lot of PRs and Issues from our various repositories and having those oneboxed with the previews helps newer users see what the topic is about without having to get in to GitHub.

sam · February 17, 2022, 8:56pm

Yes totally get that, we are working on a solution, a day later I think looking at the no store header may be the best way forward

dschaper · February 28, 2022, 7:57pm

Sorry to bump this but it’s really affecting our support site. People post links to the tricorder URL that are troubleshooting logs for credentialed staff to look over. We get a number of these posts a day and we have to manually go in and change the URL to a non-link for now.

Any ideas on a timeframe to resolution or is it “soon™”

sam · February 28, 2022, 8:21pm

We will get this sorted, I would say some time in the next 4 weeks or so

GreenOWL · March 10, 2022, 3:21pm

How to do it in Discourse Docker?

sam · March 11, 2022, 12:22am

I recommend waiting on our fix to land, we have a PR in progress:

github.com/discourse/discourse

FIX: Apply onebox blocked domain checks on every redirect

discourse:main ← discourse:fix/onebox-check-blocked-domain-on-redirect

opened 12:09AM - 10 Mar 22 UTC

OsamaSayegh

+171 -38

The `blocked onebox domains` setting lets site owners change what sites are all…owed to be oneboxed. When a link is entered into a post, Discourse checks the domain of the link against that setting and blocks the onebox if the domain is blocked. But if there's a chain of redirects, then only the final destination website is checked against the site setting. This commit amends that behavior so that every website in the redirect chain is checked against the site setting, and if anything is blocked the original link doesn't onebox at all in the post. The `Discourse-No-Onebox` header is also checked in every response and the onebox is blocked if the header is set to "1". Additionally, Discourse will now include the `Discourse-No-Onebox` header with every response if the site requires login to access content. This is done to signal to a Discourse instance that it shouldn't attempt to onebox other Discourse instances if they're login-only. Non-Discourse websites can also use include that header if they don't wish to have Discourse onebox their content. Internal ticket: t59305.

It amends the behavior so:

We do the blocklist checking on every hop in a redirect chain
We introduce a new feature so any site can opt out of oneboxing by setting a customer header

This will resolve @dschaper issue and give us more flexibility in future.

Unfortunately going with no store would cast a way too wide net and catch too many false positives.

Osama · March 11, 2022, 6:50am

The PR has been merged and it’s in the tests-passed branch. Can you upgrade @dschaper and see if it fixes the problem? URLs with the tricorder.pi-hole.net domain should no longer onebox on your site.

dschaper · March 14, 2022, 10:15pm

Looks all good here!

Thanks for the fix everyone.

Topic		Replies	Views
Prevent to linkify when there is a redirect Feature	5	648	May 24, 2022
Google Maps Onebox data is not cached Feature	13	2193	November 2, 2019
Why are oneboxes cached into posts? Feature	9	1309	December 20, 2020
"blocked onebox domains" doesn't work Bug	5	504	January 23, 2023
Cross referencing a post doesn't create onebox Support	13	1351	June 12, 2017

"blocked onebox domains" setting not respected

Related topics