About.html page prominently in search results, but missing in action

Product Support

AI-generated summary

A user, Krischan, reported that the about.html page is showing up in search results but is missing when trying to access it directly. IAmGav pointed out that Discourse does not use .html extensions in its URLs, and the correct URL is /about.

However, Krischan found that Google has indexed the about.html page, and IAmGav later discovered the page was accessible through the about.html URL, despite it not being a valid Discourse URL.

RGJ suggested that someone might have linked to /about.html, causing it to be indexed. Stephen noted that Discourse returns a 200 OK response for many invalid URLs with .html extensions.

eviltrout explained that this issue is related to how Rails routes work, allowing any endpoint to specify a :format parameter with an optional period, making about.html equivalent to /about. However, the about.html route does not exist in Ember, causing the server and client to disagree. A fix is possible but is not considered a high priority unless there is a pressing reason to address it.

1

image
image678×398 45.6 KB

There’s a link to the about.html, but it is missing - as it is here on Meta: About - Discourse Meta

Did I overlook a setting etc.? Thanks!

2

there is no .html ext in discourse.

About - Discourse Meta <— this is the correct url

3

Yes, thanks, but Google somehow found one. :man_shrugging:

4

i dont see it

image
image654×448 24.5 KB

5

That’s interesting :wink: I do and I can find it like this as well:

image
image708×402 36.9 KB

→ about.html

6

Found it.

image
image807×489 26 KB

maybe install the sitemap plugin

7

Thanks, but it is installed already.

8

The sitemap plugin does not even advertise the /about page.
All I can think of is that somewhere someone has been linking to /about.html

and Discourse is returning a 200 OK for it :scream: :scream: :scream:

image
image315×107 2.33 KB

image
image769×222 15.5 KB

9

They return a 200 response for lots of document names, despite them being invalid:

  • faq.html
  • categories.html
  • u.html
  • tags.html
  • top.html
  • badges.html

I was wondering if this just related to built-in routes, but it works for the cake day plugin

  • /cakeday/anniversaries/today.html

but not knowledge-explorer, which 404’s on:

  • /docs.html

but not:

  • badges.html

:thinking:

Wasn’t there a similar problem with a js file a week or two ago?

10

Hmm, should we be returning 200 for these unusual URLs @eviltrout?

11

This issue is related to how rails routes work. By default it allows any endpoint to specify a :format parameter with an optional period. So .html means HTML format, .json means JSON.

about.html is the same as /about (both request the HTML version). However the about.html route doesn’t exist in Ember because we never use routes with the HTML extension, so we get a disagreement from the back and front end. The server returns 200 but the client side then says, no wait that’s not found.

There are ways to fix this. We could refine our rails routes to make sure they don’t support .html but continue supporting .json. I don’t see a way to apply this globally but maybe there’s something I’m unaware of.

Honestly I’m not too excited by this work. Unless there’s a super pressing reason to fix I can’t see, it I think we can live with it.