Search Engine / No JavaScript version missing links

Contribute Feature

AI-generated summary

Search Engine / No JavaScript version missing links

The discussion revolves around the missing links on a Discourse site when browsing without JavaScript, similar to how Googlebot views the site. DeanMarkTaylor lists several missing links, including:

  • Links to user profiles
  • Link to categories list from homepage
  • Links to users activity from users page
  • Links to related topics at the bottom of a topic

DeanMarkTaylor notes that these missing links may hinder navigation for both users and search engines. However, codinghorror argues that these links are not necessary in the no-JavaScript version, as they are not relevant for search engines. Specifically, user profiles are not allowed to be indexed, according to robots.txt.

Mittineague agrees that links to user profiles should not be included, citing concerns about profile spammers. Mittineague also points out that robots.txt is not a foolproof way to prevent bots from accessing certain pages.

The discussion concludes with codinghorror emphasizing that Google is the primary source of incoming traffic, making other bots irrelevant in this context.

1

When browsing a discourse site (meta.discourse.org) without JavaScript in the same way as it is presented to Googlebot several links are missing.

  1. Links to user profiles
  2. Link to categories list from homepage
  3. Links to users activity from users page
  4. Links to related topics at the bottom of a topic.
  5. Any actual activity on users activity/topics page: Profile - sam - Discourse Meta (yes this looks the same as the users page because it is, no different content.)

EDIT: The idea being if a user would find it hard to navigate to / see related interesting info - so would a search engine.

Can we have a conversation about SEO?
2

that is just a random walk, I don’t see this as providing any value unless they are really related.

3

OK if it is just random - then ignore that point. I was imagining the topics where actually related i.e. StackOverflow style:

4

re #1
IMHO the links to member Profiles should NOT be there for search engines to find or follow, at least not until this happens.

There is enough of a problem with fly-by Profile SPAMmers already.

5

We should include incoming links, cause they are not in the body of posts.

1 me gusta
6

Irrelevant. Check robots.txt, user profiles are not allowed to be indexed anyway.

This also addresses your points #1, #3, and #5 @DeanMarkTaylor.

Suggested topics is indeed random once you’ve iterated through all the stuff that requires a user login, such as “is this new?” and “am I tracking this?”. So that rules #4 out as well.

I don’t think any of the links mentioned need to appear in the JS-off page.

7

Sorry, but you seem to be confusing “allowed” with “suggested” i.e.

There are two important considerations when using /robots.txt:

robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.

the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

So don’t try to use /robots.txt to hide information.

So though Google may not ignore the suggestion, other bots will, and robots.txt alone will not be enough to deter Profile SPAMners as indeed it hasn’t yet.

8

Google is 94.3% of all incoming traffic at Stack Overflow, for example. That’s not an exaggeration, those are the actual numbers.

These “other bots” are irrelevant.