Update to Google technical Webmaster Guidelines

(Dean Taylor) #1

Blog post today by Google:

Obviously Discourse takes specific actions delivering different content to Googlebot - I thought I would post this here for developers to review.

We recently announced that our indexing system has been rendering web pages more like a typical modern browser, with CSS and JavaScript turned on. Today, we’re updating one of our technical Webmaster Guidelines in light of this announcement.

For optimal rendering and indexing, our new guideline specifies that you should allow Googlebot access to the JavaScript, CSS, and image files that your pages use. This provides you optimal rendering and indexing for your site. Disallowing crawling of Javascript or CSS files in your site’s robots.txt directly harms how well our algorithms render and index your content and can result in suboptimal rankings.

(Jeff Atwood) #2

We do not disallow crawling of the JavaScript and CSS, so there is nothing to worry about here.

(Michael Downey) #3

I haven’t looked, but I wonder if this line would cause any problems?

Disallow: /assets/browser-update*.js

(Kane York) #5

That’s intentional, to make sure that “Your browser is too old…” never shows up in the search results. (Though I think it still did, it only went away when the js was changed to append to the end of the DOM)