Let googlebot see content, but make users log in

(jorgtron) #1

I want to require my users to login to view topics, but I assume that makes all content invisible to Google? Ideally the topics/posts would still be indexed by Google for their SEO value to help drive traffic, and then when a user clicks a result they would be taken to the front page where I can tell them why they should sign up.

Some more technical users will probably be able to spoof the googlebot useragent and get access to the content that way, but I’m sure that is a small percentage.

Any ideas on how I could implement this?


(Jeff Atwood) #2

This is not something that Google allows. If Google can index it, the assumption is that users can get to it by clicking through.

What you propose would get you delisted from Google.

(Michael - DiscourseHosting.com) #3

That’s what Quora does all the time.

But there’s also the First Click Free program, I think having support for this in Discourse would be a great idea.
Although I think that the proposed Referer: check is impossible in 2014 since Google uses https.


First Click Free

If you offer subscription-based access to your website content, or if users must register to access your content, then search engines cannot access some of your site’s most relevant, valuable content.

Implementing Google’s First Click Free (FCF) for your content allows you to include your restricted content in Google’s main search index. Our intention is to protect your content while allowing for its inclusion in Google’s search index. First Click Free has two main goals:

To include high-quality content in Google's search index, providing a better experience for Google users who may not have known that content existed.
To provide a promotion and discovery opportunity for webmasters of sites with restricted content.

To implement First Click Free, you need to allow all users who find a document on your site via Google search to see the full text of that document, even if they have not registered or subscribed to see that content. The user’s first click to your content area is free. However, once that user clicks a link on the original page, you can require them to sign in or register to read further. The user must be able to see the full content of a multi-page article. You can allow this by displaying all content on a single page to both Googlebot and users. Alternatively, you can use cookies to make sure that a user can visit each page of a multi-page article before being asked for registration or payment.

In order to successfully crawl your site, Google needs to be able to crawl your content without filling out a registration form. The easiest way to do this is to configure your webservers not to serve the registration page to our crawlers (when the user-agent is “Googlebot”) so that Googlebot can crawl these pages successfully. You can choose to allow Googlebot access to some restricted pages but not others. More information about technical requirements.

(Jeff Atwood) #4

Oh yeah, thanks, I was going to mention that program but couldn’t remember the right search keywords (irony?).

If you want to force a login dialog popup on arrival of all new users, that’s a different thing. Provided it is dismissable that could meet the Google guidelines – no different than any other popup you would show users on arrival (ads, etc).

(Jeff Atwood) #5

This is what Quora does. I would emulate this at your own risk…

(Kris) #6

It’s also the most annoying sleazy douchebaggiest behavior and everyone will hate you if you do what Quora does, whether or not Google delists you.

(Bill Ayakatubby) #7

Also, anyone with knowledge of browser dev tools can easily bypass it.

(Bcguy) #8

Isn’t this what the NY Times does also? (or something similar)…

(Jeff Atwood) #9

No, that is a paywall.

(Sam Saffron) #10

I would object quite violently to a quora like feature in core, its not behavior I would like to see emulated, build a plugin

(Mittineague) #11

Yes, please DO NOT.

Turning off CSS shows the content and the first time I went to quora was also my last.