Using official ad plugin Discourse Advertising Plugin (Ads)
The /new - /unread pages having crawler isssue, how to solve this?
The discussion revolves around AdSense ad crawler errors on a clientâs Discourse site, where the /new and /unread pages are inaccessible to crawlers due to login requirements. TheBaby5 reports that these pages return a âpage doesnât exist or is privateâ error for non-logged-in users, which affects AdSense indexing. pfaffman clarifies that such restricted content may violate AdSense policies, though some paywall sites do it. michaeld explains that the forum requires login to determine new or unread content. bartv reassures that AdSense can function without access to these pages, while michaeld suggests excluding them in robots.txt. rlauriston adds that even if AdSense works, revenue may be lost if the pages are inaccessible, and asks about proper settings for Login URL and Login method.
Using official ad plugin Discourse Advertising Plugin (Ads)
The /new - /unread pages having crawler isssue, how to solve this?
Can you share your url? Does your site require login?
Actually, its our clients website, it does require login, but the website is accessible to all without login same as this forum meta.discourse.org
So if youâre not logged in you can see the topics at mysterysite.com/new?
No, it is not visible!
Oops! That page doesnât exist or is private.
So thatâs what Google sees. Iâm pretty sure that itâs frowned on to have content listed thatâs not available, though the paywall sites do that to some degree.
Edit: I got this one wrong!
In order to determine what content is ânewâ or âunreadâ, the forum needs to know who you are, and thus requires a logged in user.
Any idea how to fix this?
Adsense will work fine without access to these two pages.
Shouldnât these two be excluded in robots.txt?