Using official ad plugin Discourse Advertising Plugin (Ads)
The /new - /unread pages having crawler isssue, how to solve this?
סיכום מבוסס בינה מלאכותית
The discussion revolves around AdSense ad crawler errors on a client’s Discourse site, where the /new and /unread pages are inaccessible to crawlers due to login requirements. TheBaby5 reports that these pages return a ‘page doesn’t exist or is private’ error for non-logged-in users, which affects AdSense indexing. pfaffman clarifies that such restricted content may violate AdSense policies, though some paywall sites do it. michaeld explains that the forum requires login to determine new or unread content. bartv reassures that AdSense can function without access to these pages, while michaeld suggests excluding them in robots.txt. rlauriston adds that even if AdSense works, revenue may be lost if the pages are inaccessible, and asks about proper settings for Login URL and Login method.
Can you share your url? Does your site require login?
Actually, its our clients website, it does require login, but the website is accessible to all without login same as this forum meta.discourse.org
So if you’re not logged in you can see the topics at mysterysite.com/new?
No, it is not visible!
Oops! That page doesn’t exist or is private.
So that’s what Google sees. I’m pretty sure that it’s frowned on to have content listed that’s not available, though the paywall sites do that to some degree.
Edit: I got this one wrong!
In order to determine what content is “new” or “unread”, the forum needs to know who you are, and thus requires a logged in user.
Any idea how to fix this?
Adsense will work fine without access to these two pages.
Shouldn’t these two be excluded in robots.txt?
