# Adsense ad crawler errors
**URL:** https://meta.discourse.org/t/adsense-ad-crawler-errors/178707
**Category:** Support
**Created:** [February 7, 2021, 10:30am UTC](https://meta.discourse.org/t/adsense-ad-crawler-errors/178707 "2021-02-07T10:30:59Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@TheBaby5](https://meta.discourse.org/u/TheBaby5)
#### Post date: [February 7, 2021, 10:30am UTC](https://meta.discourse.org/t/adsense-ad-crawler-errors/178707/1 "2021-02-07T10:30:59Z")
Using official ad plugin [Discourse Advertising Plugin (Ads)](https://meta.discourse.org/t/official-advertising-ad-plugin-for-discourse/33734)
The /new - /unread pages having crawler isssue, how to solve this?
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [February 7, 2021, 12:48pm UTC](https://meta.discourse.org/t/adsense-ad-crawler-errors/178707/2 "2021-02-07T12:48:56Z")
Can you share your url? Does your site require login?
---
### Author:  [@TheBaby5](https://meta.discourse.org/u/TheBaby5)
#### Post date: [February 7, 2021, 5:24pm UTC](https://meta.discourse.org/t/adsense-ad-crawler-errors/178707/3 "2021-02-07T17:24:19Z")
Actually, its our clients website, it does require login, but the website is accessible to all without login same as this forum [meta.discourse.org](http://meta.discourse.org)
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [February 7, 2021, 5:37pm UTC](https://meta.discourse.org/t/adsense-ad-crawler-errors/178707/4 "2021-02-07T17:37:49Z")
So if you’re not logged in you can see the topics at `mysterysite.com/new`?
---
### Author:  [@TheBaby5](https://meta.discourse.org/u/TheBaby5)
#### Post date: [February 7, 2021, 5:52pm UTC](https://meta.discourse.org/t/adsense-ad-crawler-errors/178707/5 "2021-02-07T17:52:46Z")
No, it is not visible!
> # Oops! That page doesn’t exist or is private.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [February 7, 2021, 8:37pm UTC](https://meta.discourse.org/t/adsense-ad-crawler-errors/178707/6 "2021-02-07T20:37:42Z")
So that’s what Google sees. I’m pretty sure that it’s frowned on to have content listed that’s not available, though the paywall sites do that to some degree.
Edit: I got this one wrong!
---
### Author:  [@michaeld](https://meta.discourse.org/u/michaeld)
#### Post date: [February 8, 2021, 6:28am UTC](https://meta.discourse.org/t/adsense-ad-crawler-errors/178707/7 "2021-02-08T06:28:05Z")
In order to determine what content is “new” or “unread”, the forum needs to know who you are, and thus requires a logged in user.
---
### Author:  [@TheBaby5](https://meta.discourse.org/u/TheBaby5)
#### Post date: [February 8, 2021, 7:15am UTC](https://meta.discourse.org/t/adsense-ad-crawler-errors/178707/8 "2021-02-08T07:15:53Z")
Any idea how to fix this?
---
### Author:  [@bartv](https://meta.discourse.org/u/bartv)
#### Post date: [February 8, 2021, 7:42am UTC](https://meta.discourse.org/t/adsense-ad-crawler-errors/178707/9 "2021-02-08T07:42:07Z")
Adsense will work fine without access to these two pages.
---
### Author:  [@michaeld](https://meta.discourse.org/u/michaeld)
#### Post date: [February 8, 2021, 9:18am UTC](https://meta.discourse.org/t/adsense-ad-crawler-errors/178707/10 "2021-02-08T09:18:09Z")
Shouldn’t these two be excluded in robots.txt?
---
### Author:  [@rlauriston](https://meta.discourse.org/u/rlauriston)
#### Post date: [July 17, 2023, 5:00pm UTC](https://meta.discourse.org/t/adsense-ad-crawler-errors/178707/11 "2023-07-17T17:00:04Z")
AdSense works but I get no revenue from the ads that would appear on that page. What values should I specify for **Login URL** and **Login method**?
