I’m trying to allow Swifttype’s crawler (Swiftbot) to crawl our private discourse instance and they’re telling me I need to allow authentication access on discourse itself. Except I’m finding no documentation for such. For context, I have discourse as the forum for my main site, I’m trying to add…

I wanted to try that first. Can we edit the file?

[изображение] Joey_Tuan: Can we edit the file? AFAIK there should be no need to as long as your Security Setting is allow_index_in_robots_txt: true I’m wondering if it isn’t so much a crawl thing and that they need to be allowed in the robots.txt file or if it is a widget permissions thing. …

In this case, our discourse is actually private and not indexed at all by search engines, which is why we need to make an exception somewhere for this specific user-agent Swiftbot.

I think this would need to be a plugin. It might be easier to put the Discourse Search feature into your non-Discourse pages.

I don’t think we’ve ever encountered this situation before. You would need to authenticate the crawler as a regular user, probably through the Discourse API. Do we have any examples of this @techapj ?

[изображение] codinghorror: Do we have any examples of this @techapj ? Yes we do! To crawl the topics via Discourse API just provide your API key and username, and the requests made will be authenticated as regular user. For example: examples/example.rb main …

Do NOT use the key of an admin though. That would be incredibly dangerous! Just use the key of a trust level zero user. If the key gets out there, anyone can use it.

How to allow user-agent access to private discourse?

Support

Mittineague (Mittineague) 13.Ноябрь.2015 23:56:42 2

Wouldn’t this be done in your sites robots.txt file?

Тема		Ответов	Просм.
Robots.txt to completely block all indexing of private site Support	4	994	06.03.2020
Can't login after removing all bot user agents from site setting Support	5	981	24.11.2017
Discourse SEO overview (sitemap / robots.txt) Site Management seo , explanation	0	2587	18.10.2023
How to protect myself from bots crawling my Discourse instance? Support	4	1714	21.01.2019
Menu, title collumn, and a lot a stuff are gone Support	14	1371	11.12.2017

How to allow user-agent access to private discourse?

Связанные темы