I’m trying to allow Swifttype’s crawler (Swiftbot) to crawl our private discourse instance and they’re telling me I need to allow authentication access on discourse itself. Except I’m finding no documentation for such. For context, I have discourse as the forum for my main site, I’m trying to add a site-wide search so users can search for discourse content from anywhere on my site. And this is what swifttype plugin does. Is there any way to do this within docker on digitalocean?

I wanted to try that first. Can we edit the file?

[图片] Joey_Tuan: Can we edit the file? AFAIK there should be no need to as long as your Security Setting is allow_index_in_robots_txt: true I’m wondering if it isn’t so much a crawl thing and that they need to be allowed in the robots.txt file or if it is a widget permissions thing.

In this case, our discourse is actually private and not indexed at all by search engines, which is why we need to make an exception somewhere for this specific user-agent Swiftbot.

I think this would need to be a plugin. It might be easier to put the Discourse Search feature into your non-Discourse pages.

I don’t think we’ve ever encountered this situation before. You would need to authenticate the crawler as a regular user, probably through the Discourse API. Do we have any examples of this @techapj?

[图片] codinghorror: Do we have any examples of this @techapj? Yes we do! To crawl the topics via Discourse API just provide your API key and username, and the requests made will be authenticated as regular user. For example: examples/example.rb main # frozen_string_literal: true $LOAD_PATH.unshift File.expand_path("../../lib", __FILE__) require File.expand_path("../../lib/discourse_api", __FILE__) config = DiscourseApi::ExampleHelper.load_yml client = DiscourseApi::Client.new(config["host"] || "http://localhost:3000") client.api_key = config["api_key"] || "YOUR_API_KEY" client.api_username = config["api_username"] || "YOUR_USERNAME" # get latest topics puts client.latest_topics To achieve the same via cURL request, pass api_key and api_username as param. For example: curl -X GET -d api_key="API_KEY" -d api_username="ADMIN_USERNAME" http://discourse.example.com/latest.json ``` Both of the above examples will fetch all the latest topics (even for private Discourse instance). @Joey_Tuan for more detailed API documentation, please see this topic: https://meta.discourse.org/t/discourse-api-documentation/22706?u=techapj

Do NOT use the key of an admin though. That would be incredibly dangerous! Just use the key of a trust level zero user. If the key gets out there, anyone can use it.

How to allow user-agent access to private discourse?

Product Support

Mittineague (Mittineague) 2015 年11 月 13 日 23:56 2

Wouldn’t this be done in your sites robots.txt file?

话题		回复	浏览量
Robots.txt to completely block all indexing of private site Product Support	4	1017	2020 年3 月 6 日
Can't login after removing all bot user agents from site setting Product Support	5	1008	2017 年11 月 24 日
Discourse SEO overview (sitemap / robots.txt) Site Management seo , explanation	0	2666	2023 年10 月 18 日
How to protect myself from bots crawling my Discourse instance? Product Support	4	1738	2019 年1 月 21 日
Menu, title collumn, and a lot a stuff are gone Product Support	14	1444	2017 年12 月 11 日

How to allow user-agent access to private discourse?

相关话题