Thanks for the explanation. This is starting to make more sense to me. I initially thought “allowed crawler user agents” was meant to identify allowed search engine crawlers. I initially entered “Googlebot” not understanding that WP Discourse used a Wordpress User Agent and became disallowed. Just one more newbie mistake. I should think this is a common misunderstanding but I found nothing in the limited documentation and more limited examples.
As “allowed crawler user agents,” I am now using Wordpress ( version not needed), Googlebot, and Discourse (added for use with cURL as below. Do You see a problem? Or do I need to add others?
Seems similar with cURL using Discourse API. I previously received 403 until I added the user agent as below.
First example published in Discourse API document (doesn't work for me):
curl -X GET "http://127.0.0.1:3000/admin/users/list/active.json" \
-H "Api-Key: 714552c6148e1617aeab526d0606184b94a80ec048fc09894ff1a72b740c5f19" \
-H "Api-Username: system"
This works for me after adding the user agent option.
curl -A Discourse -X GET "http://discourse.example/admin/users/list/active.json" \
-H "Api-Key: 714552c6148e1617aeab526d0606184b94a80ec048fc09894ff1a72b740c5f19" \
-H "Api-Username: system"
IMHO, examples shouldn’t be published unless they are complete and working…aargh.
@angus, many thanks for your help and assistance.