Retrieve data (e.g. topic posts) via (OS independent) Go utility

Plugin Extras
1

‘discourse-reader’ retrieves data (e.g. site, category, topic) from Discourse forum on behalf of user. The utility can be found here:

discourse-reader                                                                       

Program:
  Name    : discourse-reader
  Release : v1.0.0 - 2022/11/18
  Purpose : Discourse Reader
  Info    : Retrieves data (e.g. site, category, topic) from Discourse forum.

Usage:
  discourse-reader -forum=string -query=string -category=int -topic=int -pages=int -output=string -userapikey -sleeptime=int

Examples for general query:
  discourse-reader
  discourse-reader -query=community.openstreetmap.org/site.json -output=community.openstreetmap.org.json
  discourse-reader -query=community.openstreetmap.org/site.json -output=community.openstreetmap.org.json -userapikey=bd38603815e3f2562c3eb3988c69eb77
  discourse-reader -query=meta.discourse.org/site.json -output=meta.discourse.org.json
  discourse-reader -query=meta.discourse.org/session/current.json -output=session-current.json

Examples for category:
  discourse-reader -forum=community.openstreetmap.org -category=56 -output=category-56.json
  discourse-reader -forum=community.openstreetmap.org -category=56 -output=category-56.json -userapikey=bd38603815e3f2562c3eb3988c69eb77
  discourse-reader -forum=meta.discourse.org -category=67 -pages=99 -sleeptime=6 -output=category-67.json

Examples for topic:
  discourse-reader -forum=community.openstreetmap.org -topic=4120 -output=topic-4120.json
  discourse-reader -forum=community.openstreetmap.org -topic=4120 -pages=99 -sleeptime=6 -output=topic-4120.json
  discourse-reader -forum=community.openstreetmap.org -topic=4120 --output=topic-4120.json -userapikey=bd38603815e3f2562c3eb3988c69eb77
  discourse-reader -forum=meta.discourse.org -topic=112837 -output=topic-112837.json

Options:
  -category int
    	retrieve data (list of topics) for category with identifier (default -1)
  -forum string
    	Discourse forum URL
  -output string
    	name of JSON output file
  -pages int
    	pages of data to retrieve (default 19)
  -query string
    	general data retrieve query (full URL)
  -sleeptime int
    	sleep time in seconds before retrieving the next page (avoids user rate limiting) (default 2)
  -topic int
    	retrieve data (list of posts) for topic with identifier (default -1)
  -userapikey string
    	personal user API key (can also be set as environment var 'USER_API_KEY')

Remarks:
  - User API key can be set as environment variable [USER_API_KEY].
  - Internet proxy can be set as environment variable [HTTPS_PROXY].
  - Examples for Linux:
    export USER_API_KEY=bd38603815e3f2562c3eb3988c69eb77
    export HTTPS_PROXY=http://user:password@194.114.63.23:8080
  - Examples for Windows:
    set USER_API_KEY=bd38603815e3f2562c3eb3988c69eb77
    set HTTPS_PROXY=http://user:password@194.114.63.23:8080

Rate limiting by forum service:
  - This program does functionally no different than a user via a browser. However, the
    data is retrieved somewhat faster. This can lead to rejections (rate limiting) by the
    service. To prevent this, the program can pause between fetching pages. The pause time
    can be specified with the option '-sleeptime=int'.
  - Typical user rate limit settings are:
    - requests per minute : 20
    - requests per day    : 2880
1 Like