Category API request downloads all topics

Development

AI-generated summary

Category API request downloads all topics

nocibambi is experiencing issues with the Category API request not pulling all topics, despite them being visible in the browser. The problem is likely due to the paging system used by the UI to pull down each set of topics.

nocibambi is using the following Python code to make the request:

from dotenv import load_dotenv
import os
import requests

# ... (rest of the code)

pfaffman suggests using the Discourse Data Explorer plugin for analytics, which is a more suitable solution.

merefield confirms that the issue is likely due to the paging system and recommends checking the query sent by the browser’s network tab for examples of each call. However, merefield also agrees with pfaffman that the Category API is not optimized for analytics and should not be used for this purpose.

nocibambi is unable to install the Discourse Data Explorer plugin at this time and is looking for a workaround to fix the API issue.

1

I am creating analytics about the usage of our forum, and I download information about the topics of each category.

However, the request does not pull all the topics all the time. I only noticed this because it had previously pulled some topics that now disappeared from our latest report.

Example:

The topic is there when I open the category page in the browser.

The code I use:

from dotenv import load_dotenv
import os
import requests

load_dotenv()

headers = {
    "Content-Type": "application/json",
    "Api-Username": os.environ["USERNAME"],
    "Api-Key": os.environ["API_KEY"],
}

method = "GET"
url = "https://forum.aragon.org/c/proposals-for-main-dao/50.json"

response = requests.request(method, url, headers=headers)
topics = response.json()["topic_list"]["topics"]

found_topics = list()
for topic in topics:
    found_topics.append(topic["id"])

print(sorted(found_topics))

{topic: topic in found_topics for topic in [3519, 3528]}
>>>
[3075, 3224, 3299, 3312, 3449, 3472, 3474, 3526, 3554, 3555, 3557, 3583, 3586, 3592, 3613, 3624, 3626, 3659, 3674, 3677, 3680, 3694, 3695, 3699, 3702, 3710, 3716, 3720, 3725, 3737]

{3519: False, 3528: False}

Why is this happening, and how can I fix it?

2

Then you should use the data explorer plugin.

See Discourse Data Explorer and How to run Data Explorer queries with the Discourse API.

3

Thanks, but, currently, we cannot install that plugin. Can we do anything to fix the API?

4

This is an official, stable & well-maintained plugin, what is preventing you from installing it?

The UI uses a paging system to pull down each set of topics. That’s probably why you can’t see them all.

Check the query your browser it’s sending for examples of each call (see network tab)

However as @pfaffman rightly points out this is not the most appropriate way of getting data for analytics as it’s optimised for UI.