Category API request downloads all topics

Dev
1

I am creating analytics about the usage of our forum, and I download information about the topics of each category.

However, the request does not pull all the topics all the time. I only noticed this because it had previously pulled some topics that now disappeared from our latest report.

Example:

The topic is there when I open the category page in the browser.

The code I use:

from dotenv import load_dotenv
import os
import requests

load_dotenv()

headers = {
    "Content-Type": "application/json",
    "Api-Username": os.environ["USERNAME"],
    "Api-Key": os.environ["API_KEY"],
}

method = "GET"
url = "https://forum.aragon.org/c/proposals-for-main-dao/50.json"

response = requests.request(method, url, headers=headers)
topics = response.json()["topic_list"]["topics"]

found_topics = list()
for topic in topics:
    found_topics.append(topic["id"])

print(sorted(found_topics))

{topic: topic in found_topics for topic in [3519, 3528]}
>>>
[3075, 3224, 3299, 3312, 3449, 3472, 3474, 3526, 3554, 3555, 3557, 3583, 3586, 3592, 3613, 3624, 3626, 3659, 3674, 3677, 3680, 3694, 3695, 3699, 3702, 3710, 3716, 3720, 3725, 3737]

{3519: False, 3528: False}

Why is this happening, and how can I fix it?

2

Then you should use the data explorer plugin.

See Discourse Data Explorer and How to run Data Explorer queries with the Discourse API.

1 Like
3

Thanks, but, currently, we cannot install that plugin. Can we do anything to fix the API?

4

This is an official, stable & well-maintained plugin, what is preventing you from installing it?

The UI uses a paging system to pull down each set of topics. That’s probably why you can’t see them all.

Check the query your browser it’s sending for examples of each call (see network tab)

However as @pfaffman rightly points out this is not the most appropriate way of getting data for analytics as it’s optimised for UI.

1 Like