Api to fetch topic by page

Development

AI-generated summary

Api to fetch topic by page

The discussion revolves around fetching a topic by page using the Discourse API. gchiu initially inquired about retrieving the next page of a topic, referencing the chunk_size of 20 in the response.

blake suggested that instead of making raw API calls, gchiu could utilize the post_stream part of the response and call the /t/{id}/posts endpoint to fetch the next page of posts. blake also recommended exploring the Chrome Dev Tools to understand how this works.

pfaffman shared a script that downloads all posts from a category or topic, mentioning that it was challenging to fetch all posts due to the payload sometimes including posts before the first requested one.

gchiu clarified their use case, using a topic to hold a change-log of commits from a GitHub project and wanting to find a specific post containing a commit hash. gchiu ultimately implemented a solution using the post_stream to search for the commit hash in the last 10 posts of the topic, sharing the code on GitHub.

1

I can see there is an api to retrieve a topic, and there is a chunk_size of 20 I see in the response.

How do I get the next page?

highest_post_number 33
deleted_by _
actions_summary [
    make map! [
        id 4
        count 0
        hidden false
        can_act false
    ]
    make map! [
        id 7
        count 0
        hidden false
        can_act false
    ]
    make map! [
        id 8
        count 0
        hidden false
        can_act false
    ]
]
chunk_size 20
bookmarked _
tags []
featured_link _
topic_timer _
message_bus_last_id 36
2

Are you trying to get all the posts that belong to a topic?

This would be a good feature to add to the discourse_api ruby gem rather than making raw api calls. There is no “Next Page” api call. What you will need to do to get the “next page” is to look at the post_stream part of the response:

image.png957×219 23.8 KB

And then you will need to call the /t/{id}/posts endpoint and pass in a list of post_id’s:

image.png916×150 29.6 KB

I would recommend opening up the chrome dev tools and looking at the xhr requests on a long topic here on meta to learn how this works.

3

I wrote a little script that you can pass a category or a topic and it’ll download all of the posts. It was a little finicky getting all of the posts, as it seemed that the payload (sometimes?) includes posts before the first on requested.

Sounds like a good idea. It might need to be limited to admins or something to protect from DDS attacks?

4

I’m using a topic to hold a change-log which is all the commits as they come in each day from our GitHub project. I want to find the post that contains a particular commit hash. So, I’d want to search all the posts for that topic in reverse date order to find the post.

I thought it might be too expensive to search on the hash itself and extract that post that is in the topic.

I can work with the post_stream I guess.

5

And I’ve cooked up something that works for me. It grabs the post_stream and then searches on the last 10 posts for the topic looking for my commit hash.

https://github.com/r3n/renclib/blob/master/scripts/changes.reb