The discussion revolves around challenges with pagination in the Discourse API, particularly when using search/query.json to retrieve posts within a date range. danielt initially struggled with pagination, noting that page 2 and page 3 returned identical results for complex queries, unlike simpler searches. vinothkannans clarified that the query string should use & instead of ? for subsequent pages, correcting the URL structure. pfaffman shared a Ruby script for downloading topics, though it doesn’t handle date filtering. danielt acknowledged the mistake in URL formatting and questioned how to access additional posts when more_posts is true. sam confirmed pagination support and referenced the Discourse API documentation. faheem asked about the meaning of term in the API, and blake explained that term refers to a search query, not a time period, and suggested reverse-engineering the UI’s API calls to understand pagination. faheem expressed a desire to fetch all topics in a category at once, but blake clarified that multiple API calls are necessary due to performance constraints, and suggested using data explore queries as an alternative.
I don’t know if it’ll help, but I wrote a little ruby program that will download all of a topic or category. (It doesn’t pay attention to dates, though.)
Oops. Getting the links wrong is what made me believe discourse support pagination using page=X in the first place (my own bad conversion from curl to a web browser).
Perhaps I should have ask a more general question: If search/query.json has a grouped_search_result with more_posts set true, how do I get to see the additional posts?
I’m trying to understand how to use the Discourse API.
In the URLs below, I’m guessing that term refers to a period or duration. This is based on erm=after:2016-01-01%20before:2017-01-01 etc. But I’m unclear what term=API refers to. Can anyone clarify?
There are actually two search endpoints you can hit /search.json?q= and /search/query?term=
term just means search term and does not refer to period or duration.
For understanding the search API it would be best to follow Reverse engineer the Discourse API and perform the searches you are intending to do via the API and see how the UI makes the same API requests.
Actually, I’d like the grab the entire contents of a page (in my case, a list of topics in a category, see Obtaining a list of topics from a category). I don’t want to go page by page if it’s not necessary. And I don’t think reverse engineering this would work, because Discourse just expands the page as you scroll down. There is no option that I am aware of to show the whole page. Is there a way to do this?
There is no single API call that will fetch ALL of the topics in the category. You will have to make multiple api calls. This is because a category could have 1 million+ topics and that could be a huge query and affect the performance of your site. When you reverse engineer the scrolling you can see how the discourse UI makes the api calls for more topics and you can replicate that behavior in your api.
Depending on what you are trying to achieve you can also use the data explore query to get the number of topics in a category and make an api call to that saved query.