API: Getting all posts in a topic

Development

AI-generated summary

Getting All Posts in a Topic via API

The discussion revolves around retrieving all posts in a topic using the Discourse API. The initial issue is that the /t/blah/TOPIC_ID.json endpoint only returns 20 posts.

@blake suggests using the stream array in the response to get all post IDs and then making a request to /t/{id}/posts.json with the post IDs to retrieve all posts. To avoid making a single large request, it’s recommended to break it up into smaller requests, fetching 20 posts at a time.

@jandres encounters an issue where the /t/blah/TOPIC_ID.json endpoint returns only 10 posts instead of 20. @blake explains that this is due to the slow_chunk_size being triggered, which can be avoided by passing print=true in the request, setting the chunk size to 1000.

Rate Limiting and Large Topics

@bjornekstrom notes that there is a rate limit of 5 calls per hour for the ?print=true command. @polv and @fifi_joon discuss the limitations of the ?print=true solution, including rate limiting and its ineffectiveness for “mega topics.”

@pfaffman advises handling rate limiting in the code, while @polv shares their TypeScript code for fetching all posts, which includes a 1-second delay between requests to avoid rate limiting.

Additional Solutions

@Daniil_Bazhenov shares a PHP Guzzle solution where passing 'print' => 'true' instead of 'print' => true resolves the issue. They suggest adding a print = 1 handler.

1

I’m working on a script that will evaluate participation in a discussion and produce a number based on how many messages they wrote, likes, and replies and such that will become students’ grade for “participating” in a discussion.

/t/blah/TOPIC_ID.json returns only 20 posts. Is there a way to get all of them, or will I need to do a request for all of them?

I looked a little at what gets passed to poll, but it wasn’t immediately apparent that I could somehow pass it something like a range or number of posts that I wanted.

2

Maybe the easiest way is to get the data I want from the data explorer plugin.

Now I’m thinking that it would be cool to write a plugin that showed people’s scores next to their profile pic in their posts in that topic.

3

When you GET /t/blah/TOPIC_ID.json the output will also contain a stream array that has all the id’s for the topic.

You then can call: /t/blah/TOPIC_ID/posts.json?post_ids[] and pass in all the ids from the stream array.

4

Thanks! And the problem with data explorer is that there is no way (that I can see quickly) to pass in the topic_id that I want.

5

Also its probably a good idea to still break up fetching all the posts into multiple requests rather than 1 big one. So if you have a topic with 100 posts. You should break it up into 5 smaller requests fetching 20 posts at a time.

6

Hi @blake @pfaffman

I don’t know why i’m getting a response from /t/blah/TOPIC_ID.json with only the last 10 posts.
I see the chunk_size parameter (in response) set to 10, but i don’t know why is like that or where can i change this value.

Thank you very much

7

I once wrote code that downloaded all of the posts in a topic.

I don’t know if the code still works, but it looks like how many posts you get is harder to predict than you might think. You pass the first post-id to control what posts you get.

See https://github.com/pfaffman/discourse-downloader/blob/master/discourse-downloader#L69

8

The best way to download all the posts to a topic via the api is to mimic what Discourse is doing in the web browser so please checkout Reverse engineer the Discourse API for details. Basically go the the topic in the browser open up your dev tools and look at the xhr requests as you scroll through the topic.

Here are the steps to download all the posts in the topic via the api:

  1. Hit /t/-/{id}.json. This will contain a ‘posts_stream’ hash that contains a ‘posts’ array and a ‘stream’ array. The ‘posts’ array will give you the first 20 posts.

  2. Now you need to loop through the ‘stream’ array which gives you all of the post ids in the topic. Remove the first 20 post ids from the stream (otherwise you are re-downloading them for no reason).

  3. In chunks of 20 pass in all the post_ids to /t/{id}/posts.json like this:
    http://localhost:3000/t/8/posts.json?post_ids[]=46&post_ids[]=47&post_ids[]=48&post_ids[]=49&post_ids[]=50&post_ids[]=51&post_ids[]=52&post_ids[]=53&post_ids[]=54&post_ids[]=55&post_ids[]=56&post_ids[]=57&post_ids[]=58&post_ids[]=59&post_ids[]=60&post_ids[]=61&post_ids[]=62&post_ids[]=63&post_ids[]=64&post_ids[]=65

9

Thanks @blake and @pfaffman for your soon response.

I agree with @blake steps for get all posts in a topic.

According with step 1
I just wanted to know if there are any parameter (maybe in the header of request) to set the chunk_size in /t/blah/TOPIC_ID.json request, because if i made a request from POSTMAN i get the first 20 posts as described previosly, but if i made the request using my web app using angular, i just get the first 10.

So i think there are something in the request that change the response from discourse server.

I use this request because from her i can get the post stream and the first 20 posts in one request. So i take her like my first request base to get all post in a topic.

I know this question is not critical, i can figure out to get my solution using multiple request. I am just curious to know why

10

For some reason your angular app is triggering slow_chunk_size.

from

So that might be something to look into.

There is not a chunk_size parameter you can set, but if you pass in print=true like /t/-/{id}.json?print=true it will set the chunk size to 1000.

11

Thank you so much :ok_hand:

That is the trick. I am running my app (actually is an ionic v3 app) from chrome dev tools on android device and i alwasy get the first 10. When i swtich to browser mode, i get the 20.

12

this saved my life !

13

print parameter just saved me :star_struck:

14

?print=true 명령은 정말 훌륭하네요! 다만 ?print=true 명령의 경우 시간당 5회 호출이라는 속도 제한이 있는 것 같습니다. 시간당 더 많은 API 호출을 할 수 있는 방법이 있을까요?

15

이 솔루션은 "메가 토픽"에서는 작동하지 않습니다. 이에 대한 해결책이 있으신가요?

16

그렇지 않다면, 나머지 데이터를 가져오려면 여러 요청을 보내야 합니다. 몇 개를 반환하나요?

17

?print=true는 작동합니다(&page=2도 마찬가지입니다). 하지만 print=true가 없을 때보다 레이트 리밋이 더 엄격하게 적용되는 것 같습니다. 상태 코드 422를 받지 않도록 안전하게 요청할 수 있는 횟수가 얼마나 되는지 궁금합니다.

약 9,000개의 게시글을 읽으려고 하는데, 한 번에 20개씩 요청할 경우 매우 느려지거나 레이트 리밋에 걸릴 수 있습니다…

18

속도 제한이 걸릴 수 있다고 가정하고 코드를 작성하는 것을 권장합니다.

19

It’s in my UserScript. typescript 언어 하이라이팅이 작동하지 않으며, js도 이상하게 작동합니다. javascript여야 합니다. 다만 StackOverflow에서는 ts와 typescript 모두 작동합니다.

interface IPost {
  id: number
  username: string
  post_number: number
  cooked: string
}

interface ITopicResponse {
  actions_summary: {}[]
  archetype: string
  fancy_title: string
  title: string
  post_stream: {
    posts: IPost[]
    stream: number[]
  }
  posts_count: number
  reply_count: number
}

export async function jsonFetch<T>(url: string): Promise<T | null> {
  const r = await fetch(url)
  if (r.ok) {
    const json = await r.json()
    if (!json.errors) {
      return json
    }
  }

  logger('error', r)
  return null
}

export async function fetchAll(urlBase: string) {
  const r0 = await jsonFetch<ITopicResponse>(urlBase + '.json?print=true')
  if (!r0) return []

  const posts: IPost[] = r0.post_stream.posts
  let page = 2
  while (posts.length < r0.posts_count) {
    const r = await jsonFetch<ITopicResponse>(
      urlBase + '.json?print=true&page=' + page++
    )
    if (!r || !r.post_stream.posts.length) {
      break
    }
    posts.push(...r.post_stream.posts)
    await new Promise((resolve) => setTimeout(resolve, 1000))
  }

  return posts
}

fetchAll('https://community.wanikani.com/t/16404').then(console.log)

Axios를 사용하는 ts-node 스크립트에서,

(node:1102374) UnhandledPromiseRejectionWarning: Error: Request failed with status code 422

충분히 오래 기다리면, 예를 들어 10분 정도면 2페이지에서 실패하지만, 지금 바로 반복하면 어떤 URL도 로드할 수 없습니다.

그리고 print 없이 하면 정상 작동합니다.

사실 print=true를 피함으로써 작동하게 만들었습니다.

export async function fetchAll(urlBase: string) {
  const r0 = await jsonFetch<ITopicResponse>(urlBase + '.json');
  if (!r0) return [];

  const stream = r0.post_stream.stream || [];
  const chunks: number[][] = [];
  while (stream.length) {
    chunks.push(stream.splice(0, 300));
  }

  const posts: IPost[] = [];
  let isContinue = true;
  while (chunks.length && isContinue) {
    const rs = await Promise.all(
      chunks
        .splice(0, 10)
        .map((ids) =>
          jsonFetch<ITopicPostResponse>(
            urlBase +
              '/posts.json?' +
              ids.map((id) => `post_ids[]=${id}`).join('&'),
          ),
        ),
    ).then((rs) =>
      rs.map((r) => {
        if (!r) {
          isContinue = false;
          return [];
        }
        return r.post_stream.posts;
      }),
    );

    rs.map((r) => {
      posts.push(...r);
    });

    if (chunks.length) {
      await new Promise((r) => setTimeout(r, 1000));
    }
  }
  posts.push(...r0.post_stream.posts);

  if (!isContinue) {
    logger(
      'error',
      `Total posts: ${r0.posts_count} != real count: ${posts.length}, due to Rate Limit?`,
    );
  }

  return posts;
}
20

'print' => true가 작동하지 않고, 'print' => 'true'만 작동하는 문제를 겪었습니다.
PHP Guzzle입니다.
print = 1 핸들러도 추가해야 할 것 같습니다.