I’m working on a script that will evaluate participation in a discussion and produce a number based on how many messages they wrote, likes, and replies and such that will become students’ grade for “participating” in a discussion.
/t/blah/TOPIC_ID.json returns only 20 posts. Is there a way to get all of them, or will I need to do a request for all of them?
I looked a little at what gets passed to poll, but it wasn’t immediately apparent that I could somehow pass it something like a range or number of posts that I wanted.
Also its probably a good idea to still break up fetching all the posts into multiple requests rather than 1 big one. So if you have a topic with 100 posts. You should break it up into 5 smaller requests fetching 20 posts at a time.
I don’t know why i’m getting a response from /t/blah/TOPIC_ID.json with only the last 10 posts.
I see the chunk_size parameter (in response) set to 10, but i don’t know why is like that or where can i change this value.
I once wrote code that downloaded all of the posts in a topic.
I don’t know if the code still works, but it looks like how many posts you get is harder to predict than you might think. You pass the first post-id to control what posts you get.
The best way to download all the posts to a topic via the api is to mimic what Discourse is doing in the web browser so please checkout How to reverse engineer the Discourse API for details. Basically go the the topic in the browser open up your dev tools and look at the xhr requests as you scroll through the topic.
Here are the steps to download all the posts in the topic via the api:
Hit /t/-/{id}.json. This will contain a ‘posts_stream’ hash that contains a ‘posts’ array and a ‘stream’ array. The ‘posts’ array will give you the first 20 posts.
Now you need to loop through the ‘stream’ array which gives you all of the post ids in the topic. Remove the first 20 post ids from the stream (otherwise you are re-downloading them for no reason).
In chunks of 20 pass in all the post_ids to /t/{id}/posts.json like this: http://localhost:3000/t/8/posts.json?post_ids[]=46&post_ids[]=47&post_ids[]=48&post_ids[]=49&post_ids[]=50&post_ids[]=51&post_ids[]=52&post_ids[]=53&post_ids[]=54&post_ids[]=55&post_ids[]=56&post_ids[]=57&post_ids[]=58&post_ids[]=59&post_ids[]=60&post_ids[]=61&post_ids[]=62&post_ids[]=63&post_ids[]=64&post_ids[]=65
I agree with @blake steps for get all posts in a topic.
According with step 1
I just wanted to know if there are any parameter (maybe in the header of request) to set the chunk_size in /t/blah/TOPIC_ID.json request, because if i made a request from POSTMAN i get the first 20 posts as described previosly, but if i made the request using my web app using angular, i just get the first 10.
So i think there are something in the request that change the response from discourse server.
I use this request because from her i can get the post stream and the first 20 posts in one request. So i take her like my first request base to get all post in a topic.
I know this question is not critical, i can figure out to get my solution using multiple request. I am just curious to know why
That is the trick. I am running my app (actually is an ionic v3 app) from chrome dev tools on android device and i alwasy get the first 10. When i swtich to browser mode, i get the 20.
The ?print=true command is great, indeed! It seems however that there is a rate limit for ?print=true commands of five calls per hour. Is there a way to make more API calls per hour?
Alors que ?print=true fonctionne (ainsi que \u0026page=2), il semble y avoir plus de limites de débit que sans print=true. Je me demande combien de requêtes je peux faire, et considérées comme sûres, pour ne pas atteindre le statut 422.
J’essaie de lire environ 9000 articles, donc pour 20 articles à la fois, ce sera soit très lent, soit limité par le débit…
C’est dans mon UserScript. La coloration syntaxique pour typescript ne fonctionne pas ; et js fonctionne étrangement. Il faut que ce soit javascript. ts et typescript fonctionnent tous deux sur StackOverflow, cependant.
interface IPost {
id: number
username: string
post_number: number
cooked: string
}
interface ITopicResponse {
actions_summary: {}[]
archetype: string
fancy_title: string
title: string
post_stream: {
posts: IPost[]
stream: number[]
}
posts_count: number
reply_count: number
}
export async function jsonFetch<T>(url: string): Promise<T | null> {
const r = await fetch(url)
if (r.ok) {
const json = await r.json()
if (!json.errors) {
return json
}
}
logger('error', r)
return null
}
export async function fetchAll(urlBase: string) {
const r0 = await jsonFetch<ITopicResponse>(urlBase + '.json?print=true')
if (!r0) return []
const posts: IPost[] = r0.post_stream.posts
let page = 2
while (posts.length < r0.posts_count) {
const r = await jsonFetch<ITopicResponse>(
urlBase + '.json?print=true&page=' + page++
)
if (!r || !r.post_stream.posts.length) {
break
}
posts.push(...r.post_stream.posts)
await new Promise((resolve) => setTimeout(resolve, 1000))
}
return posts
}
fetchAll('https://community.wanikani.com/t/16404').then(console.log)
Dans un script ts-node utilisant Axios,
(node:1102374) UnhandledPromiseRejectionWarning: Error: Request failed with status code 422
Si j’attends assez longtemps, comme 10 minutes, cela échouera à la page 2 ; mais si je répète maintenant, je ne peux charger aucune URL.
Et sans print cela fonctionne OK.
En fait, j’ai réussi à le faire fonctionner en évitant print=true.
J’ai rencontré un problème où 'print' => true ne fonctionne pas, mais 'print' => 'true' fonctionne.
PHP Guzzle.
Peut-être devriez-vous également ajouter le gestionnaire print = 1
