Obtaining a list of topics from a category

[faheem (Faheem Mitha)]

This is my first post on this forum.

forum.valuepickr.com is a Discourse forum for investing in the Indian stock market.

Consider the category of “Stock Opportunities”: Stock Opportunities - ValuePickr Forum

I would like a way to get a list of the topics in this category, preferably with some meta-information attached, like Replies, Views and Activity. Preferably as a CSV/TSV file or similar. Right now I don’t even see an obvious way to find out the number of topics.

I’m just a user of the Valuepickr forum. I am not a site maintainer, nor do I have any admin privileges.

I though initially that using the Discourse API might be a way to do so. However, I found understanding the API quite complex and confusing, I also don’t know whether this particular site supports API usage, and in any case, I’m not sure if the API even supports this use case.

Suggestions, thoughts, tips welcome. Thanks in advance.

[maiki]

You can use the API to get that information from public categories (a lot of it anyhow, I’m not sure if “activity” is timestamped in or not… but there is a lot of info).

For your example, start at https://forum.valuepickr.com/c/stock-opportunities.json.

Then use your tool of choice to convert to the data format you want.

Good luck, the rest is up to you!

[faheem (Faheem Mitha)]

Hi @maiki,

Thank you for the quick reply. It sounds like what I want to do is possible, at least.

I was just looking at what appears to be the JSON API documentation
Get a list of topics in the specified category, which seems to be in the right ballpark, at least. This looks to my eyes like it should correspond to the URL https://forum.valuepickr.com/c/stock-opportunities/11.json

This gives me something that appears superficially similar to the URL you posted (I think 11 is the id corresponding to that category, though I could be wrong about that.). Please excuse if I have got this wrong, because I have no experience with this kind of thing, and the documentation isn’t exactly chatty.

Obviously, I don’t know where you got the URL you posted from. If you could explain that, it would helpful.

However, in neither case is the JSON generated (54 K in each case) sufficient to contain the hundreds of listed topics in this category, though it clearly contains some of them. So, what am I missing?

[Monkey (EEE EEE OOOOH AAAH)]

These links should be able to give you more information:

Struggling with pagination within search/query.json

Oops. Getting the links wrong is what made me believe discourse support pagination using page=X in the first place (my own bad conversion from curl to a web browser). Perhaps I should have ask a more general question: If search/query.json has a grouped_search_result with more_posts set true, how do I get to see the additional posts?

Tutorial provided in that thread:

Reverse engineer the Discourse API

Discourse is backed by a complete JSON api. Anything you can do on the site you can also do using the JSON api. Many of the endpoints are properly documented in the discourse_api gem, however some endpoints lack documentation. To determine how to do something with the JSON API here are some steps you can follow. Example: recategorize a topic. Go to a topic and start editing a category: [image] Open Chrome dev tools, switch to the Network tab, select XHR filter: [image] Perform the op…

Following that turorial on the website you listed here gets you the following when you have network developer tools open:

Which has a lot of options when you right click it, Copy > Copy URL (in Firefox) provides:
https://forum.valuepickr.com/c/stock-opportunities/11/l/latest.json?ascending=false&order=default&page=1

An incremental json link for posts in that category.