I am looking to semi-automate the process of turning a nominations thread into a table of nominations. I am using python (specifically the beautiful soup library) to parse the html. The nominations thread consists of a bunch of posts by users in which they link to a thread or topic that they like.…

You might want to use the API. Also, remember that almost all URLs will respond to a .json suffix. E.g. this thread https://meta.discourse.org/t/scrape-a-discourse-forum-with-python/98520.json

If access the raw markdown pages you won’t have to parse html and will have the URLs without accessing the counters. This topic is https://meta.discourse.org/raw/98520 . Edit: like others say, the API is really what you want. Here’s the scraper I wrote: GitHub - pfaffman/discourse-downloader: Downlo…

Thanks, all, your suggestions are very helpful! I particularly like that examples were offered in two languages. I’m going to go with the Python, but it’s cool to see the Ruby example as well.

Scrape a discourse forum with python

Development

mcmcclur (Mark McClure) October 2, 2018, 2:08pm 4

I also recommend that you use the API, rather than the HTML. That’s how I set up the archival tool described in this topic.

Topic		Replies	Views
Search by link Development search	7	72	December 7, 2024
Converting Wiki Posts to HTML Support	3	559	February 13, 2022
Grabbing Facebook reactions as posts Feature	8	1362	December 4, 2025
Converting links from raw markdown to HTML Development	2	1835	November 21, 2018
What method is used by Discourse to identify urls in post body Development	6	1151	December 12, 2017

Scrape a discourse forum with python

Related topics