I am looking to semi-automate the process of turning a nominations thread into a table of nominations. I am using python (specifically the beautiful soup library) to parse the html. The nominations thread consists of a bunch of posts by users in which they link to a thread or topic that they like.…

You might want to use the API. Also, remember that almost all URLs will respond to a .json suffix. E.g. this thread https://meta.discourse.org/t/scrape-a-discourse-forum-with-python/98520.json

If access the raw markdown pages you won’t have to parse html and will have the URLs without accessing the counters. This topic is https://meta.discourse.org/raw/98520 . Edit: like others say, the API is really what you want. Here’s the scraper I wrote: GitHub - pfaffman/discourse-downloader: Downlo…

Thanks, all, your suggestions are very helpful! I particularly like that examples were offered in two languages. I’m going to go with the Python, but it’s cool to see the Ruby example as well.

Scrape a discourse forum with python

Ontwikkelaar

mcmcclur (Mark McClure) 2 oktober 2018 om 14:08 4

I also recommend that you use the API, rather than the HTML. That’s how I set up the archival tool described in this topic.

Topic		Antwoorden	Weergaven
Search by link Development search	7	105	7 december 2024
Converting Wiki Posts to HTML Support	3	576	13 februari 2022
Grabbing Facebook reactions as posts Feature	7	1363	26 januari 2015
Converting links from raw markdown to HTML Development	2	1871	21 november 2018
What method is used by Discourse to identify urls in post body Development	6	1167	12 december 2017

Scrape a discourse forum with python

Gerelateerde topics