If access the raw markdown pages you won’t have to parse html and will have the URLs without accessing the counters. This topic is https://meta.discourse.org/raw/98520.
Edit: like others say, the API is really what you want. Here’s the scraper I wrote: GitHub - pfaffman/discourse-downloader: Download a Discourse topic for offline analysis · GitHub