Scraping excerpts with latest.json (or plugin)

AI-generated summary

The discussion revolves around scraping excerpts from individual topics using the latest.json endpoint. furkle initially struggled to find the excerpt property in the topic objects, which Mittineague pointed out might be due to looking in the wrong place. However, furkle clarified that the desired excerpts are from individual topics, not category descriptions.

Mittineague suggested that the RSS feed might be the solution, as it contains excerpts, and furkle confirmed that this approach works, as their content is already syndicated through multiple RSS feeds.

Hi all,

I was looking at this post here by Daemon Labs, about scraping the latest.json endpoint and displaying this as a card layout. Simple, elegant, easy to follow. The problem comes in that I can’t figure out how to get the excerpt property to populate in the topic objects. It’s not that it’s empty, it’s that it doesn’t exist, and when I check the latest.json endpoint for this website (meta.discourse.org), I don’t see any excerpts there either.

I’m not sure whether this was an old feature that no longer exists (it’s still in the API, as far as I saw), or whether there’s some sort of configuration I missed to enable excerpts, or whether this is something I should be doing with a plugin that produces a new endpoint (e.g. latest-with-endpoints.json) with excerpts baked in.

Are you looking in the wrong place?

"unpinned":null,
"excerpt":"Support on configuring and using Discourse after it is up and running. For installation questions, use the install category.",
"visible":true,

That’s only the category description excerpt, no? What the Daemon post discusses is scraping excerpts from individual topics. Also, I’d like to be able to do this in a single request, without having to scrape multiple category topics or know which topics exist beforehand. Hopefully I’m not misunderstanding!

Hmm, I thought getting the Category excerpts was what it was intended to do. But looking again, you’re correct, it does mention topic excerpts.

I notice arguments are being passed into the function, but I don’t see where they are defined. I have a feeling I’m missing something obvious. It’s a heat wave here and that isn’t helping me think (that’s the excuse I’m sticking to)

One of the arguments is “feed” and I know excerpts are in RSS feeds, so unless there’s some magic happening that crucial bit is missing in the example?

As for getting what you want “in a single request” I guess that will depend on what you want.

Did you already try it?

Ah, somehow I entirely missed that there was an RSS feed. Should’ve expected as much! That’s actually easier as everything else is syndicated through several RSS feeds already.

Thanks a bunch!