Crawlable & Bulk open data / open apis


(Aiden) #1

Firstly YES! Discourse looks amazing. Secondly, im here to flag a vote for open data access. I think if a post/thread/poll/anything is in a public forum/thread, it should be as public to other program’s “eyes” as it is people’s eyes.

I noticed the .json API is very broad and very cool … I hope this feature stays for the long-run (not just as a side-effect of the system being JS driven). Open (no-auth required, machine readable) data as well as open source is, imvho, hugely important to the web in preventing non-spiderable walled-gardens/silos such as FB/Twitter, encouraging innovation, integration and unlocking the potential of stored data from human interaction.

Just my 2p :smile:


(Robin Ward) #2

One of the main reasons I pushed for a Javascript client is because I love the idea of API first development.

I personally think having a battle-tested API is a huge advantage in the forum space, so I don’t think you have to worry about us losing it.


(Aiden) #3

I think the main issue is the potential to slip in to data lockdown (as an example, Twitter has a well rounded API and JS interface but crappy access / rate limiting / etc ). Creating a wild-west for programs to munch forum data the same way search engines crawl HTML is an exciting prospect (to me at least).

Just casting my vote for public data (webhooks, rss feeds, open json apis, bulk downloads)


(Kane York) #4

I think you’d be glad to hear that, in my opinion, this has worked out just fine. As long as you properly rate-limit your crawler, no Discourse forum should have any (technical) problem with it.

(There may be the standard political issues about “hm, what have I said that I maybe don’t want kept around forever”)