NNTP support?

(Michael Burns) #1

Any more experienced devs care to weigh in on how difficult would it be to add something like NNTP support for Discourse? Natively, or maybe by way of a simple proxy.

Is the API complete enough to be able to attempt that, or would it take patching up Discourse itself?


(Waleed Khan) #2

I’ve never before heard of NNTP (probably because Usenet was before my time). Is it still used? Why would it be better than RSS?

(Michael Burns) #3

Usenet is still actively used, comparable to any of the largest forum websites (Google Groups, for instance), but federated and distributed. But granted, it is marginalized.

NNTP and RSS are not directly comparable. NNTP is more like a simplified email (IMAP) protocol, which includes replying to messages (something RSS doesn’t handle). There are advantages with accessibility and for users in bandwidth-limited or slow Internet areas (emerging markets).

This is probably not a ‘sexy’ feature request, but one I’m interested in helping provide.

(est) #4

It would be cool if both NNTP and POP will be supported. Or maybe SMTP?

(Johann Visagie) #5

You’re making me feel old. Merely due to knowing what you’re talking about.

(Kelly E Major) #6

I strongly prefer to use NNTP groups whenever possible vs. a web based forum. They are far more efficient for me to manage and process. I’ve been disappointed by the number of companies dropping NNTP for web only forums.

I think what you’re doing here is very interesting, I’m going to keep watching the progress of this.

  • KEM

(Johann Visagie) #7

Not to be rude (and believe me I’ve banged the drum of how web forums are in many ways inferior to Usenet or even mailing lists a lot), but you’re having that particular bit of angst about a decade and a half too late. That ship, I’m afraid, has long sailed. And circumnavigated the globe by now. :-/

(Valts) #8

[quote=“Mr_V, post:7, topic:1836”]web forums are in many ways inferior to Usenet or even mailing lists a lot[/quote]Out of curiosity - how can something be inferior than a mailing list?

(Johann Visagie) #9

Oh god, where to start?

Without starting to go into something I’ve already spent too much of my life going on about, let’s just say that I have entire communities going back to the early 90s, neatly archived and searchable here on my very laptop. An absolute breeze to pick out particular threads, or contributions by particular individuals.

On the other hand, I have spent hours creating content going into web-based forums just last year… content which has since disappeared into the ether.

Yes, it does offload much work onto the user. But I certainly see no way any web-based interface can even compete in the same ballpark as a decent MUA when it comes to archiving, organising and searching conversational information.

(Valts) #10

So… you have, like, the ENTIRE mailing list on your home PC? Tens of thousands of conversations, millions of messages? Isn’t that a bit… extreme?

(Johann Visagie) #11

Have you noticed how expensing storage ISN’T these days?

Also… text. Not big to start with; compresses really well.

(Valts) #12

That’s true, but we’re talking about the Internet here. You can’t expect to have an archive of every mailing list everywhere ever. Even if you could maybe compress it into a few terabytes of pure text. If you want the information to be truly available and searchable for everyone, you really do need to put it online for Google to see.

That said, an “export” feature might be interesting, but then again - for a medium sized community the archives for over 10 years will grow to a few hundred MB easily. Which isn’t a storage problem, but the bandwidth is, if everyone starts to use it. Maybe an “incremental export” so you can keep your home archive automatically up-to-date?

But then again, I’m not sure if a home archive is something worth pursuing. Seems pretty limited - only one person can use it.

Hmm… this brings up another idea, which is prefectly suited to try out the new “fork-a-thread” feature. :slight_smile:

(Johann Visagie) #13

Hence, a federated, distributed system like Usenet. It’s the logical next step. At least, it was… in the 80s.

Looking at my hard drive, I think you overestimate the size by about an order of magnitude.

I think you’ve just described a mailing list!

Mailing lists are definitely better if they have a searchable web archive. Which they pretty much all do nowadays.

Look, email has its problems, but people who know their tools, and who just want to exchange information have been and will continue to be using them for some time. Web forums are great for discussing videogames or… or transcendental snowboard yoga, or whatever. But if you look at any of the major open source software components underlying, say, Discourse… be it Postgres, Redis, Ruby or the Linux kernel… one guess how their developers communicate!

(Johann Visagie) #14


Isn’t Google Groups basically just a big Usenet archive with a web front-end? It came into existence when Google bought Usenet archive firm DejaNews, IIRC.

(Valts) #15

There’s definitely some major difference of perspectives between us…

When searching the web for information, I dread coming upon a mailing list archive, because it’s a total pain to find ANYTHING in there. Even if you find someone asking the same question, it’s often borderline impossible to find any answers. Sometimes the archive software just displays messages by time, so the question and the answer tend to have dozens of totally unrelated messages between them. When the software does attempt to group them by “thread” (probably meaning “subject line”), it often fails at it. And even where the grouping is correct, it’s still hard to dig anything out, because most replies tend to include the original text in full with the reply scattered inbetween. You have to look really carefully to find it.

So… how come you like this better than a web forum which has a much cleaner interface, much more organization between messages, and much better searchability (because you can search only certain categories and only find the messages that contain your keywords, not all the messages that followed it and included it in full)?

As for companies and information exchange - in everyday life I see email being used more as a chat than forum. Emails are passed back and forth, read once, and then forgotten. If there are some more special emails, you copy them to a special folder for quicker reference. You rarely search for anything older than a couple of weeks. You rarely search at all.

(Valts) #16

[quote=“Mr_V, post:13, topic:1836”]Hence, a federated, distributed system like Usenet. It’s the logical next step. At least, it was… in the 80s.[/quote]I’m unfortunately too young to have used Usenet. :frowning: So I don’t really “get it”. How is it different from modern forum systems? If anything, it’s actually more centralized, as I understand it, because there are just a few core servers that keep in sync all the time. Or, if I’ve misunderstood it, and there are thousands of decentralized Usenet servers, then how is it different from forums (in terms of information persistence)?

(Michael Burns) #17

Only partly. They do include the old Usenet index, and have an nntp portal for ongoing newsgroups for a subset of the total Google Groups lists.

Another huge part of Google Groups (and I think the default now, not positive) is actually not exposed to nntp at all, and they are just an internally hosted web forum (and mailing list).

(Johann Visagie) #18

I suspect it’s generational.

Even now, two seconds ago, I read your message perfectly formatted in my MUA (as forwarded there by Discourse), had a reply half-formed in my head… and then had that sinking feeling when realising I couldn’t just hit “reply” and compose my thoughts in my highly customised editor I have spent 20+ years learning the muscle memory for, but would instead have to click a link and go edit it in someone else’s idea of an HTML-based “editor”. (Even if it is Discourse’s better-than-average one.)

I’ll grant you, many MLMs have terrible indexing and search facilities for their archives. But… so do most web forums. Thank god for Google, eh?

I guess it comes down to your level of commitment to a particular community. If I’m just randomly using Google to find some arbitrary nugget of information coughed up by a community I’m not engaged in, it matters little whether the answer lies on a web forum or a web archive of a mailing list.

But when I am engaged in a community to the extent that I would subscribe to a mailing list or register for a web forum then, in the former case, my own local toolset takes over and I’m suddenly rocket-powered. I’m not going to go into another set of platform/tool wars, so I won’t name names, but a good, extensible, programmable MUA will give you all the power you need to organise and search a mailing list archive.

To recap: Yes, web-based list archives suck at representing the information. But they’re only ever used if you’re googling random bits of information on lists you’re not subscribed to.

Good archiving software — like a good MUA — should thread by Message-ID header.

Unfortunately many people these days are ignorant of how such things work, and instead of starting a new thread on a mailing list, they’ll sometimes hit “reply” on an existing thread and just change the Subject… leading to seemingly mis-threaded messages.

Applies equally to web forums, with the added annoyance of graphical doodads and half-page-long animated graphical signatures all over the bloody place. At least with a list I have the tools to collect, index and search textual data locally. Don’t confuse a mailing list as its web archive. That’s a purely incidental add-on.

(Heh, true story: A couple of years back I was heavily involved in a long-running, large web forum. And I got so irritated that I actually wrote a… thing… that would sit and scrape the forum for new post and deliver them to my inbox. I even got to the point where I could reply, and it would post back to the forum. It kinda 90% worked before I realised it was just too sad to live.)

Eh? Each time I’m faced by yet another technicolor bunch of blinking forum pages, I shudder. Aesthetic objections aside, there’s precious little consistency between web forum user interfaces. Frankly, it’s a mess — there’s no standard way of performing specific actions across forums. Or even a standard concept of what things you’d want to do.

A bunch of flat threads in a two-level hierarchy? With accumulated Message-ID headers you can mine as much structure out of email messages as you wish.

…and you can only search how and why the forum admin and/or forum software wants you to search, which is often severely limited. Unless you reach for teh Google.

Not sure where you work, but in all the companies, startups, NGOs and academic institutions I’ve ever worked in, email has been the core means of intra-org communication.

And that certainly remains the case for all the open source projects I’ve ever participated in. A biggish project tends to have a sprawling mass of mailing lists as its core means of communication between developers. Web forums are sort of thrown out as an afterthought to keep the end users happy and/or left to non-core contributors to run on their own initiative.

(…and I said I wouldn’t get into this argument again.)

(Johann Visagie) #19

Ah, right. Interesting.

(Michael Burns) #20

nntp can (optionally) be mirrored on the serverside, as opposed to a single, central webserver or mailing list server that has to handle 100% of the user traffic. The primary practical advantages with nntp are in clients however, and the screen reading, mass sorting/caching/searching and accessibility improvements for the user they bring.

That the server side can be more complexly designed (federated, mirrored) versus just a plain webserver is just gravy on top.