Import Usenet group?

I’m thinking of setting up a local Discourse instance and importing some Usenet groups into it, just as a matter of having a personal archive of them. I’m not interested in trying to do a two-way mirror, or even necessarily trying to keep up with current topics in those groups (though if it could be readily accomplished I’d be interested in this). What I’m not sure of is how to do it. It looks like I can download messages from some of the groups I’m interested in via Google Groups, though they won’t have full email addresses (I’m not sure if this is going to cause problems in importing the messages)–and some of the other groups just don’t download.

As I think about the question, I realize that I have accounts with a couple of news servers. In principle, it seems it should be pretty straightforward, with the right software, to download all the articles into a .mbox file (or series of files)–but I’m not sure what that software would be. Any suggestions?

Edit: It looks like slrn (specifically, slrnpull) can handle the download, though it looks like each post will be in its own file. But I think the .mbox import script can handle that, albeit with a bit of work on the directory structure.

Yeah, that’s going to be a problem, because each message will be assigned to a new, different user. That’s why Importing mailing lists (mbox, Listserv, Google Groups, emails, ...) doesn’t allow that by default. You need admin rights on the Google Group to download the full messages including email addresses.

Yes, Importing mailing lists (mbox, Listserv, Google Groups, emails, ...) works with mbox files as well as each message in it’s own file.

2 Likes

Thought it might be. Turns out slrnpull is much faster than the Google Groups download–it’s getting 20-30 messages/second, rather than over a second per message. import_mbox.sh is running right now on some of the downloaded data, and so far it’s looking good.

1 Like

Yeah, if a group is available via NNTP then slrnpull is most likely the better alternative.

1 Like