Import Usenet group?

Product Support

AI-generated summary

In a Support topic titled “Import Usenet group?”, danb35 seeks advice on importing Usenet groups into a local Discourse instance for archival purposes, noting that Google Groups downloads lack full email addresses—which gerhard confirms would cause user assignment issues during import. danb35 discovers that slrnpull (part of slrn) efficiently downloads messages via NNTP, at 20-30 messages/second, and confirms that Discourse’s import_mbox.sh script can handle individual message files. gerhard agrees that NNTP via slrnpull is preferable when available. The discussion concludes with successful progress on the import, avoiding Google Groups’ limitations.

1

I’m thinking of setting up a local Discourse instance and importing some Usenet groups into it, just as a matter of having a personal archive of them. I’m not interested in trying to do a two-way mirror, or even necessarily trying to keep up with current topics in those groups (though if it could be readily accomplished I’d be interested in this). What I’m not sure of is how to do it. It looks like I can download messages from some of the groups I’m interested in via Google Groups, though they won’t have full email addresses (I’m not sure if this is going to cause problems in importing the messages)–and some of the other groups just don’t download.

As I think about the question, I realize that I have accounts with a couple of news servers. In principle, it seems it should be pretty straightforward, with the right software, to download all the articles into a .mbox file (or series of files)–but I’m not sure what that software would be. Any suggestions?

Edit: It looks like slrn (specifically, slrnpull) can handle the download, though it looks like each post will be in its own file. But I think the .mbox import script can handle that, albeit with a bit of work on the directory structure.

2

Yeah, that’s going to be a problem, because each message will be assigned to a new, different user. That’s why Migrate a mailing list to Discourse (mbox, Listserv, Google Groups, etc) doesn’t allow that by default. You need admin rights on the Google Group to download the full messages including email addresses.

Yes, Migrate a mailing list to Discourse (mbox, Listserv, Google Groups, etc) works with mbox files as well as each message in it’s own file.

3

Thought it might be. Turns out slrnpull is much faster than the Google Groups download–it’s getting 20-30 messages/second, rather than over a second per message. import_mbox.sh is running right now on some of the downloaded data, and so far it’s looking good.

4

Yeah, if a group is available via NNTP then slrnpull is most likely the better alternative.