Migrate a mailing list to Discourse (mbox, Listserv, Google Groups, etc)

@gerhard Can your mbox importer be used only when you first install Discourse or can it be used later after other users are using Discourse? If the importer is used when Discourse is being used by others will they see any side effects?

1 Like

In order to get the importer to scrape messages from Google Groups, I had to reverse this change in /script/import_scripts/google_groups.rb

I put the line

    wait_for_url { |url| url.start_with?("https://accounts.google.com") }

back to

    wait_for_url { |url| url.start_with?("https://myaccount.google.com") }

Otherwise, I would get this message every time:

Logging in...
Failed to login. Please check the content of your cookies.txt

@gerhard I noticed after the import that although the messages look okay, there are no staged users at all, even though it seems like there should be (I used the default staged: true). The outputs look like:

indexing replies and users

creating categories
        1 / 1 (100.0%)  [13440860 items/min]  
creating users

creating topics and posts
     7399 / 7399 (100.0%)  [1421 items/min]     

Is there supposed to be a user counter shown as well?

I also tried running with staged: false and the same output was shown, and none of the mailing list users are in any groups. In case it helps to see what’s actually being processed, here is one of the many .mbox files that is being imported:

2020-December.zip (49.5 KB)

The only non-default setting was adding:

  "Mne_analysis": "mne_analysis"

It would be great to have these users show up as staged so that they can claim their old posts when signing up, so any tips or ideas appreciated!

1 Like

It should probably just accept both of those?

Have you had a look at the database? Gut feeling on this issue is for some reason the email field is not getting created correctly there and thus can’t be read.


See 2.3 in the OP for checking the index database.


The Mailman 2 list that I am considering importing into Discourse has had (for part of its existence) from_is_list set to Munge From, so that the “From:” header is

From: Listname <listname-bounces@listdomain.com> On Behalf Of [Original sender's name]

instead of

From: [Original sender's name] <username@example.com>

This made me think the importer would import each of these messages as if from the same user (with email address listname-bounces@listdomain.com)… BUT…

The initial line marking the beginning of a new email in the mbox file still begins with

From username@example.com [Date time group]

(and the Hyperkitty archives show the original sender’s email address as normal also).

So my question is – does the importer script take the sender’s address from the “From:” header or the "From " line? Thank you.

I discussed this briefly in a previous topic: Working on a mailman2 to discourse migration script - #10 by dachary

1 Like

It’s using the From: header.

1 Like

Thanks for the quick reply! How hard would it be to change this? Not necessarily officially – though it might help others – but just for me to change the script before running it. I don’t know any Ruby (yet!) but if it’s just changing a colon to a space…

It’s not a simple change, but it should be doable. You don’t necessarily have to implement it in the import script. If you know another scripting language, I’m sure it won’t be too hard to update the From: headers in the mbox files before running the import…

But, feel free to fix it in the import script. A PR is welcome!
A good starting point for fixing the header should be the each_mail method…


Cheers. Looks like this is what currently decides it, from line 69-70 of indexer.rb:

parsed_email = receiver.mail
from_email, from_display_name = receiver.parse_from_field(parsed_email)

Would it be possible at that point to obtain the first line of the mbox email (i.e. the “From [email address] [date time]” line) from parsed_email and extract the email address from that?


No, that line is filtered when the mbox is split into individual messages. You need to save that value in the each_mail method in order to use it later.


I had some fun trying to do this, before spotting that Mailman stores the emails in the mbox in their original, unadulterated form, so that the “From:” line contains the same (original sender’s) email address as the "From " line in all cases, even when the email has been sent “From: listname-bounces@listname.domain.com”). :man_facepalming:

I was limited by not having a development Discourse installation, or even Ruby, but was able to make some headway with https://rubular.com/ and https://replit.com/languages/ruby (and DuckDuckGo). If you would be willing to have a look at it, I’d be grateful if you’d let me know whether this would have worked (or nearly worked) had it been necessary…

    def each_mail(filename)
      raw_message = +''
      first_line_number = 1
      last_line_number = 0

      each_line(filename) do |line|
        if line.scrub =~ @split_regex
          if last_line_number > 0
            #We're at the start of the NEXT email now
            yield raw_message, first_line_number, last_line_number
            raw_message = +''
            first_line_number = last_line_number + 1
            #We're at the start of THIS email now, so get the email address 
            new_email = line.match(/^From (\S+@\S+).*/).captures
          raw_message << line

        last_line_number += 1

      #Get old email ("From:" line) 
      old_email = raw_message.match(/^From: .*<(\S+@\S+)>/).captures

      #Put "From " address into "From:" line
      raw_message = raw_message.sub(old_email, new_email)

      yield raw_message, first_line_number, last_line_number if raw_message.present?

Well, let’s call it nearly:wink:

1 Like

Haha… “So you’re telling me there’s a chance!?”


After a successful import of mail archives (mbox), the content of the messages will display email addresses that would have been obfuscated by Gmane or the mailman2 archive server. This allows bots collecting addresses to harvest them and I’m looking for a way to avoid this.

  1. globally removing email in the posts (a display plugin maybe?)
  2. some site setting that already does that
  3. another idea?

Thanks in advance for your help!


Is it an either/or thing?

When I tried to import my MM2 mbox to MM3, about a quarter of the emails were orphaned (replies were wrongly treated as beginning new threads) because they didn’t have the right headers. Pipermail in MM2 can structure the archive using Subject (if there is no Message-ID or whatever the other header is called - I forget) but, the last time I checked, Postorius in MM3 ignores the Subject. So ideally your script would do the same as Pipermail and mostly get it right on my list :slight_smile:

Also – if emails get imported higgledypiggledy, as noted above, is there any way in Discourse to fix that? Or is the only answer to try index_only and either add headers to the mbox file or rejig the index.db as suggested in the post quoted below?



Yes, it is.

Not really. Well, you could move posts, but that is tedious even with automation.

I think that’s the best way to solve your problem unless you feel comfortable working on the import script and adding some kind of hybrid mode that groups by Message-ID and subject in case the former is missing.


Importing from Google Groups is currently broken, because Google changed the UI and removed the AJAX crawling scheme they deprecated back in 2015.

Has anyone managed to use Google Takeout for exporting mbox files yet?



How can we use this to import the google groups to SAAS discourse instead of on-premise?

If you pay for business hosting for a year then they’ll do it for free. Otherwise, you do it on your own server and upload the backup to your instance and email support to ask them to restore it.

The Google group script can be finicky to get the authentication to work just right. Last time I used it, I had to fiddle with the log in endpoint to get it to work.

1 Like