Importing mailing lists (mbox, Listserv, Google Groups, emails, ...)

Have you had a look at the database? Gut feeling on this issue is for some reason the email field is not getting created correctly there and thus can’t be read.

https://github.com/discourse/discourse/blob/f6e87e1e5ebdd6b6cfafa9e23cdd0a29190d1a7c/script/import_scripts/mbox/importer.rb#L72

See 2.3 in the OP for checking the index database.

2 Likes

The Mailman 2 list that I am considering importing into Discourse has had (for part of its existence) from_is_list set to Munge From, so that the “From:” header is

From: Listname <listname-bounces@listdomain.com> On Behalf Of [Original sender's name]

instead of

From: [Original sender's name] <username@example.com>

This made me think the importer would import each of these messages as if from the same user (with email address listname-bounces@listdomain.com)… BUT…

The initial line marking the beginning of a new email in the mbox file still begins with

From username@example.com [Date time group]

(and the Hyperkitty archives show the original sender’s email address as normal also).

So my question is – does the importer script take the sender’s address from the “From:” header or the "From " line? Thank you.

I discussed this briefly in a previous topic: Working on a mailman2 to discourse migration script - #10 by dachary

1 Like

It’s using the From: header.

1 Like

Thanks for the quick reply! How hard would it be to change this? Not necessarily officially – though it might help others – but just for me to change the script before running it. I don’t know any Ruby (yet!) but if it’s just changing a colon to a space…

It’s not a simple change, but it should be doable. You don’t necessarily have to implement it in the import script. If you know another scripting language, I’m sure it won’t be too hard to update the From: headers in the mbox files before running the import…

But, feel free to fix it in the import script. A PR is welcome!
A good starting point for fixing the header should be the each_mail method…

4 Likes

Cheers. Looks like this is what currently decides it, from line 69-70 of indexer.rb:

parsed_email = receiver.mail
from_email, from_display_name = receiver.parse_from_field(parsed_email)

Would it be possible at that point to obtain the first line of the mbox email (i.e. the “From [email address] [date time]” line) from parsed_email and extract the email address from that?

2 Likes

No, that line is filtered when the mbox is split into individual messages. You need to save that value in the each_mail method in order to use it later.

3 Likes

I had some fun trying to do this, before spotting that Mailman stores the emails in the mbox in their original, unadulterated form, so that the “From:” line contains the same (original sender’s) email address as the "From " line in all cases, even when the email has been sent “From: listname-bounces@listname.domain.com”). :man_facepalming:

I was limited by not having a development Discourse installation, or even Ruby, but was able to make some headway with https://rubular.com/ and https://replit.com/languages/ruby (and DuckDuckGo). If you would be willing to have a look at it, I’d be grateful if you’d let me know whether this would have worked (or nearly worked) had it been necessary…

    def each_mail(filename)
      raw_message = +''
      first_line_number = 1
      last_line_number = 0

      each_line(filename) do |line|
        if line.scrub =~ @split_regex
          if last_line_number > 0
            #We're at the start of the NEXT email now
            yield raw_message, first_line_number, last_line_number
            raw_message = +''
            first_line_number = last_line_number + 1
          else
            #We're at the start of THIS email now, so get the email address 
            new_email = line.match(/^From (\S+@\S+).*/).captures
          end
        else
          raw_message << line
        end

        last_line_number += 1
      end

      #Get old email ("From:" line) 
      old_email = raw_message.match(/^From: .*<(\S+@\S+)>/).captures

      #Put "From " address into "From:" line
      raw_message = raw_message.sub(old_email, new_email)

      yield raw_message, first_line_number, last_line_number if raw_message.present?
    end
2 Likes

Well, let’s call it nearly:wink:

1 Like

Haha… “So you’re telling me there’s a chance!?”

1 Like

After a successful import of mail archives (mbox), the content of the messages will display email addresses that would have been obfuscated by Gmane or the mailman2 archive server. This allows bots collecting addresses to harvest them and I’m looking for a way to avoid this.

  1. globally removing email in the posts (a display plugin maybe?)
  2. some site setting that already does that
  3. another idea?

Thanks in advance for your help!

2 Likes