### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [May 14, 2021, 12:33pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/182 "2021-05-14T12:33:04Z")
It’s using the `From:` header.
---
### Author:  [@Jonathan5](https://meta.discourse.org/u/Jonathan5)
#### Post date: [May 14, 2021, 12:36pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/183 "2021-05-14T12:36:28Z")
Thanks for the quick reply! How hard would it be to change this? Not necessarily officially – though it might help others – but just for me to change the script before running it. I don’t know any Ruby (yet!) but if it’s just changing a colon to a space…
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [May 14, 2021, 12:46pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/184 "2021-05-14T12:46:29Z")
It’s not a simple change, but it should be doable. You don’t necessarily have to implement it in the import script. If you know another scripting language, I’m sure it won’t be too hard to update the `From:` headers in the mbox files before running the import…
But, feel free to fix it in the import script. A PR is welcome!
A good starting point for fixing the header should be the `each_mail` method…
>
---
### Author:  [@Jonathan5](https://meta.discourse.org/u/Jonathan5)
#### Post date: [May 14, 2021, 1:26pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/185 "2021-05-14T13:26:19Z")
Cheers. Looks like this is what currently decides it, from line 69-70 of indexer.rb:
```plaintext
parsed_email = receiver.mail
from_email, from_display_name = receiver.parse_from_field(parsed_email)
```
Would it be possible at that point to obtain the first line of the mbox email (i.e. the “From [email address] [date time]” line) from `parsed_email` and extract the email address from that?
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [May 14, 2021, 1:28pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/186 "2021-05-14T13:28:15Z")
No, that line is filtered when the mbox is split into individual messages. You need to save that value in the `each_mail` method in order to use it later.
---
### Author:  [@Jonathan5](https://meta.discourse.org/u/Jonathan5)
#### Post date: [May 14, 2021, 9:22pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/187 "2021-05-14T21:22:46Z")
I had some fun trying to do this, before spotting that Mailman stores the emails in the mbox in their original, unadulterated form, so that the “From:” line contains the same (original sender’s) email address as the "From " line in all cases, even when the email has been sent “From: [listname-bounces@listname.domain.com](mailto:listname-bounces@listname.domain.com)”). 🤦♂️
I was limited by not having a development Discourse installation, or even Ruby, but was able to make some headway with [https://rubular.com/](https://rubular.com/) and [Ruby Online Compiler & Interpreter - Replit](https://replit.com/languages/ruby) (and DuckDuckGo). If you would be willing to have a look at it, I’d be grateful if you’d let me know whether this _would_ have worked (or nearly worked) had it been necessary…
```ruby
def each_mail(filename)
raw_message = +''
first_line_number = 1
last_line_number = 0
each_line(filename) do |line|
if line.scrub =~ @split_regex
if last_line_number > 0
#We're at the start of the NEXT email now
yield raw_message, first_line_number, last_line_number
raw_message = +''
first_line_number = last_line_number + 1
else
#We're at the start of THIS email now, so get the email address
new_email = line.match(/^From (\S+@\S+).*/).captures
end
else
raw_message << line
end
last_line_number += 1
end
#Get old email ("From:" line)
old_email = raw_message.match(/^From: .*<(\S+@\S+)>/).captures
#Put "From " address into "From:" line
raw_message = raw_message.sub(old_email, new_email)
yield raw_message, first_line_number, last_line_number if raw_message.present?
end
```
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [May 14, 2021, 9:31pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/188 "2021-05-14T21:31:25Z")
> [@Jonathan5](#):
>
> I’d be grateful if you’d let me know whether this _would_ have worked (or nearly worked) had it been necessary…
Well, let’s call it _nearly_… 😉
---
### Author:  [@Jonathan5](https://meta.discourse.org/u/Jonathan5)
#### Post date: [May 14, 2021, 9:39pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/189 "2021-05-14T21:39:07Z")
Haha… “So you’re telling me there’s a chance!?”
[ ](https://www.youtube.com/watch?v=wGdhc9k07Ms)
---
### Author:  [@dachary](https://meta.discourse.org/u/dachary)
#### Post date: [June 10, 2021, 3:45pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/190 "2021-06-10T15:45:23Z")
After a successful import of mail archives ( **mbox** ), the content of the messages will display email addresses that would have been obfuscated by Gmane or the mailman2 archive server. This allows bots collecting addresses to harvest them and I’m looking for a way to avoid this.
1. globally removing email in the posts (a display plugin maybe?)
2. some site setting that already does that
3. another idea?
Thanks in advance for your help!
---
### Author:  [@Jonathan5](https://meta.discourse.org/u/Jonathan5)
#### Post date: [July 5, 2021, 7:04pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/191 "2021-07-05T19:04:32Z")
> [@gerhard](#):
>
> If your emails do not contain a `Message-ID` header (like messages stored by Listserv), you should enable the `group_messages_by_subject` setting.
Is it an either/or thing?
When I tried to import my MM2 mbox to MM3, about a quarter of the emails were orphaned (replies were wrongly treated as beginning new threads) because they didn’t have the right headers. Pipermail in MM2 can structure the archive using Subject (if there is no Message-ID or whatever the other header is called - I forget) but, the last time I checked, Postorius in MM3 ignores the Subject. So ideally your script would do the same as Pipermail and mostly get it right on my list 🙂
Also – if emails get imported higgledypiggledy, as noted above, is there any way in Discourse to fix that? Or is the only answer to try `index_only` and either add headers to the mbox file or rejig the `index.db` as suggested in the post quoted below?
> [@gerhard](#):
>
> Enable `index_only` in `settings.yml` and take a look at the `index.db` (a SQLite database) before you run the actual import.
Thanks.
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [July 14, 2021, 8:47am UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/192 "2021-07-14T08:47:59Z")
> [@Jonathan5](#):
>
> Is it an either/or thing?
Yes, it is.
> [@Jonathan5](#):
>
> is there any way in Discourse to fix that?
Not really. Well, you could move posts, but that is tedious even with automation.
> [@Jonathan5](#):
>
> Or is the only answer to try `index_only` and either add headers to the mbox file or rejig the `index.db` as suggested in the post quoted below?
I think that’s the best way to solve your problem unless you feel comfortable working on the import script and adding some kind of hybrid mode that groups by Message-ID and subject in case the former is missing.
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [October 6, 2021, 5:38pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/193 "2021-10-06T17:38:14Z")
Importing from Google Groups is currently broken, because Google changed the UI and removed the AJAX crawling scheme they deprecated back in 2015.
Has anyone managed to use Google Takeout for exporting mbox files yet?
---
### Author:  [@Anjana\_Raghavendra\_P](https://meta.discourse.org/u/Anjana_Raghavendra_P)
#### Post date: [October 30, 2021, 3:28pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/194 "2021-10-30T15:28:22Z")
Hi,
How can we use this to import the google groups to SAAS discourse instead of on-premise?
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [October 30, 2021, 7:42pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/195 "2021-10-30T19:42:03Z")
If you pay for business hosting for a year then they’ll do it for free. Otherwise, you do it on your own server and upload the backup to your instance and email support to ask them to restore it.
The Google group script can be finicky to get the authentication to work just right. Last time I used it, I had to fiddle with the log in endpoint to get it to work.
---
### Author:  [@Anjana\_Raghavendra\_P](https://meta.discourse.org/u/Anjana_Raghavendra_P)
#### Post date: [October 31, 2021, 1:00pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/196 "2021-10-31T13:00:37Z")
Do you remember the change you did to make the login work? I am getting the following error even though I used the same extension as mentioned in the initial steps to generate the cookies file. By the way it is private domain group I am working with.
```plaintext
Logging in...
2021-10-31 12:54:41 WARN Selenium [DEPRECATION] [:browser_options] :options as a parameter for driver initialization is deprecated. Use :capabilities with an Array of value capabilities/options if necessary instead.
Traceback (most recent call last):
31: from script/import_scripts/google_groups.rb:293:in `'
30: from script/import_scripts/google_groups.rb:237:in `crawl'
29: from script/import_scripts/google_groups.rb:181:in `login'
28: from script/import_scripts/google_groups.rb:196:in `add_cookies'
27: from script/import_scripts/google_groups.rb:196:in `each'
26: from script/import_scripts/google_groups.rb:200:in `block in add_cookies'
25: from /usr/local/lib/ruby/gems/2.7.0/gems/selenium-webdriver-4.0.3/lib/selenium/webdriver/common/manager.rb:61:in `add_cookie'
24: from /usr/local/lib/ruby/gems/2.7.0/gems/selenium-webdriver-4.0.3/lib/selenium/webdriver/remote/bridge.rb:349:in `add_cookie'
#0 0x557491640f93 : invalid cookie domain: Cookie 'domain' mismatch (Selenium::WebDriver::Error::InvalidCookieDomainError)
```
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [October 31, 2021, 1:37pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/197 "2021-10-31T13:37:50Z")
I’m sorry, but fixing the login isn’t enough.
> [@gerhard](#):
>
> Importing from Google Groups is currently broken, because Google changed the UI and removed the AJAX crawling scheme they deprecated back in 2015.
>
> Has anyone managed to use Google Takeout for exporting mbox files yet?
---
### Author:  [@anon73664359](https://meta.discourse.org/u/anon73664359)
#### Post date: [October 31, 2021, 1:44pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/198 "2021-10-31T13:44:50Z")
Did the more recent redesign fix anything?
> **[Google Groups](https://en.wikipedia.org/wiki/Google_Groups)**
>
> Google Groups is a service from Google that provides discussion groups for people sharing common interests. Until February 2024, the Groups service also provided a gateway to Usenet newsgroups, both reading and posting to them, via a shared user interface. In addition to accessing Google Groups, registered users can also set up mailing list archives for e-mail lists that are hosted elsewhere.
> Google Groups became operational in February 2001, following Google's acquisition of Deja's Usenet arc...
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [October 31, 2021, 1:49pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/199 "2021-10-31T13:49:33Z")
No, unless they re-added the feature in the last 25 days. I don’t think they will, so the scraper will need a complete overhaul.
[Previous page](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773.md?page=2)
[Next page](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773.md?page=4)