'"]
```
As previously, I can share the specific message if it’s helpful – this time the error message gives me the specific line numbers so we can at least have high confidence that we’ve identified the correct message.
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [11 באוקטובר, 2020, 1:09am UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/167 "2020-10-11T01:09:13Z")
Sure, please share the messages with me and will take a look. If there’s a problem we can fix it will not only improve the importer but Discourse itself as it uses the same parser for incoming emails.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [15 באוקטובר, 2020, 3:29pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/168 "2020-10-15T15:29:03Z")
> [@gerhard](#):
>
> In order to download messages, the crawler needs to have access to a Google account that has the owner role for your group. Please visit [https://myaccount.google.com/](https://myaccount.google.com/) in your browser and sign in if you aren’t already logged in. Then use a browser extension of your choice to export your cookies for `google.com` in a file named `cookies.txt` .
I’ve been running this script daily for the past several months for a site that really needs to switch to subscribing the category to the group, but that’s not done that. It works fine except every so often I need to get a new cookies.txt file. About a month ago, something happened and it started complaining that “It looks like you do not have permissions to see email addresses. Aborting.” I did . . . something . . . and it started working again. Just over a week ago it happened again and I’ve re-downloaded the cookies with multiple browsers/cookie plugins and keep getting the no-email version of the posts. I can see the addresses when logged in with the web browser.
Has anyone else had trouble lately? Any ideas on what to do? I’ve tried playing around with what domains are in the `add_cookies` call in the script, but that hasn’t helped.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [5 בנובמבר, 2020, 12:18am UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/169 "2020-11-05T00:18:10Z")
Well, I’m looking at this again and it appears that links like
```
https://groups.google.com/forum/message/raw?msg=GROUP_NAME/THREAD_ID/POST_ID
```
used to include the full email addresses, but now doesn’t. I can confirm that when I’m logged in, I can click about and see full email addresses in the google groups web interface, but if I hit the above URL that the scrape script is hitting in the very same web browser, it gets the data with redacted email addresses.
My guess is that they’ve increased privacy or something.
~~Here’s another clue: I can open that link in my browser and it works, but if I snag the “copy as cURL” that `curl` command isn’t getting the email addresses.~~ Sigh. Well, I tried with another browser and the curl command worked. I can’t quite figure why the script isn’t getting the email addresses.
So maybe there’s some other browser-specific thing it’s doing?
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [18 בנובמבר, 2020, 10:23am UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/170 "2020-11-18T10:23:25Z")
I haven’t tried recently, so it’s possible that there have been changes that the scraper can’t handle right now.
@riking noticed that [Google Takeout](https://takeout.google.com/) exports mbox files for **group owners** , so that might be an option to check out.
> **[Export your Groups data - Google Groups Help](https://support.google.com/groups/answer/9975859)**
>
> You can export and download your data from Google Groups. You can download data that hasn't been deleted. You can create an archive to keep for your records or use the data in another service.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [18 בנובמבר, 2020, 10:09pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/171 "2020-11-18T22:09:14Z")
Thanks. Well, a week ago it worked for a second site and then I updated my cookies file one more time and it downloaded the data for the first site. It seems to have only worked for a day or two, and now again it’s not working. For either site. I see the full email address in my browser, download the cookies for that tab, and no joy.
I’ll check out the takout. EDIT: Well, to get the mbox file it looks like you need to be a super administrator, not just an owner.
> [@](#):
>
> # Export your Groups data
>
> You can export and download your data from Google Groups. You can download data that hasn’t been deleted. You can create an archive to keep for your records or use the data in another service. Learn how to [download your data](https://support.google.com/accounts/answer/3024190).
>
> If you’re a super administrator of your Google domain, you can download or migrate your organization’s data. Learn how to [export your organization’s data](https://support.google.com/a/answer/100458). If you’re using a work or school account, some data might not be available for download.
---
### Author:  [@dachary](https://meta.discourse.org/u/dachary)
#### Post date: [29 בנובמבר, 2020, 8:17pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/172 "2020-11-29T20:17:24Z")
A command line tool to convert a mailman2 mailing list (i.e. the content of the `config.pck` with [options, members, moderators, private or public flags etc.](https://lab.enough.community/discourse/mailman2discourse#details-of-the-configpck-conversion-into-discourse)) into a discourse category is available here: [Client Challenge](https://pypi.org/project/mailman2discourse/)
---
### Author:  [@larsoner](https://meta.discourse.org/u/larsoner)
#### Post date: [10 בדצמבר, 2020, 8:52pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/173 "2020-12-10T20:52:43Z")
@gerhard any ideas how to adapt these instructions to using a [dev install](https://meta.discourse.org/t/beginners-guide-to-install-discourse-for-development-using-docker/102009/94) rather than the standard installation? I feel like I’ve come close to getting a `listserv` migration working using just a few commands, but I can’t get what I assume is the last step to work using either of:
```plaintext
ruby /src/script/import_scripts/mbox.rb ~/import/settings.yml
bundle exec ruby /src/script/import_scripts/mbox.rb /home/discourse/import/settings.yml
```
Both fail to pull in all dependencies. [See here](https://github.com/mne-tools/mne-python/wiki/Mailing-list-migration/_edit#1-in-ubuntu-host) for the full set of commands I used and the errors. Any ideas? Missing some `d/bundle` calls perhaps?
Next thing I will try is using an Ubuntu VM and doing a “[standard install](https://meta.discourse.org/t/142537?silent=true)” there but this seems a bit overkill given the `dev` install otherwise works quite nicely.
I am a total `discourse` (and `ruby`, and mostly `docker`) newbie, so sorry if this is obvious or (worse) irrelevant!
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [10 בדצמבר, 2020, 9:56pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/174 "2020-12-10T21:56:49Z")
It looks like you figured out most of it.
I’ve never tried this with the Docker based dev install, but I guess you need to add `"gem 'sqlite3'"` to the `Gemfile` and execute `apt install -y libsqlite3-dev` inside the container before running `d/bundle install`
Afterwards `bundle exec ruby ...` should work.
---
### Author:  [@larsoner](https://meta.discourse.org/u/larsoner)
#### Post date: [11 בדצמבר, 2020, 2:40pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/175 "2020-12-11T14:40:58Z")
@gerhard thanks for the gentle nudge – I re-ran from absolute scratch (`git clone` onward) while adding `gem 'sqlite3'` to the end of `/src/Gemfile`, since I assumed this was the one you meant, and it worked! For the record here are the instructions I used (for `mne_analysis` listserv):
> ****
>
> # 1. In Ubuntu host
>
> ```plaintext
> git clone https://github.com/discourse/discourse.git
> cd discourse
> d/boot_dev --init
> d/rails db:migrate RAILS_ENV=development
> d/shell
> vim /src/Gemfile # add gem 'sqlite3' to end
> exit
> d/bundle
>
> ```
>
> # 2. In docker shell
>
> ```plaintext
> sudo mkdir -p /shared/import/data
> sudo chown -R discourse:discourse /shared/import
> wget -r -l1 --no-parent --no-directories "https://mail.nmr.mgh.harvard.edu/pipermail//mne_analysis/" -P /shared/import/data/mne_analysis -A "*-*.txt.gz"
> rm /shared/import/data/mne_analysis/robots.txt.tmp
> gzip -d /shared/import/data/mne_analysis/*.txt.gz
> wget https://gist.githubusercontent.com/larsoner/940cd6c7100b87c4c5668cb0bc540afb/raw/9e78513620d11355ad0e10f4a2470996c26ebc8c/mailmanToMBox.py -O ~/mailmanToMBox.py
> python3 ~/mailmanToMBox.py /shared/import/data/mne_analysis/
> rm /shared/import/data/mne_analysis/*.txt
> sudo apt install -y libsqlite3-dev # no-op for me
>
> # check results
> cat /shared/import/data/mne_analysis/*.mbox > ~/all.mbox
> sudo apt install -y procmail
> mkdir -p ~/split
> export FILENO=0000
> formail -ds sh -c 'cat > ~/split/msg.$FILENO' < ~/all.mbox
> rm -rf ~/split ~/all.mbox
>
> # settings
> wget https://raw.githubusercontent.com/discourse/discourse/master/script/import_scripts/mbox/settings.yml -O /shared/import/settings.yml
>
> # run it
> cd /src
> bundle exec ruby script/import_scripts/mbox.rb /shared/import/settings.yml
>
> ```
>
> This had a bunch of informative output, and at the end:
>
> ```plaintext
> ...
> Updating featured topics in categories
> 5 / 5 (100.0%) [6890 items/min] ]
> Resetting topic counters
>
> Done (00h 06min 21sec)
>
> ```
>
> Then exiting and on the Ubuntu host:
>
> ```plaintext
> d/unicorn &
> google-chrome http://0.0.0.0:9292
>
> ```
>
> Done!
I’ll probably tweak the settings to get rid of the `[Mne_analysis]` prefix, but I’m thrilled it’s working this well already!
---
### Author:  [@JasonGlazer](https://meta.discourse.org/u/JasonGlazer)
#### Post date: [14 בדצמבר, 2020, 4:56pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/176 "2020-12-14T16:56:51Z")
@gerhard Can your mbox importer be used only when you first install Discourse or can it be used later after other users are using Discourse? If the importer is used when Discourse is being used by others will they see any side effects?
---
### Author:  [@Brad\_Wood](https://meta.discourse.org/u/Brad_Wood)
#### Post date: [29 בדצמבר, 2020, 7:30pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/177 "2020-12-29T19:30:24Z")
In order to get the importer to scrape messages from Google Groups, I had to reverse this change in `/script/import_scripts/google_groups.rb`
[https://review.discourse.org/t/fix-google-groups-import-changed-login-url-9432/10615](https://review.discourse.org/t/fix-google-groups-import-changed-login-url-9432/10615)
I put the line
```plaintext
wait_for_url { |url| url.start_with?("https://accounts.google.com") }
```
back to
```plaintext
wait_for_url { |url| url.start_with?("https://myaccount.google.com") }
```
Otherwise, I would get this message every time:
```plaintext
Logging in...
Failed to login. Please check the content of your cookies.txt
```
---
### Author:  [@larsoner](https://meta.discourse.org/u/larsoner)
#### Post date: [6 בינואר, 2021, 9:28pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/178 "2021-01-06T21:28:07Z")
@gerhard I noticed after the import that although the messages look okay, there are no staged users at all, even though it seems like there should be (I used the default `staged: true`). The outputs look like:
```plaintext
...
indexing replies and users
creating categories
1 / 1 (100.0%) [13440860 items/min]
creating users
creating topics and posts
7399 / 7399 (100.0%) [1421 items/min]
...
```
Is there supposed to be a user counter shown as well?
I also tried running with `staged: false` and the same output was shown, and none of the mailing list users are in any groups. In case it helps to see what’s actually being processed, here is one of the many .mbox files that is being imported:
[2020-December.zip](https://meta.discourse.org/uploads/short-url/qPdAO13sR35LAT0zWPHPrDqLApe.zip) (49.5 KB)
The only non-default setting was adding:
```plaintext
tags:
"Mne_analysis": "mne_analysis"
```
It would be great to have these users show up as staged so that they can claim their old posts when signing up, so any tips or ideas appreciated!
---
### Author:  [@riking](https://meta.discourse.org/u/riking)
#### Post date: [6 בינואר, 2021, 11:35pm UTC](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773/179 "2021-01-06T23:35:11Z")
It should probably just accept both of those?
[Previous page](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773.md?page=1)
[Next page](https://meta.discourse.org/t/migrate-a-mailing-list-to-discourse-mbox-listserv-google-groups-etc/79773.md?page=3)