# (Superseded) Importing from Google Groups
**URL:** https://meta.discourse.org/t/superseded-importing-from-google-groups/76860
**Category:** Migrating to Discourse
**Tags:** migrations, how-to
**Created:** [December 28, 2017, 4:09am UTC](https://meta.discourse.org/t/superseded-importing-from-google-groups/76860 "2017-12-28T04:09:07Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@yanokwa](https://meta.discourse.org/u/yanokwa)
#### Post date: [December 28, 2017, 4:09am UTC](https://meta.discourse.org/t/superseded-importing-from-google-groups/76860/1 "2017-12-28T04:09:07Z")
> ⚠ **This Howto is out-of-date.**
>
> Please check out [Migrate a mailing list to Discourse (mbox, Listserv, Google Groups, etc)](https://meta.discourse.org/t/importing-mailing-lists-mbox-listserv-google-groups-emails/79773) for updated instructions.
* * *
@pacharanero’s fantastic post on [Migration of Google Groups to Discourse](https://meta.discourse.org/t/migration-of-google-groups-to-discourse/48012) only works with Discourse v1.7.2, so I wanted to share the steps I followed to import Google Groups into the Discourse v1.8.x (the latest stable version).
If you’ve installed and configured Discourse before and are comfortable on the command line, I hope this guide will be as helpful to you as @pacharanero’s guide has been to me.
If you run into any problems with these steps, just ask below and I’m glad to help. And if you have suggestions on how I can improve the guide, please share so we can make this process easier for everyone!
### A few warnings before we start
Moving a community from Google Groups to Discourse is time-consuming, fragile, and annoying. If you can find someone who has done it before to do it for you, I’d strongly recommend that option!
If you’d like to try a move yourself, please read through all the steps first before attempting it. And please try it on a test server before actually moving your community. Again, it’s a fragile process.
The rough steps are to install and configure Discourse, install some prerequisites, scrape all your Google Groups messages (you’ll need to be a “Manager” to do this) into mbox files, then import those mbox files into Discourse.
The scrape of the messages from Google Groups is slow so give yourself a few hours. I’d also recommend you put the Google Group in read-only mode so you don’t miss any new messages while doing the import.
The import is CPU and RAM intensive, I’d recommend using the biggest machine you can find, backup the Discourse data after the import, and then restore it to a production machine.
I’ve had trouble getting this guide to work on recent versions of Discourse v1.9.x, but it does work great on v1.8.x. On your import machine, edit `/var/discourse/containers/app.yaml`, set version to `version: stable`, do a rebuild, and use that version to be safe.
### 1. Install and configure Discourse
1. [Install Discourse](https://github.com/discourse/discourse/blob/master/docs/INSTALL.md)!
2. Skip Setup by clicking Maybe Later. Then in Settings…
### 2. Install prerequisites inside Docker
1. Log into the server. You might want to use [mosh/tmux](https://blog.filippo.io/my-remote-shell-session-setup/) instead of SSH since the process requires many long running tasks.
2. Install pngout and pngquant. This is needed to compress images on import.
3. Install sqlite3 for mbox import.
4. Prep the Gemfile.
5. Add sqlite3 to `/tmp/Gemfile`. The beginning of the file will look like this:
6. Install the needed gems. There’s likely a better way of installing gems in deployment mode, but this is how I did it.
7. Edit `/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/discourse_image_optim-0.24.5/lib/image_optim/worker/jhead.rb` to fix compression errors during import. The beginning of the file will look like this:
### Scrape messages from Google Groups
1. Download @pacharanero’s awesome [google\_group.to\_discourse](https://github.com/pacharanero/google_group.to_discourse) script.
2. Edit `/var/www/discourse/script/import_scripts/googlegroup.rb` to comment out import\_to\_discourse. The end of the file will look like this:
3. To scrape the emails from Google Groups, you will need a Google account with a “Manager” level access to the Google Group.
4. Scrape messages from the Google Group. Depending on the size of your group, this step will take a few hours. Since it is a long process, it might be wise to backup the googlegroup-export folder after the scrape has finished.
5. Setup your data directory for the import.
6. Edit `/var/www/discourse/script/import_scripts/mbox/settings.yml` to point to data directory. The file will look like this:
7. Move all the `my-list` emails into a `my-category` folder. `my-category` will be created by the import script automatically.
8. Edit `/var/www/discourse/script/import_scripts/mbox/importer.rb` to make sure users are not staged. The create\_users method will look like this:
### 3. Import mbox files into Discourse
1. Run the import. Depending on the size of your group, this step may take a few hours. The driver seems to be how many images need to be compressed by pngquant/pngout.
2. That’s it! All the messages from `my-list` should now be in `my-category`!
### Frequently asked questions
- **Import is failing on a particular topic, how do I recover?**
- **How do I add members who signed up for the Google Group, but didn’t send messages?**
- **How do I update the Google Group scrape with new messages?**
---
### Author:  [@pacharanero](https://meta.discourse.org/u/pacharanero)
#### Post date: [December 31, 2017, 1:06pm UTC](https://meta.discourse.org/t/superseded-importing-from-google-groups/76860/2 "2017-12-31T13:06:13Z")
Thanks @yanokwa for the name-check on my google-group migration script! This is a great HOWTO you’ve done here.
I keep meaning to update the importer script, but lack of time and a profusion of other projects keep getting in the way. I’ll get to it eventually. I can provide google-group=\> discourse migration services and as part of any future work I’ll update and publish the script and hopefully put in some tests and proper production values so it can be maintained and pulled into Discourse core.
I took a look at it recently but I was a little confused by the current state of the mbox importer scripts - there are now **two** of them in [discourse/script/import-scripts/](https://github.com/discourse/discourse/tree/master/script/import_scripts) but there’s no documentation that I can find anywhere either in the source code, or in Meta, or any README.md anywhere.
As far as I can tell, [discourse/script/import\_scripts/mbox.rb](https://github.com/discourse/discourse/blob/master/script/import_scripts/mbox.rb) is the original mbox importer, which was contributed to by (among others) @eviltrout @pfaffman @sam. I tried to use it for a straight mbox archive import recently and it didn’t work on discourse 1.9, but this may have been a problem with the mbox files I was importing.
Then we have [discourse/script/import\_scripts/mbox-experimental.rb](https://github.com/discourse/discourse/blob/master/script/import_scripts/mbox-experimental.rb), which has a little bit of documentation in the source code, but is not referenced anywhere in Meta apart from in [this post](https://meta.discourse.org/t/experimental-mbox-importer-fails-on-subset-of-messages/69247). This script looks like it uses the code in discourse/script/import\_scripts/mbox/ and has a yaml file for the import settings. It was contributed to by @gerhard @tgxworld @techAPJ, and would seem to have been updated much more recently.
It would be nice to have a bit more clarity around which one is the ‘official’ Discourse mbox importer (the older non-experimental one that works with 1.8 but not 1.9, or the newer ‘experimental’ one), and perhaps an updated/wikified howto for 1.9 mbox imports?
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [January 1, 2018, 10:10pm UTC](https://meta.discourse.org/t/superseded-importing-from-google-groups/76860/3 "2018-01-01T22:10:30Z")
> [@pacharanero](#):
>
> It would be nice to have a bit more clarity around which one is the ‘official’ Discourse mbox importer
Both are “official”, but the “experimental” one – which was my playground for a little while 🙂 – will surely replace the older one in the next month or two.
> [@yanokwa](#):
>
> Edit /var/www/discourse/script/import\_scripts/mbox/importer.rb to make sure users are not staged.
I think this could be added as a setting to the import script. IMHO there shouldn’t be a need for manually editing code. You can send a PR if you want. 😉
---
### Author:  [@pacharanero](https://meta.discourse.org/u/pacharanero)
#### Post date: [January 2, 2018, 3:20pm UTC](https://meta.discourse.org/t/superseded-importing-from-google-groups/76860/4 "2018-01-02T15:20:26Z")
Thanks @gerhard good to know what the roadmap is
---
### Author:  [@yanokwa](https://meta.discourse.org/u/yanokwa)
#### Post date: [January 12, 2018, 3:33am UTC](https://meta.discourse.org/t/superseded-importing-from-google-groups/76860/5 "2018-01-12T03:33:29Z")
PR has been merged: [Allow user staging via setting (#5468) · discourse/discourse@77a92e8 · GitHub](https://github.com/discourse/discourse/commit/77a92e8878e550a3c9bb3e435f827c8941d7f8a3)
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [March 21, 2019, 10:57am UTC](https://meta.discourse.org/t/superseded-importing-from-google-groups/76860/7 "2019-03-21T10:57:34Z")
---
### Author:  [@osioke](https://meta.discourse.org/u/osioke)
#### Post date: [July 15, 2021, 11:06am UTC](https://meta.discourse.org/t/superseded-importing-from-google-groups/76860/8 "2021-07-15T11:06:10Z")