# An Importer for Google Groups
**URL:** https://meta.discourse.org/t/an-importer-for-google-groups/45624
**Category:** Support
**Created:** [June 13, 2016, 11:20pm UTC](https://meta.discourse.org/t/an-importer-for-google-groups/45624 "2016-06-13T23:20:33Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@steinwaywhw](https://meta.discourse.org/u/steinwaywhw)
#### Post date: [June 13, 2016, 11:20pm UTC](https://meta.discourse.org/t/an-importer-for-google-groups/45624/1 "2016-06-13T23:20:33Z")
Hi,
I would like to open a new thread to continue the discussion [here](https://meta.discourse.org/t/import-from-google-groups-to-discourse/) about an importer for Google Groups. In short, I have a simple python scripts based on Scrapy that scraps all messages of a group to a JSON file. I hope someone who knows Ruby and Discourse API could make it into a real importer (json-discourse importer).
That prior discussion mentioned an importer too. But it seems to me that it can not scroll down to the bottom of a page to load complete messages. It sounds like that one doesn’t really work. I report here that my script can follow links like “more topics” until there is no more, thus scraping all messages.
Another trick is that I found the Google Group url has the format of `base_forum_url`+`range`. Not sure if anyone noticed, but that basically works pretty well for me. For example, you can use
`https://groups.google.com/forum/#!forum/ats-lang-users[10-20]`
to get a range of topics from index 10 to index 20. Therefore you have a way to iterate all topics in a group.
Anyway, I would like to share my scripts as a starting point. I hope someone who has experience with Discourse can pick it up and make it one-click to import Google Groups to Discourse.
The Github Repo is here: [GitHub - steinwaywhw/google-group-exporter · GitHub](https://github.com/steinwaywhw/google-group-exporter)
There is a python script, depends on [scrapy.org](http://scrapy.org) `<-` it should be a clickable link, but I’m a new user, I can’ put more than two links in a post …
There is an example Google Group topic page as seen by the scraper
There is an example output from my scripts.
BTW, “New users can only have two links in a post” - why is that? it’s annoying.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [June 14, 2016, 12:06am UTC](https://meta.discourse.org/t/an-importer-for-google-groups/45624/2 "2016-06-14T00:06:16Z")
> [@steinwaywhw](#):
>
> BTW, “New users can only have two links in a post” - why is that? it’s annoying.
Its a spam protection measure, you can disable in site settings, on import you generally bypass all validation so this should not be an issue for import.
---
### Author:  [@mcwumbly](https://meta.discourse.org/u/mcwumbly)
#### Post date: [June 14, 2016, 2:19am UTC](https://meta.discourse.org/t/an-importer-for-google-groups/45624/3 "2016-06-14T02:19:25Z")
If you want to check out a couple examples of importers that use JSON files as the starting point, there are at least a few precedents:
- [Ning](https://github.com/discourse/discourse/blob/master/script/import_scripts/ning.rb)
- [Quandora](https://github.com/discourse/discourse/tree/master/script/import_scripts/quandora)
- [Socialcast (In Progress)](https://github.com/mcwumbly/discourse/tree/socialcast-import/script/import_scripts/socialcast)
---
### Author:  [@steinwaywhw](https://meta.discourse.org/u/steinwaywhw)
#### Post date: [June 14, 2016, 3:21am UTC](https://meta.discourse.org/t/an-importer-for-google-groups/45624/4 "2016-06-14T03:21:33Z")
You are right. I just realized spams are all about posting links.
---
### Author:  [@steinwaywhw](https://meta.discourse.org/u/steinwaywhw)
#### Post date: [June 14, 2016, 3:22am UTC](https://meta.discourse.org/t/an-importer-for-google-groups/45624/5 "2016-06-14T03:22:29Z")
Those are really helpful. Thanks a lot and i will take time learn some new skills 🙂
---
### Author:  [@erlend\_sh](https://meta.discourse.org/u/erlend_sh)
#### Post date: [June 14, 2016, 3:16pm UTC](https://meta.discourse.org/t/an-importer-for-google-groups/45624/6 "2016-06-14T15:16:53Z")
Did you look into either of these projects?
- [GitHub - icy/google-group-crawler: [Deprecated] Get (almost) original messages from google group archives. Your data is yours. · GitHub](https://github.com/icy/google-group-crawler)
- [GitHub - henryk/gggd: Get Google Groups Data – Tool to fetch all messages of a Google Group as raw mbox data · GitHub](https://github.com/henryk/gggd)
They output mbox data, which Discourse already has a script for:
>
So as long as you could get either of those projects to output clean mbox data that the importer script can work with, we’re golden 🕶
p.s. @pacharanero also took [the scraper approach](https://meta.discourse.org/t/import-from-google-groups-to-discourse/7307/17) and successfully migrated several sites with it.
---
### Author:  [@steinwaywhw](https://meta.discourse.org/u/steinwaywhw)
#### Post date: [June 14, 2016, 5:09pm UTC](https://meta.discourse.org/t/an-importer-for-google-groups/45624/7 "2016-06-14T17:09:57Z")
Thanks for the pointers. I think it largely works, but the mbox importer has some null pointer exceptions at [discourse/script/import\_scripts/mbox.rb at main · discourse/discourse · GitHub](https://github.com/discourse/discourse/blob/master/script/import_scripts/mbox.rb#L210) and [discourse/script/import\_scripts/base.rb at main · discourse/discourse · GitHub](https://github.com/discourse/discourse/blob/master/script/import_scripts/base.rb#L218). It seems that the connection to sqlite db is somehow not working. I might need some time to figure it out.
---
### Author:  [@steinwaywhw](https://meta.discourse.org/u/steinwaywhw)
#### Post date: [June 14, 2016, 6:21pm UTC](https://meta.discourse.org/t/an-importer-for-google-groups/45624/8 "2016-06-14T18:21:48Z")
Alright, I figured it out and I hope this could help others.
I’m using [GitHub - icy/google-group-crawler: [Deprecated] Get (almost) original messages from google group archives. Your data is yours. · GitHub](https://github.com/icy/google-group-crawler) with the latest version (b42f28d4c36eddce1df039566f3615b0753a2dc6) of mbox import script.
## Crawler Part
1. `export _GROUP=your_group_name`
2. `./crawler.sh -sh > wget.sh`
3. `./wget.sh` _note that `chmod +x wget.sh` might be needed_
## Import Part
1. Need to install sqlite3
a. inside docker, find Gemfile at `/var/www/discourse` and add dependency for sqlite3
b. `apt-get install -y sqlite3 libsqlite3-dev`
c. `bundle install --no-deployment` as `root`
2. Need to modify `mbox.rb` in `script/import_scripts`
a. find `MBOX_DIR` and update it accordingly.
b. find `#{MBOX_DIR}/messages/*` in `all_messages` and modify the subdir to be `mbox` as this is what the crawler names it.
c. comment all calls `all_records_exist`, namely, line 210, line 252, and line 317. These lines causing troubles connecting to sqlite3 database. Not sure if you will see the same problem. You can play with these.
3. `su discourse`
4. `RAILS_ENV=production ruby ./mbox.rb`
Refresh your Discourse webpage in the browser, and good to go.
Actually, I hope someone with experience could tell me what’s going on with that “all\_records\_exist?” error. That function is from the `base.rb` file, and the exception basically says the connection object is `null`.
## Update (July 2016)
Hi, I just discussed with the author of google\_group\_crawler and he confirmed that now the email addresses seen by the crawler is not full address, but only a portion of it. Probably for security reasons from Google side. Therefore, I extended the importer script as shown here on [glot.io](http://glot.io): [Google Group Importer for Discourse - Ruby Snippet - glot.io](https://glot.io/snippets/efyndm7qs5), which will load a separate users list csv file (can be exported from Google group as a group owner/manager), and run a cross scan to recover emails. Hope it helps.
---
### Author:  [@erlend\_sh](https://meta.discourse.org/u/erlend_sh)
#### Post date: [July 11, 2016, 7:56am UTC](https://meta.discourse.org/t/an-importer-for-google-groups/45624/10 "2016-07-11T07:56:30Z")
Continued here:
> [@(Superseded) Migrate Google Groups to Discourse](https://meta.discourse.org/t/migration-of-google-groups-to-discourse/48012):
>
> warning This Howto is out-of-date. Please check out [Migrate a mailing list to Discourse (mbox, Listserv, Google Groups, etc)](https://meta.discourse.org/t/importing-mailing-lists-mbox-listserv-google-groups-emails/79773) for updated instructions. Here is a ‘working prototype’ of a google group migration script that attempts to be more ‘all-in-one’ and reduce the number of steps, complexity and difficulty of doing google group imports. [https://github.com/pacharanero/discourse/blob/master/script/import\_scripts/googlegroups.rb](https://github.com/pacharanero/discourse/blob/master/script/import_scripts/googlegroups.rb) Having finally gotten around to doing a pro-bono google g…