An Importer for Google Groups

Product Support
8

Alright, I figured it out and I hope this could help others.

I’m using GitHub - icy/google-group-crawler: [Deprecated] Get (almost) original messages from google group archives. Your data is yours. · GitHub with the latest version (b42f28d4c36eddce1df039566f3615b0753a2dc6) of mbox import script.

Crawler Part

  1. export _GROUP=your_group_name
  2. ./crawler.sh -sh > wget.sh
  3. ./wget.sh note that chmod +x wget.sh might be needed

Import Part

  1. Need to install sqlite3
    a. inside docker, find Gemfile at /var/www/discourse and add dependency for sqlite3
    b. apt-get install -y sqlite3 libsqlite3-dev
    c. bundle install --no-deployment as root
  2. Need to modify mbox.rb in script/import_scripts
    a. find MBOX_DIR and update it accordingly.
    b. find #{MBOX_DIR}/messages/* in all_messages and modify the subdir to be mbox as this is what the crawler names it.
    c. comment all calls all_records_exist, namely, line 210, line 252, and line 317. These lines causing troubles connecting to sqlite3 database. Not sure if you will see the same problem. You can play with these.
  3. su discourse
  4. RAILS_ENV=production ruby ./mbox.rb

Refresh your Discourse webpage in the browser, and good to go.

Actually, I hope someone with experience could tell me what’s going on with that “all_records_exist?” error. That function is from the base.rb file, and the exception basically says the connection object is null.

Update (July 2016)

Hi, I just discussed with the author of google_group_crawler and he confirmed that now the email addresses seen by the crawler is not full address, but only a portion of it. Probably for security reasons from Google side. Therefore, I extended the importer script as shown here on glot.io: Google Group Importer for Discourse - Ruby Snippet - glot.io, which will load a separate users list csv file (can be exported from Google group as a group owner/manager), and run a cross scan to recover emails. Hope it helps.