Ning import script

Feature
21

Should be we have used it 3 times with 2 customers. We always check importers into the source tree at the usual location.

1 Like
22

any tutorial on how to use it? i’m just new to discourse and would want to migrate my json files to discourse over the weekend.

23

Assuming you have a fully functional Discourse development environment:

  • put your Ning json files in one directory
  • edit ning.rb and put the path in JSON_FILES_DIR.
  • edit other constants at the top of that file as needed for your Ning export and which file attachments you want to allow.
  • Look at the comments in the initialize and execute methods near the top of the file and customize how you like.
  • Run the import. bundle exec ruby script/import_scripts/ning.rb

Good luck!

4 Likes
24

thanks for this. hopefully this will be successful …

25

@neil, how do i edit ning.rb? where to go to edit it? in my consul?
or i’ll edit github directly?
sorry, really noob here :frowning:

26

when trying to access the script via consul, i get “bundler not installed”
when installing bundler, i get “unable to locate package budler”

any help will be appreciated :frowning:

27

Before attempting to import from Ning, you’ll need to setup a development environment. Please follow this guide.

1 Like
28

@neil, already reinstalled my dev environment set-up using that tutorial or does this mean that other tutorials for installing discourse thru docker will not work if you’re trying to use import script?

because i’ve successffully installed discourse using those tutorials, but i’m pretty stuck with the import script on how to go about it. :frowning:

tried using nano but still don’t know where to paste it.

31

I have some questions about configuring ning.rb.

My Ning export looks like this:

  • What exactly are ATTACHMENT_PREFIXES? The default value is: ["discussions", "pages", "blogs", "members", "photos"]. These look like they should map to the exported folders. Should I add events and members to this list, as they are missing?

  • I assume that I should put all filetypes that I want imported in EXTRA_AUTHORIZED_EXTENSIONS. Are these values case insensitive? (I look into the existing files and some extensions are in all caps for some reason.)

I have taken a stab at configuring the JSON import as follows:

Initialize

@users_json       = load_ning_json("ning-members-local.json")
@discussions_json = load_ning_json("ning-discussions-local.json")

# An example of a custom category from Ning:
@blogs_json       = load_ning_json("ning-blogs-local.json")

# My additions
@events_json      = load_ning_json("ning-events-local.json")
@groups_json      = load_ning_json("ning-groups-local.json")
@moreusers_json   = load_ning_json("ning-members.json")
@pages_json       = load_ning_json("ning-pages-local.json")
@photos_json      = load_ning_json("ning-photos-local.json")

Import

import_users
import_categories
import_discussions

import_blogs # Remove this and/or add more as necessary

# My additions
import_events
import_groups
import_moreusers
import_pages
import_photos
  • How do I create functions to import all of these? Is it sufficient to do something like this:
def import_XYZ
  puts "", "Importing XYZ"
  import_topics(@XYZ_json, "XYZ")
end

(Where XYZ one of events, groups, pages, etc.)

Finally, for reasons that are unclear to me, I have two member files: ning-members-local.json and ning-members.json.

  • Is it better to combine these at the JSON level or duplicate the users stuff twice?

Any help would be much appreciated :slightly_smiling:

32

You have to use the -local ones. It references the attachment files in the directories you show in your screenshot.

Yes add the attachment directories that you have to that list.

It’s not case sensitive. Add the extensions you want to allow in Discourse.

Remove this one. The @users_json already loads the users. Also, you can’t use any of the json files that are missing the “-local” ending.

Copy and paste the import_blogs example and change the variable name and category where the imported topics will go.

3 Likes
33

Thank you that answers my questions for the moment, though I’m sure I’ll be back :slightly_smiling:

The JSON user file was giving me an error and I want to download it again, but the Ning archive tool only allows you to download every 6 hours. Sigh.

34

Yeah the NING json is full of errors like missing commas, extra brackets, etc. I think I handled all of them in the repair_json method, but you might have discovered more. It’s difficult to find out what the errors are…

To share my pain, I’ll paste the repair code here:

  def repair_json(arg)
    arg.gsub!(/^\(/, "")     # content of file is surround by ( )
    arg.gsub!(/\)$/, "")

    arg.gsub!(/\]\]$/, "]")  # there can be an extra ] at the end

    arg.gsub!(/\}\{/, "},{") # missing commas sometimes!

    arg.gsub!("}]{", "},{")  # surprise square brackets
    arg.gsub!("}[{", "},{")  # :troll:

    arg
  end

Also… note that the Ning exporter will truncate discussions to the most recent (or oldest?) 100 posts. If you have any topics with more posts than that, you’ll have to contact Ning support directly to get the full export data.

5 Likes
35

I finally got round to re-downloading all our content from Ning, customising and running the import script. Everything seems to have gone smoothly. I didn’t run into any issues with the JSON this time round.

A couple of broader issues:

  • from what I can tell the discussions outside of the main forum aren’t categorized at all. Most of our content is under various groups and under events. Is there a way to preserve to carry these groups as categories in Discourse?
  • Pages from Ning’s groups are also migrated into the Pages Discourse category. It would be preferable if they were slotted in the appropriate group-category. Also, it would be nice if they were added as locked topics.
36

I don’t remember seeing anything in the exported data about group-categories. It’s possible that it’s encoded in there somewhere, but I didn’t see it. If Ning is exporting that data, then the importer could probably be improved.

1 Like
37

Huh. I figured that would’ve been an obvious piece of information to include. But I had a look at the exported data and I didn’t see anything, either. Oh, Ning.

P.S. I really appreciate the swift reply and I’m sorry for not replying sooner. I didn’t receive an email notification for some reason.

2 Likes
38

For anyone playing along at home.

If you want to add import custom categories like this:

Make sure to also edit this line in ning.rb accordingly:

create_categories((["Blog", "Pages", "XYZ"] + @discussions_json.map { |d| d["category"] }).uniq.compact) do |name|

Otherwise you’ll end up with everything in the “Uncategorized” category.

4 Likes
39

hi, were you able to successfully import your ning files to discourse?
do let me know. cheers

40

Yes, it works fine.

There’s some stuff that the NingNetworkArchiver application doesn’t export at all. But the stuff that it does export imports in Discourse just fine. @neil’s done a swell job.

AFAICT, the following issue doesn’t affect us:

But I’ve asked Ning about some of the other content that the exporter doesn’t include and I’m waiting to hear back from them.

4 Likes
41

Glad to hear it worked for you!

FYI, I added a note to the first post mentioning that Ning 3.0 doesn’t have a way to export your data at all. Let’s hope they add it eventually, and then we’ll need to see if this importer works for Ning 3.

2 Likes
42

I was planning to re-import everything on my production site in docker, but apparently that’s frowned upon. So I decided to use my local Vagrant installation to do the import and use the backup feature to move the database to production.

In that case, I need to resolve some errors that hadn’t I bothered with up to now.

Example:

jhead worker: `jhead` not found, `jpegtran` not found; please provide proper binary or disable this worker (--no-jhead argument or `:jhead => false` through options)
jpegoptim worker: `jpegoptim` not found; please provide proper binary or disable this worker (--no-jpegoptim argument or `:jpegoptim => false` through options)
jpegtran worker: `jpegtran` not found; please provide proper binary or disable this worker (--no-jpegtran argument or `:jpegtran => false` through options)
gifsicle worker: `gifsicle` not found; please provide proper binary or disable this worker (--no-gifsicle argument or `:gifsicle => false` through options)
gifsicle worker: `gifsicle` not found; please provide proper binary or disable this worker (--no-gifsicle argument or `:gifsicle => false` through options)
svgo worker: `svgo` not found; please provide proper binary or disable this worker (--no-svgo argument or `:svgo => false` through options)
       20 / 337 (  5.9%)  optipng worker: `optipng` not found; please provide proper binary or disable this worker (--no-optipng argument or `:optipng => false` through options)

I realise that this is not a Ning import script issue per se, but I bringing it up since it’ll probably affect other users.

You can clear most of this up by installing the required tools:

sudo apt-get update
sudo apt-get install jhead libjpeg-turbo-progs jpegoptim gifsicle optipng

That installs everything except svgo, which I didn’t manage to install.

I tried the following:

sudo apt-get install npm
sudo npm install -g imagemin svgo

This looks like it should work, but it doesn’t. It spits out some npm errors I didn’t manage to troubleshoot. I’m curious but not bothered, since I can run svgo before I start the import from outside of Vagrant.

Hope this helps.

2 Likes