# Ning import script

**URL:** https://meta.discourse.org/t/ning-import-script/24358
**Category:** Feature
**Tags:** feedback
**Created:** [January 23, 2015, 5:29pm UTC](https://meta.discourse.org/t/ning-import-script/24358 "2015-01-23T17:29:28Z")
**Posts on this page:** 20
**Page:** 3

<div class="post-metadata">

### Author: ![zogstrip](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/zogstrip/32/512781_2.png) [@zogstrip](https://meta.discourse.org/u/zogstrip)
#### Post date: [March 13, 2016, 5:52pm UTC](https://meta.discourse.org/t/ning-import-script/24358/43 "2016-03-13T17:52:54Z")

</div>

I don’t think you should be using `sudo` when `npm installing` stuff. I’ve had some bad experiences doing so.

---

<div class="post-metadata">

### Author: ![alehandrof](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/alehandrof/32/119526_2.png) [@alehandrof](https://meta.discourse.org/u/alehandrof)
#### Post date: [March 13, 2016, 7:24pm UTC](https://meta.discourse.org/t/ning-import-script/24358/44 "2016-03-13T19:24:07Z")

</div>

I think I tried it without `sudo` first without success and then just threw in the `sudo` in case it helped, which it didn’t. I don’t even have any .svgs among the files I’m importing. ~~, so I’m not going to try and troubleshoot this.~~ I resolved this by re-installed node/npm following the instructions from the official repo.

It would be nice to figure out how to suppress the error message about ~~`svgo`~~ `gifsicle`, because there’s other errors that I’m having a hard time spotting.

(I did try `bundle exec ruby script/import_scripts/ning.rb --no-svgo`, but clearly that option needs to go somewhere else.)

Here are two errors that may be related.

The first:

```plaintext
       20 / 257 ( 7.8%) Upload not valid :( /vagrant/ning/discussions/54-HinchliffeAMICAL2.pdf
#<ActiveModel::Errors:0xc8c02b0 @base=#<Upload id: nil, user_id: -1, original_filename: "54-HinchliffeAMICAL2.pdf", filesize: 5,198,132, width: nil, height: nil, url: "", created_at: nil, updated_at: nil, sha1: "be0daf8ed4d4cbd980c43e3710f620500f8b8778", origin: nil, retain_hours: nil>, @messages={:filesize=>["Sorry, the file you are trying to upload is too big (maximum size is 3072KB)."]}>

```

The is a large file error. There were several large files causing problems, but they disappeared when I increased `max image size kb` before running the importing script. But this one doesn’t, which is strange. The file is only 5 MB and I’ve set `max image size kb` to `100000`.

The second:

```plaintext
189 / 257 ( 73.5%) Attachment file doesn't exist: /vagrant/ning/FileNotFound

```

This is a missing file. ~~It’s either the previous large file or something else. If the latter,~~ I’d like to track down the filename, but I can’t figure out how. Any suggestion of how to find which topic is “189 / 257” in the Ning data?

---

<div class="post-metadata">

### Author: ![alehandrof](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/alehandrof/32/119526_2.png) [@alehandrof](https://meta.discourse.org/u/alehandrof)
#### Post date: [March 14, 2016, 12:13pm UTC](https://meta.discourse.org/t/ning-import-script/24358/46 "2016-03-14T12:13:27Z")

</div>

> [@alehandrof](#):
>
> The is a large file error. There were several large files causing problems, but they disappeared when I increased max image size kb before running the importing script. But this one doesn’t, which is strange. The file is only 5 MB and I’ve set max image size kb to 100000.

I’m a doofus. There are two settings. One for images (`max image size kb`) and one for other file types (`max attachment size kb`). Increasing the size of both settings fixes all the issues with large files.

I still haven’t figured out how to track the missing file. Any ideas?

---

<div class="post-metadata">

### Author: ![Rodelio\_Lagahit](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/rodelio_lagahit/32/121638_2.png) [@Rodelio\_Lagahit](https://meta.discourse.org/u/Rodelio_Lagahit)
#### Post date: [March 18, 2016, 3:22am UTC](https://meta.discourse.org/t/ning-import-script/24358/47 "2016-03-18T03:22:54Z")

</div>

you guys are amazing, it has been a year already that i’ve been trying to import mine with the help of someone here but to no avail. i’ve left it hanging. i’ve looked at buddypress and even drupal just to stay away from ning.

anyhow, ill just keep track of your progress for me to understand it better.

on another note, @alehandrof would be willing to help me import my stuff once your successful for a little fee? that’s the least i could do to say thanks.

---

<div class="post-metadata">

### Author: ![alehandrof](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/alehandrof/32/119526_2.png) [@alehandrof](https://meta.discourse.org/u/alehandrof)
#### Post date: [March 18, 2016, 8:16am UTC](https://meta.discourse.org/t/ning-import-script/24358/48 "2016-03-18T08:16:44Z")

</div>

@Rodelio_Lagahit I found the help in this forum – including from members of the core dev team – to be remarkable. But you do need to have some decent sysadmin skills to tackle this stuff in the first place.

I read through lots of forums and docs in order to figure out everything I needed to do.  
If there was one thing that I wish I figured out earlier, it’s this: I needed to setup a local Vagrant instance and do the import and configuration there. _Then_ I exported the database and re-imported it in my production Docker.

If you aren’t able to get things going on your own, check with [Discourse](http://www.discourse.org) and [[Communiteq](https://www.communiteq.com) (formerly DiscourseHosting)]([https://www.communiteq.com](https://www.communiteq.com)). I seem to recall that both can help you with importing/installing for a one-time fee.

---

<div class="post-metadata">

### Author: ![erlend\_sh](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/erlend_sh/32/119475_2.png) [@erlend\_sh](https://meta.discourse.org/u/erlend_sh)
#### Post date: [March 18, 2016, 12:06pm UTC](https://meta.discourse.org/t/ning-import-script/24358/49 "2016-03-18T12:06:47Z")

</div>

We only do migrations for Enterprise customers or Business customers who pre-pay for a year of service.

For assisted migrations, we highly recommend using the #Marketplace category. There’s a growing number of very capable freelance developers in our community. Fair compensation for migration work is usually in the $150-$500 range, depending on the complexity of the migration.

---

<div class="post-metadata">

### Author: ![Rodelio\_Lagahit](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/rodelio_lagahit/32/121638_2.png) [@Rodelio\_Lagahit](https://meta.discourse.org/u/Rodelio_Lagahit)
#### Post date: [April 1, 2016, 1:33am UTC](https://meta.discourse.org/t/ning-import-script/24358/50 "2016-04-01T01:33:25Z")

</div>

This is noted. I’ve tried that before months ago. Anyhow, worth revisiting.

---

<div class="post-metadata">

### Author: ![Rodelio\_Lagahit](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/rodelio_lagahit/32/121638_2.png) [@Rodelio\_Lagahit](https://meta.discourse.org/u/Rodelio_Lagahit)
#### Post date: [April 1, 2016, 1:34am UTC](https://meta.discourse.org/t/ning-import-script/24358/51 "2016-04-01T01:34:13Z")

</div>

This is what we did together with @pl3bs but we’re not successful. ☹

---

<div class="post-metadata">

### Author: ![Rodelio\_Lagahit](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/rodelio_lagahit/32/121638_2.png) [@Rodelio\_Lagahit](https://meta.discourse.org/u/Rodelio_Lagahit)
#### Post date: [April 2, 2016, 1:21pm UTC](https://meta.discourse.org/t/ning-import-script/24358/52 "2016-04-02T13:21:34Z")

</div>

can you share your tutorial on how to get this done? im not really a coder but would be willing to get my hands dirty for this to work. @alehandrof @zogstrip

---

<div class="post-metadata">

### Author: ![alehandrof](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/alehandrof/32/119526_2.png) [@alehandrof](https://meta.discourse.org/u/alehandrof)
#### Post date: [July 26, 2016, 10:59am UTC](https://meta.discourse.org/t/ning-import-script/24358/53 "2016-07-26T10:59:29Z")

</div>

A user just alerted me that some (all?) of the links to files form our old Ning forum aren’t working. They appear linked but the links are empty:

 ![](https://global.discourse-cdn.com/meta/original/3X/4/b/4b8aa927d6ddfda8b883a59f2547e818ef8297bc.png)

These were fine at some point, but apparently have stopped working somewhere along the line.

Any ideas?

---

<div class="post-metadata">

### Author: ![neil](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/neil/32/102150_2.png) [@neil](https://meta.discourse.org/u/neil)
#### Post date: [July 26, 2016, 2:30pm UTC](https://meta.discourse.org/t/ning-import-script/24358/54 "2016-07-26T14:30:16Z")

</div>

You mean the attachments were imported into Discourse but went missing?

The example you show mentions a spreadsheet, so maybe it wasn’t an allowed file type. Before starting the import, there’s a setting to configure:

```plaintext
EXTRA_AUTHORIZED_EXTENSIONS = ["bmp", "ico", "txt", "pdf"]

```

The importer output would mention attachments that it couldn’t import.

---

<div class="post-metadata">

### Author: ![alehandrof](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/alehandrof/32/119526_2.png) [@alehandrof](https://meta.discourse.org/u/alehandrof)
#### Post date: [July 27, 2016, 9:10am UTC](https://meta.discourse.org/t/ning-import-script/24358/55 "2016-07-27T09:10:50Z")

</div>

My current understanding is that the files were imported into Discourse, but went missing somewhere along the way.

I had asked about extensions [way back when](https://meta.discourse.org/t/ning-import-script/24358/31) and had included all the ones I needed:

```plaintext
EXTRA_AUTHORIZED_EXTENSIONS = ["bin", "bmp", "doc", "docx", "gif", "htm", "ico", "jpeg", "jpg", "json", "pdf", "png", "ppsx", "ppt", "pptx", "txt", "xls", "xlsx", "zip"]

```

It’s not just spreadsheets. I just checked more thoroughly and every file except for images seems to be missing. 🤕

For good measure, I ran `rake posts:rebake` as that helped with some [images woes I had in the past](https://meta.discourse.org/t/broken-images-in-production-after-importing-backup/41200/). But no joy.

The files are included in `/var/www/discourse/public/uploads/default/original/1X`.

I was able to cross-reference a topic in Discourse with Ning.

This code:

```plaintext
<li>
<a>AMICAL 2014 Members Council Meeting - Minutes</a>&nbsp;(14 May 2014)</li>

```

Should be linking to a file called:

- Ning: `AMICAL2014MembersCouncilMinutes_distmembers.pdf`
- Discourse: `0baa9dd232fe29c1b7cb53376b0d2feda905e091.pdf`

Does this help me at all?

---

<div class="post-metadata">

### Author: ![alehandrof](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/alehandrof/32/119526_2.png) [@alehandrof](https://meta.discourse.org/u/alehandrof)
#### Post date: [August 23, 2016, 6:10am UTC](https://meta.discourse.org/t/ning-import-script/24358/56 "2016-08-23T06:10:33Z")

</div>

@neil Any ideas about how to troubleshoot the above issue?

---

<div class="post-metadata">

### Author: ![neil](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/neil/32/102150_2.png) [@neil](https://meta.discourse.org/u/neil)
#### Post date: [August 23, 2016, 2:42pm UTC](https://meta.discourse.org/t/ning-import-script/24358/57 "2016-08-23T14:42:21Z")

</div>

Hmm, it’s hard to debug someone else’s import work. I read your other topic where you mention that all your upload url’s used “[http://localhost:3000](http://localhost:3000)”. Here you say you ran `rake posts:rebake`, which would probably not go well in a production environment where they are all using localhost:3000. After moving from a dev environment (localhost:3000) to a production server with a real domain, you should fix all the links:

`RAILS_ENV=production bundle exec ruby script/discourse remap 'localhost:3000' 'forum.mydomain.com'`

So my guess is that rebaking all posts with broken links caused them to be lost.

---

<div class="post-metadata">

### Author: ![alehandrof](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/alehandrof/32/119526_2.png) [@alehandrof](https://meta.discourse.org/u/alehandrof)
#### Post date: [August 23, 2016, 3:02pm UTC](https://meta.discourse.org/t/ning-import-script/24358/58 "2016-08-23T15:02:27Z")

</div>

I don’t entirely follow, but I’ll take your word for it. I am puzzled as to why rebaking corrected images, but not other kinds of links.

Is there anything useful I can do in my current situation? (The site is live and has been active with new content since March.)

---

<div class="post-metadata">

### Author: ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [February 6, 2017, 6:35pm UTC](https://meta.discourse.org/t/ning-import-script/24358/59 "2017-02-06T18:35:17Z")

</div>

Hey, @neil. I’ve tried what seems like dozens of times (It might only be 6) to use the Ning Network Archiver to download the data from a forum, but it keeps giving errrors like “couldn’t download x messages” and the resulting JSON files are corrupt. (Not just the stuff that the script fixes, but parts of records being truncated such that parts of values are truncated and fields are missing.)

That was on a Windows machine and a Windows VM running under Linux. Did you download the data? If you did, did you use a Mac? Any other hints about just getting the JSON files downloaded?

---

<div class="post-metadata">

### Author: ![alehandrof](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/alehandrof/32/119526_2.png) [@alehandrof](https://meta.discourse.org/u/alehandrof)
#### Post date: [February 6, 2017, 6:58pm UTC](https://meta.discourse.org/t/ning-import-script/24358/60 "2017-02-06T18:58:40Z")

</div>

FWIW, I did this on a Mac. (And, as you can read above, it wasn’t all puppy dogs and unicorns.)

---

<div class="post-metadata">

### Author: ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [February 6, 2017, 7:03pm UTC](https://meta.discourse.org/t/ning-import-script/24358/61 "2017-02-06T19:03:34Z")

</div>

> [@alehandrof](#):
>
> FWIW, I did this on a Mac.

Sigh. So much for that silver bullet. Did you download Air in August? My next hail Mary will be to see if I can download an older version of Air in hopes that there is some incompatibility between newer versions of Air and the Ning Network Archiver.

My last attempt, this time on a mac, says “26400/26729 files successfully downloaded. 329 files could not be downloaded.” If experience is any indicator, this means that the JSON file is corrupted (perhaps 329 times, but I haven’t gotten that far yet).

---

<div class="post-metadata">

### Author: ![neil](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/neil/32/102150_2.png) [@neil](https://meta.discourse.org/u/neil)
#### Post date: [February 6, 2017, 7:13pm UTC](https://meta.discourse.org/t/ning-import-script/24358/62 "2017-02-06T19:13:07Z")

</div>

I’ve never been the one to get the data from Ning, but your experience sounds typical. I’ve received [corrupted json files and had to fix them](https://github.com/discourse/discourse/blob/master/script/import_scripts/ning.rb#L60-L72) in the importers as best I could. I’ve also seen files that simply stop and can’t be repaired. Typically people need to contact Ning support to receive files that work and have all the data. I’m not sure if that’s possible anymore. If you can reach Ning support, also make sure they send you all replies in each topic (since they’re truncated at 100 or 200), and get them to send you all the attachments too.

---

<div class="post-metadata">

### Author: ![alehandrof](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/alehandrof/32/119526_2.png) [@alehandrof](https://meta.discourse.org/u/alehandrof)
#### Post date: [February 6, 2017, 7:14pm UTC](https://meta.discourse.org/t/ning-import-script/24358/63 "2017-02-06T19:14:09Z")

</div>

I did the migration in March 2016. My Adobe Air is 22.0.0.153 ([how to find it](https://helpx.adobe.com/air/kb/determine-version-air-runtime.html)), but I don’t know what it was at the time.

FWIW, I had that issue as well, but my community site was much smaller. I had 3 files that could not be downloaded either. This was not, as far as I could tell, an indicator that the JSON files were corrupted. Well, that’s partly true. The JSON files that Ning creates are [bleep] [bleeping]ing weird. So most things that read JSON files will have nothing to do with them. But they import fine because Neil’s accounted for their weirdness.

I never did find out exactly what those files were but because we had all critical files backed up elsewhere, and they were so few, we decided to not try to figure this out and go ahead with the migration.

I must have [broken the links to files later](https://meta.discourse.org/t/ning-import-script/24358/58), but given that the site was live at this point, I just stuck with it. I’m pretty sure this is an unrelated issue to the export.

[Previous page](https://meta.discourse.org/t/ning-import-script/24358.md?page=2)

[Next page](https://meta.discourse.org/t/ning-import-script/24358.md?page=4)
