[Complete] [Paid] Migration script from NodeBB (redis) to Discourse

I’d definitely love to work with you. You’ve clearly done some great work on the project!

I personally don’t need categories. I’d come to terms with it being somewhat easier to manually make categories and just replace the category ID in my data with the new categories. That would definitely break the spirit of the full import though.

I definitely don’t need anything for doing 301s or avatars. Many users on my board use Gravatar or can just reupload. I don’t think redirects will add enough value to be worth spending time on. I’d just let people know they can expect some broken links for self-referencing links.

Do you want the Redis rdb file or did I turn this into a refresh of the json importer? I think to keep with the Discource folks plans of covering half the cost, we’d have to do the rdb file.

Email me at support@literatecomputing.com.

I’m not sure. I think what I want is JSON files, since you started talking about JSON files, but we can work that out via email.

1 Like

I think a better approach will be connecting to the NodeBB database and get the data directly, as other importers do. I wrote an importer for a custom forum so I don’t think it will be that complicated.

What Data store are you using for the NodeBB forum? MongoDB or Redis?

2 Likes

I agree in principle, at least. It appears to be in mongodb. I’ve not used Mongodb, so it’ll take me a couple hours to figure out how to install it, get data into it, and so on before I pass Go.

The yahoogroup importer uses mongodb, so there’s a starting place.

If you’re interested, please contact @tehspaceg, and, especially if you can hit the $600 budget, please let me know so that I don’t spend the time coming up with an estimate.

1 Like

I definitely agree that the better route would be straight from the source DB into Discourse. It isn’t too complicated to extract the data via scripts into json format. The most painful one was the users, but that was because I didn’t want to write a Python script to iterate through the API, so I just did it manually.

Unfortunately I’m using Redis. Before about a year ago, the recommendation was to use Redis as the database. It looks like NodeBB is transitioning to MongoDB as the recommended database. They still officially support both, and they have not published their Redis -> MongoDB converter yet (they hold it behind a paywall).

My data is redis. I just exported it out into json to make life “easier”.

2 Likes

Cool, way off my normal rate, but since it’s going to be open sourced, I could probably do it.

@tehspaceg send me a PM or an email at orlando@hashlabs.com

@codinghorror do you guys want to chime in for a NodeBB Redis only importer implementation?

1 Like

I think we can contribute ~$1000 to this effort if @erlend_sh is OK with it?

12 Likes

Cool, if that’s the case I’ll do it. Payment after delivery since it’s my first gig on the #marketplace

6 Likes

Email sent.

Thanks for pitching in @codinghorror

3 Likes

Quick update; this is being worked by @eatcodetravel, pending some confirmation of @codinghorror/Discourse’s generous offer.

2 Likes

Consider it confirmed :+1:

8 Likes

Update

Working on this on the weekends at the moment, but I’ll put some time also early next week. I’m expecting on having it ready by middle next week.

As for now, we have migrated

  • Groups
  • Categories
    • For categories, since Discourse doesn’t support more than 1 level of nesting in children categories, I’m putting all the categories under the first parent. So it will look like this.
      Category -> Sub Category -> Sub Sub Category
      Category -> Sub Category, Sub Sub Category

Cheers

5 Likes

Thanks for the update. I think that category mapping makes sense. Discourse handles categories better than NodeBB (IMO), so I was planning on shifting things around anyway.

1 Like

On thing that I’ve done for some imports is map sub-sub categories to Discourse tags. It’s always on a per-job basis, though, as there’s no automated way to guess what the right thing is. The map sub-subs up a level is a good solution. I think many importers just bump them to the top level.

3 Likes

Quick update, this is how it looks now

We have migrated

  • Categories
  • Groups
  • Group membership
  • Users
  • Topics and Posts

There still a few things I need to resolve, like supporting local attachments, this nodebb dataset uses a plugin to upload all attachments to imgur, but I can test that with my own installation.

I won’t be able to work on this until the weekend, but hopefully I should be able to wrap up this by next Monday.

Thanks!

12 Likes

I think that there are a couple scripts that download images or you could just leave the links to imgur and optionally let Discourse pull them to local.

2 Likes

Yeah, for external images I’m not doing anything, but in the case of avatars I’ll save them as Discourse uploads. I’ll double check what other options I have.

Also, I need to fix references to posts inside posts, so these are not broken after the migration.

I’ll post here if I encounter any issue.

2 Likes

Nice! This is looking great.

I personally can live without avatar migration. I think of all the things that is simple enough to tell users to re-upload manually, especially since I think the imgur plugin forced all of those to get uploaded there instead of locally. That is likely an edge case for NodeBB users.

1 Like

I just created a PR for this https://github.com/discourse/discourse/pull/5842

Thanks to Discourse and @tehspaceg for sponsoring this work :heart:

11 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.