Questions about importing json

dev
1

I’ll preface that my knowledge of Ruby is essentially zero, so sorry if anything I say sounds dumb.

While I wait for someone in #marketplace on my bounty, I figured I could poke around a bit at importing data on my own. I see that there is a json _generic.rb file in GitHub, which requires the base.rb file.

I’ve got a few rookie questions and some questions on what kind of data the base.rb script is looking for.

create_users
It looks like this function is designed to iterate through a data set and send it over to create_user. For create user, is there a list of available fields that I can pass into this function?

I have a rough idea from the code, but I want to make sure I know all of them. Here is the list I collected; id (required, from original source), bio_raw, website, location, avatar_url, name, username, email (required), trust_level, active, import_mode, last_emailed_at, password

What data format is needed? Is it just straight json that I can pass in to create_users? Or is there a certain Ruby format I should follow?

Is it possible to import the original creation date of the user for consistency?

create_posts
This one will be the hardest part. The create_post function appears to only take the post without looking at topics. The way NodeBB (my data source) stores data is to have a topic and posts that are linked to that topic via topic ID. I’m not seeing anything in this function that creates the topics, then handles importing posts to link to that topic. Am I missing something?

Similar to users, I don’t see a field that would take creation date/time in account for topics or posts. The json_generic.rb file appears to take that data, so I assume it gets done at some point.

json_generic.rb
Last set of questions is for this one. I’m trying to understand what kind of format it expects the imported json data to be in. It looks like it is expecting one file with all of the content. Here is an example (missing some fields) that I think it is looking for:

{"topics": [
	{
		"tid": 2,
		"cid": 1,
		"title": "topic_title",
		"user_id": "string_username",
		"raw": "content of first post",
		"blah": "blah"
	},
	{
		"tid": 3,
		"cid": 2,
		"title": "topic_title",
		"user_id": "username for first post",
		"raw": "content of first post",
		"blah": "blah"
	}],
"posts": [
	{
		"pid": 89,
		"user_id": "username",
		"raw": "post content",
		"timestamp": "date"
	}],
"users": [
	{
		"uid": 1,
		"username": "teh g",
		"email": "fake@fake.xyz",
		"joindate": "date"
	}]
}
2

I didn’t test it myself, but after reading the code, I think the format of the JSON should be similar to this:

  {
    "topics": [
      {
        "id": 1,
        "title": "The title",
        "pinned": false,
        "posts": [
          {
            "id": 2,
            "body": "The body",
            "date": "The date",
            "author": "username"
          }
        ]
      }
    ]
  }
3 Likes
3

Perfect, thanks for taking a look. I’ll have to figure out a good way to merge my topics and posts files into one, but I think I can come up with something for that.