Questions about importing json

Development

AI-generated summary

Importing JSON Data into Discourse

A user with limited Ruby knowledge is attempting to import data into Discourse using the json_generic.rb script. They have several questions regarding the format of the data and the available fields for importing users, topics, and posts.

For importing users, the create_users function requires specific fields, including id, email, and optional fields such as bio_raw, website, location, avatar_url, name, username, trust_level, active, import_mode, last_emailed_at, and password. The user is unsure about the required data format and whether it can be passed in as straight JSON.

The user is also unsure about importing the original creation date of users, topics, and posts. They note that the json_generic.rb script appears to handle this data, but they are not sure how to format it correctly.

mbcahyono provides an example of the expected JSON format for importing topics, which includes nested posts. The format includes fields such as id, title, pinned, posts, id, body, date, and author.

The original poster, tehspaceg, is struggling to merge their topics and posts files into a single file and is seeking guidance on how to format the data correctly for import.

1

I’ll preface that my knowledge of Ruby is essentially zero, so sorry if anything I say sounds dumb.

While I wait for someone in Marketplace on my bounty, I figured I could poke around a bit at importing data on my own. I see that there is a json _generic.rb file in GitHub, which requires the base.rb file.

I’ve got a few rookie questions and some questions on what kind of data the base.rb script is looking for.

create_users
It looks like this function is designed to iterate through a data set and send it over to create_user. For create user, is there a list of available fields that I can pass into this function?

I have a rough idea from the code, but I want to make sure I know all of them. Here is the list I collected; id (required, from original source), bio_raw, website, location, avatar_url, name, username, email (required), trust_level, active, import_mode, last_emailed_at, password

What data format is needed? Is it just straight json that I can pass in to create_users? Or is there a certain Ruby format I should follow?

Is it possible to import the original creation date of the user for consistency?

create_posts
This one will be the hardest part. The create_post function appears to only take the post without looking at topics. The way NodeBB (my data source) stores data is to have a topic and posts that are linked to that topic via topic ID. I’m not seeing anything in this function that creates the topics, then handles importing posts to link to that topic. Am I missing something?

Similar to users, I don’t see a field that would take creation date/time in account for topics or posts. The json_generic.rb file appears to take that data, so I assume it gets done at some point.

json_generic.rb
Last set of questions is for this one. I’m trying to understand what kind of format it expects the imported json data to be in. It looks like it is expecting one file with all of the content. Here is an example (missing some fields) that I think it is looking for:

{"topics": [
	{
		"tid": 2,
		"cid": 1,
		"title": "topic_title",
		"user_id": "string_username",
		"raw": "content of first post",
		"blah": "blah"
	},
	{
		"tid": 3,
		"cid": 2,
		"title": "topic_title",
		"user_id": "username for first post",
		"raw": "content of first post",
		"blah": "blah"
	}],
"posts": [
	{
		"pid": 89,
		"user_id": "username",
		"raw": "post content",
		"timestamp": "date"
	}],
"users": [
	{
		"uid": 1,
		"username": "teh g",
		"email": "fake@fake.xyz",
		"joindate": "date"
	}]
}
2

I didn’t test it myself, but after reading the code, I think the format of the JSON should be similar to this:

  {
    "topics": [
      {
        "id": 1,
        "title": "The title",
        "pinned": false,
        "posts": [
          {
            "id": 2,
            "body": "The body",
            "date": "The date",
            "author": "username"
          }
        ]
      }
    ]
  }
3

Perfect, thanks for taking a look. I’ll have to figure out a good way to merge my topics and posts files into one, but I think I can come up with something for that.