Topic and Category Export/Import

[sam (Sam Saffron)]

For CSV you can use Data Explorer, to export the results of a query. We can easily install that for you and create an initial query if you let us know what your spec is for it.

Are you looking for a CSV containing post bodies and links to posts or is this just a category overview?

[pfaffman (Jay Pfaffman)]

I have a potential client who is a hosted customer who wants to merge data from Ning to his existing Discourse instance. This seems like a way to do that.

I don’t see attachments mentioned as part of what gets exported, but it seems like if it didn’t include attachments it wouldn’t be very useful.

[hellekin]

Run your command, for example:

RAILS_ENV=production bundle exec script/discourse export_category 1

This seems to work differently for Multisites. Not sure how to do it though. I was a bit surprised to use DISCOURSE_DB_NAME and it worked… Then during import I understood it was only to (re-)import data already present on this very same site! Using RAILS_DB at least seems to break, so it’s on the right path.

Any more up-to-date documentation on this. Maybe if @hawk shows us how to write a run-book with Discourse we can have that one available!

[neil (Neil Lalonde)]

Exporting and importing from a multisite setup will need RAILS_DB.

RAILS_ENV=production RAILS_DB=sitename bundle exec script/discourse export_category 1
RAILS_ENV=production RAILS_DB=othersite bundle exec script/discourse import_category ...

Using RAILS_DB at least seems to break

How does it break?

[hellekin]

Thank you for your help @neil!

There’s an (unrelated) error message:

/var/www/discourse/app/models/topic_tracking_state.rb:249: warning: circular argument reference - archive_user_id

Then the category-export file is empty: {"categories":[],"groups":[],"topics":[],"users":null}

I tried with RAILS_DB=sitename where entries in data.yml are like sitename: followed by the list of pups commands, addressing a database name that’s different. I tried adding DISCOURSE_DB_NAME as well without result. I got the category_id from the Rails console: Category.find_by_name('Example Name').id

[neil (Neil Lalonde)]

That’s only a warning. The output should look like this:

Starting export of categories...

/var/www/discourse/app/models/topic_tracking_state.rb:249: warning: circular argument reference - archive_user_id
Topic title here
Another topic title
Export saved to category-export-2018-03-19-142232.json

Done

What’s your output?

[hellekin]

Hi @neil, sorry I didn’t respond, I was swamped under the workload but now it’s over and I may start looking into it again, although I don’t really have a reason anymore. It might be some time before it happens though. Just notifying the topic so others can take over if they encounter the same difficulties.

[evilnick (Nick Veitch)]

Slightly related question - is there any way to extract the raw text of a post from discourse, rather than a cooked string?
I expect this can probably be done through postgres, but I was hoping there was a less hacky way

[pfaffman (Jay Pfaffman)]

Something like https://meta.discourse.org/raw/38930/29

[evilnick (Nick Veitch)]

ah, many thanks , I didn’t see that mentioned in the API docs

[tophee (Christoph)]

This could be used for splitting up a forum into two forums, for example when some sub-community/category wants to run its own community on a separate site, right?

[codinghorror (Jeff Atwood)]

Yes, that is definitely one of the goals here.

[Walker_Blackwell]

Does this Export/Import fully support attachments yet?

I see some very bare code that does export an //upload id in the post json but this seems not to properly call when doing an import back into another discourse and also doesn’t seem to be pointed to an actual file in the uploads (X1,X2, etc) directories.

What would be good.

1. Export all attachment urls to json (the original files) and also cp these files to an export folder (next to the json)
2. Then ideally there would be an arg that could be passed during the import_topics or import_category that would tell the import script the location of the attachment folder.
3. Then attachments + json are imported and optimized images are created for each jpg,png, etc.

Does this already exist somewhere already and I’m just not seeing it?

best all,
Walker

[neil (Neil Lalonde)]

This tool doesn’t support that, but the discourse_merger tool does.

How to merge two Discourse sites

If you have two Discourse sites that you wish were one, this guide is for you. There’s a tool called discourse_merger that can take one Discourse site and merge it into another. Prereqs This is not an easy task, and should be treated like any other migration to Discourse. You will not be running discourse_merger on a live production site. You will perform the merge in another environment where you can review the output before moving the result to production. Copy vs Merge Almost everything …

[Walker_Blackwell]

oh jeez. it’s done then.

Thank you @neil you are a life saver.

-W

[hellekin]

Today we moved a category with 2 groups and a dozen users from one instance to the next. It took about half an hour for a group of three people (one admin on site 1, another on site 2, a coordinator).

Step 1: backup

• Coordinator announces move:
  About the Prepare FOSDEM 2019 category - Prepare FOSDEM 2019 - Enough
• Coordinator changes group permissions to ‘See’ (to keep groups across the migration while preventing new edits)
• Admin makes a backup
• Admin runs RAILS_ENV=production script/discourse export_category 15

Note: the category number was taken from the JSON representation of the category.

Step 2: restore

We passed the file to Admin2, and he put it in shared/app/ to have it available in the container.

Running RAILS_ENV=production script/discourse import_category /shared/file.json created the users and categories. But somehow the restoration does not seem to have worked in full:

About the Prepare FOSDEM 2019 category - Prepare FOSDEM 2019 - Enough looks very different from https://socialhub.network/t/about-the-prepare-fosdem-2019-category/288 (from -> to)

We’re looking into solutions to fix the unmigrated stuff.

So here’s the cutoff:

• posts in the “About this category” topic (the category’s description topic) are not imported.
• likes seem not to be imported, but that might just be a side-effect of the importing site being frozen, and might be fixed by unfreezing the site.

We’re going to export the topic and import it again to see how it works.

---

We completed the move. Here are the caveats:

• users who were part of the original group but did not post were not imported.
• likes were not imported
• wiki status of some posts was not imported (this may be due to the lower trust level at the destination)
• user avatars were not imported, unless it was a gravatar
• notification level was ignored
• the category topic’s posts were imported using export_topic and import_topic, then posts manually moved to the existing topic.

Edit: here’s another couple of things that missed the import:

• associated links
• oneboxes (requires rebuilding HTML)

Before:

After:

End of Report.

[gslexie (Graham Lexie)]

Exports I think are broken in the most recent version:

Looks like the script at lib/import_export/importexport.rb was recently moved to lib/import_export.rb (https://github.com/discourse/discourse/commit/427d54b2b00fa94474c0522eaed750452c4e7f43), but the ./script/discourse.rb still refers to the old location :

  def load_import_export
    require File.expand_path(File.dirname(__FILE__) + "/../lib/import_export/import_export")
  end

For now, there’s an easy workaround: just locate this function and fix the broken path manually. It worked great for me after I figured that out.

[kris.kotlarek (Kris)]

Hey Graham,

Thank you that you found this, I will take care of that.

[Robert_Fay]

We are currently thinking of taking a category that we have making it into separate forum of it’s own. It seems like this tool would work for situation. My questions is are we able to redirects in for the pages that are linked to from external sites. I want to make sure someone coming into one of the posts does not get a page doesn’t exist page. Is that possible?