Topic and Category Export/Import

Need to move some stuff from one instance of Discourse to another?

There is a command line tool to export a topic, a set of topics, or an entire category, and another to import them in at the other end.

Topic Export/Import

  • one or more topics and their replies
  • the users who posted in the topic

Category Export/Import

  • the category and all its subcategories
  • its security settings
  • custom groups mentioned in the security settings
    • :warning: if any of the groups have ‘membership requests’ enabled, it will crash. Turn this off before the export
  • all topics and posts in the category and subcategories
  • all users who posted, and assigns them to the groups

Begin: Identify the IDs

First, identify the ID(s) of the =Export= that you wish to export. You’ll find this at the end of the URL used to access it.

  • https://meta.discourse.org/t/how-to-contribute-to-discourse/53797

  • https://meta.discourse.org/t/how-to-contribute-to-discourse/53797/12
    (in this case you are in the middle of a topic - grab the first number)

  • https://meta.discourse.org/c/howto/faq/4

Category or Topic ID(s)

Multiple topics

You can select more than one topic by adding IDs separated by a space; this works in the field above (I don’t think that this works for categories though):

  • 53797 200943 124096

Run the Export Script

The tool can be run as a root user inside the container:

cd /var/discourse
./launcher enter app

Run this command:

discourse export_=Export= =ID=

The output should look something like this:

Starting export of =Export=…

Where are the new users all coming from?
Export saved to =Export=-export-2021-08-27-015437.json

Done

Move the Exported File

The exported .json file needs to be moved from the inside of the application container where you did the export to the inside of the container of the receiving instance. This takes a few steps.

Copy it to the ‘outside’

Exit the container if you haven’t already

exit

You should now have a prompt like this:

root@yourserver:/var/discourse

Use docker to copy the file out of the container (you can’t move it I don’t think)

docker cp app:/var/www/discourse/=FileName= .

Move it to the other instance

This is beyond the scope of this guide, but this might help you get there:
https://www.ubackup.com/windows-server/transfer-files-from-server-to-server.html

Copy it into the container in the receiving instance

SSH into your receiving instance as root and get to the Discourse directory:

cd /var/discourse

Use docker once again to copy the file into the other container

docker cp =FileName= app:/var/www/discourse

Import the Content

Enter the container in the receiving instance:

./launcher enter app

Run the import script

discourse import_=Export= =FileName=

:cake: Done!

You might need to tidy up some loose ends, but hopefully, it should all be done.

Advanced Tips

Missing Ruby Gem

When doing a complex category import, my instance was missing the ‘Listen’ Ruby.
This was solved by doing this: (using the Discourse user inside the container)

./launcher enter app
su discourse
bundle config --delete without
bundle config --delete with
bundle install
exit
exit

This broke the rails console, necessitating a rebuild afterwards

./launcher rebuild app

Multisite

In a multisite environment, use the RAILS_DB env variable.

RAILS_DB=xxxmoves discourse export_=Export= =ID=
RAILS_DB=dancerehab discourse discourse import_=Export= =FileName=

Please report bugs in the #bug category, and request enhancements in the #feature category.

36 Likes

A post was split to a new topic: UI: Topic and Category Export/Import

If a category gets imported, do links get preserved if a bunch of topics refer to each other?

I think that you left out

sudo su -
cd /var/discourse
./launcher enter app

But running the exporter gives this:

discourse@discourse:/var/www/discourse$` bundle exec script/discourse export_category 5`
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:661:in `rescue in connect': FATAL:  database "discourse_development" does not exist (ActiveRecord::NoDatabaseError)

Do we think this should still work, @neil?

It should work. Maybe you need to put RAILS_ENV=production in front of it if it’s your production database you’re trying to export from?

RAILS_ENV=production bundle exec script/discourse export_category 5

5 Likes

That did the trick! I knew that I needed to tell it to use the production database, but had no idea how to accomplish that. Many thanks, @neil.

RAILS_ENV=production
export RAILS_ENV
bundle exec script/discourse export_category 5
5 Likes

After following all steps outlined above I receive the following error:

 Failed to initialize site default
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in `initialize': FATAL:  Peer authentication failed for user "discourse" (PG::ConnectionBad)```

This is on a nearly fresh install from the Digital Ocean droplet image. What have I missed?

Have we checked this recently @neil for any regressions or code change related breakages?

I used it on Car Talk this week to move a category from a sandbox site to the production site. It worked fine for me. Sounds like something different in your install. Can you access db console?

root@gtmdiscourse:~# cd /var/discourse/
root@gtmdiscourse:/var/discourse# ./launcher enter app
root@gtmdiscourse-app:/var/www/discourse# rails db
psql (9.5.3)
Type "help" for help.

discourse=> 
3 Likes

I tried the above on two separate discourse installations. One installation was done the standard docker way, the other was created from the official Digital Ocean snapshot.

I can access db console on both of them.

I have just realised I hadn’t quite followed all the instructions.

The problem I was experiencing was because I hadn’t changed to the discourse user. I thought I had followed all instructions but obviously my tiredness was preventing me from seeing the light. :facepalm:

Thanks guys for building a cool tool. It will really help me out. :clap:

4 Likes

I just wanted to post a quick step by step for anyone else struggling with this who needs a bit more detailed guidance.

Steps to make this work:

  1. SSH to your server where you have Discourse running.

  2. Change directory to /var/discourse.

  3. Enter the docker container running Discourse: ./launcher enter app. If you have called your container something else, replace app with the name of your container.

  4. Change to the discourse user (inside the container): su - discourse.

  5. Navigate to the discourse directory inside the container: cd /var/www/discourse

  6. Run your command, for example:

     RAILS_ENV=production bundle exec script/discourse export_category 1
    

If you follow all of these steps to the letter then it should work for you. I was initially getting confused as to where I should be the discourse user in what directory.

Copying from container to host

By default the file you generate will be stored in the container itself. To copy from the container to the host (your server running the container) run the following:

docker cp <container-id>:<path-to-file-on-container> <path-to-copy-to-on-host>

You will need to replace everything inside of <> with real values. To obtain your container id run docker ps.

My command looked like this:

docker cp a72e14c978ae:/var/www/discourse/category-export-2016-07-25-144953.json /tmp
14 Likes

4 posts were split to a new topic: “Let’s get discussion started” still shows after importing posts

Just a heads up for anyone else using this to do a site migration, this won’t transfer media. You will need to copy it from your existing uploads folder on your server to your new uploads folder.

5 Likes

I was wondering how trivial it would be to also transfer across user site stats (likes, bookmarks etc) and private messages. A few of my regulars have complained that since I did the site migration they’ve lost access to that stuff. I’m sure they built up quite a collection over the two years.

I don’t mind giving it a go myself if somebody can point me in the right direction.

Thanks

Could this reply be incorporated in the main topic post since it provides crucial howto steps that are missing from the original?

3 Likes

Good idea. I linked to his post from near the beginning of the first post.

3 Likes

One heads up, it’s not clear if this still works; My recent attempt to migrate media along with a category hasn’t been successful as of yet.

1 Like

Hi James, I’m a community manager, not a developer but I’m doing some research on ways to export topics from a category and display them in an Excel spreadsheet or similar. Would your instructions allow that, or is your solution for things like migrating the information to another site/platform?

We want to use our community to gather information/ideas and then export it out to allow us to group the data into different themes and further inform some work we’re doing internally.

5 Likes

The data Explorer plugin will let you export any data in discourse to a csv.

4 Likes