Topic and Category Export/Import

import

(James Cook) #7

After following all steps outlined above I receive the following error:

 Failed to initialize site default
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in `initialize': FATAL:  Peer authentication failed for user "discourse" (PG::ConnectionBad)```

This is on a nearly fresh install from the Digital Ocean droplet image. What have I missed?

(Jeff Atwood) #8

Have we checked this recently @neil for any regressions or code change related breakages?


(Neil Lalonde) #9

I used it on Car Talk this week to move a category from a sandbox site to the production site. It worked fine for me. Sounds like something different in your install. Can you access db console?

root@gtmdiscourse:~# cd /var/discourse/
root@gtmdiscourse:/var/discourse# ./launcher enter app
root@gtmdiscourse-app:/var/www/discourse# rails db
psql (9.5.3)
Type "help" for help.

discourse=> 

(James Cook) #10

I tried the above on two separate discourse installations. One installation was done the standard docker way, the other was created from the official Digital Ocean snapshot.

I can access db console on both of them.


(James Cook) #11

I have just realised I hadn’t quite followed all the instructions.

The problem I was experiencing was because I hadn’t changed to the discourse user. I thought I had followed all instructions but obviously my tiredness was preventing me from seeing the light. :facepalm:

Thanks guys for building a cool tool. It will really help me out. :clap:


(James Cook) #12

I just wanted to post a quick step by step for anyone else struggling with this who needs a bit more detailed guidance.

Steps to make this work:

  1. SSH to your server where you have Discourse running.

  2. Change directory to /var/discourse.

  3. Enter the docker container running Discourse: ./launcher enter app. If you have called your container something else, replace app with the name of your container.

  4. Change to the discourse user (inside the container): su - discourse.

  5. Navigate to the discourse directory inside the container: cd /var/www/discourse

  6. Run your command, for example:

     RAILS_ENV=production bundle exec script/discourse export_category 1
    

If you follow all of these steps to the letter then it should work for you. I was initially getting confused as to where I should be the discourse user in what directory.

Copying from container to host

By default the file you generate will be stored in the container itself. To copy from the container to the host (your server running the container) run the following:

docker cp <container-id>:<path-to-file-on-container> <path-to-copy-to-on-host>

You will need to replace everything inside of <> with real values. To obtain your container id run docker ps.

My command looked like this:

docker cp a72e14c978ae:/var/www/discourse/category-export-2016-07-25-144953.json /tmp

(Erlend Sogge Heggen) #13

4 posts were split to a new topic: “Let’s get discussion started” still shows after importing posts


(James Cook) #14

Just a heads up for anyone else using this to do a site migration, this won’t transfer media. You will need to copy it from your existing uploads folder on your server to your new uploads folder.


(James Cook) #15

I was wondering how trivial it would be to also transfer across user site stats (likes, bookmarks etc) and private messages. A few of my regulars have complained that since I did the site migration they’ve lost access to that stuff. I’m sure they built up quite a collection over the two years.

I don’t mind giving it a go myself if somebody can point me in the right direction.

Thanks


(Stefano Costa) #16

Could this reply be incorporated in the main topic post since it provides crucial howto steps that are missing from the original?


(Neil Lalonde) #17

Good idea. I linked to his post from near the beginning of the first post.


(Matthew Gaudet) #18

One heads up, it’s not clear if this still works; My recent attempt to migrate media along with a category hasn’t been successful as of yet.


(Yvonne Brill) #19

Hi James, I’m a community manager, not a developer but I’m doing some research on ways to export topics from a category and display them in an Excel spreadsheet or similar. Would your instructions allow that, or is your solution for things like migrating the information to another site/platform?

We want to use our community to gather information/ideas and then export it out to allow us to group the data into different themes and further inform some work we’re doing internally.


(Jay Pfaffman) #21

The data Explorer plugin will let you export any data in discourse to a csv.


(Sam Saffron) #22

For CSV you can use Data Explorer, to export the results of a query. We can easily install that for you and create an initial query if you let us know what your spec is for it.

Are you looking for a CSV containing post bodies and links to posts or is this just a category overview?


(Jay Pfaffman) #23

I have a potential client who is a hosted customer who wants to merge data from Ning to his existing Discourse instance. This seems like a way to do that.

I don’t see attachments mentioned as part of what gets exported, but it seems like if it didn’t include attachments it wouldn’t be very useful.


#24

This seems to work differently for Multisites. Not sure how to do it though. I was a bit surprised to use DISCOURSE_DB_NAME and it worked… Then during import I understood it was only to (re-)import data already present on this very same site! Using RAILS_DB at least seems to break, so it’s on the right path. :wink:

Any more up-to-date documentation on this. Maybe if @hawk shows us how to write a run-book with Discourse we can have that one available!


(Neil Lalonde) #25

Exporting and importing from a multisite setup will need RAILS_DB.

RAILS_ENV=production RAILS_DB=sitename bundle exec script/discourse export_category 1
RAILS_ENV=production RAILS_DB=othersite bundle exec script/discourse import_category ...

How does it break?


#26

Thank you for your help @neil!

There’s an (unrelated) error message:

/var/www/discourse/app/models/topic_tracking_state.rb:249: warning: circular argument reference - archive_user_id

Then the category-export file is empty: {"categories":[],"groups":[],"topics":[],"users":null}

I tried with RAILS_DB=sitename where entries in data.yml are like sitename: followed by the list of pups commands, addressing a database name that’s different. I tried adding DISCOURSE_DB_NAME as well without result. I got the category_id from the Rails console: Category.find_by_name('Example Name').id


(Neil Lalonde) #27

That’s only a warning. The output should look like this:

Starting export of categories...

/var/www/discourse/app/models/topic_tracking_state.rb:249: warning: circular argument reference - archive_user_id
Topic title here
Another topic title
Export saved to category-export-2018-03-19-142232.json

Done

What’s your output?