How to use the bbpress import script - or any other import script with mysql dependency

import

#1

So, you’ve been weeping when you tried to execute the import script inside the docker container, it telling you that you haven’t got the mysql gem installed even though you have. You’ve been trying for hours to setup a dev instance of discourse, failing hard. Weep no more, this will solve all your problems.

First we are going to set up a docker instance called ‘import’, which includes the mysql dependency, then we will set up the mysql database as an additional docker container ( if you cant access it otherwise) and then we will run the import script.

1.1. Installing Discourse

Install Discourse by following the official installation guide.

1.2. Preparing the Docker container

Copy the container configuration file app.yml to import.yml and edit it with your favorite editor.

cd /var/discourse
cp containers/app.yml containers/import.yml

nano containers/import.yml

Now inside containers/import.yml add - "templates/import/mysql-dev.template.yml" to the list of templates. Afterwards it should look something like this:

templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
  #- "templates/web.ssl.template.yml"
  #- "templates/web.letsencrypt.ssl.template.yml"
  - "templates/import/mysql-dev.template.yml"

That’s it. You can save the file, close the editor and build the container and you will finally have mysql support inside your container.

/var/discourse/launcher stop app
/var/discourse/launcher rebuild import

2. Setting up the mysql database - optional

You don’t have to do this step, if you can already access your database from within the docker container.

Put the mysql backup onto the machine you are running the discourse instance, change to the directory that contains the backup and start a mysql database instance like this:

docker run -d -e MYSQL_ROOT_PASSWORD=pass -e MYSQL_USER=user -e MYSQL_PASSWORD=pass -e MYSQL_DATABASE=db -v "$PWD":/backup --name=mysql mysql

Now enter the docker instance

docker exec -it mysql bash

and import the database

mysql -u$MYSQL_USER -p$MYSQL_PASSWORD $MYSQL_DATABASE < /backup/yourbackup.sql
logout

finally grep your instances IP address for usage during the import

docker inspect mysql | grep IPAddress

3. Running the import script

Enter the import container you’ve created

cd /var/discourse
./launcher enter import

Open the bbpress import script

nano script/import_scripts/bbpress.rb

and adapt the mysql configuration

@client = Mysql2::Client.new(
      host: "172.17.0.X",
      username: "user",
      database: "db",
      password: "pass",
    )

Now you are good to go, to import run the script like this

su discourse -c "bundle exec ruby script/import_scripts/bbpress.rb"

Hopefully it succeeded and you can now move on to your next big thing!

Thx to @gerhard for his post on importing phpbb3


Importing bbPress into Discourse
Migrating from bbPress WordPress plugin to Discourse
Issues while migrating from bbpress
Bulk deleting / mass modifying users
How to migrate from SMF2 to Discourse
Drupal import script for beginner QQ
SMF2 to discourse - Troubles importing attachments
(Walker Blackwell) #2

This worked for me. Thank you.

I have one question. My current dilemma is this.

I started with two forums that I’m working on merging.

First forum: Vbulletin 4. Migration went smooth, no problems.

Second forum: BBPress. Migration into a new fresh discourse install is fine but I want to merge it with the existing forum that I migrated VB4 to.

In my offline docker I was able to merge existing users from BBpress with the 1st discourse forum but of course it’s just an ID to ID merge so user post data from BBpress is not mapped to the correct users on the 1st forum on import. It would be very useful to match emails instead of user IDs on import so that existing users from both forums are merged correctly. As a second, if a user email from the BBpress forum does not match an existing discourse user but their ID does match, it should actually give the new user a new unique ID so merging does not happen at that point.

Is this at all possible do you think? I am not code-minded so would not exactly know how to start in on this with Ruby . . . Is it possible to merge existing Discourse forums?

This seems like something that would be useful across all import scripts now that I think of it. This MERGING capability would be very useful in many situations . . . .

All the best,
Walker


(Mittineague) #3

It might be easier to merge the two before importing to Discourse. That is, merge the bbPress into vBulletin or vBulletin into bbPress, and then merge into Discourse.


(Jay Pfaffman) #4

@Mittineague is right, it’d be easier, and not require any coding, to merge the data and then run one of the other of those import scripts.

The logic in the solution that you describe seems pretty good. Implementing that, if you are “not code-minded”, is not something that can be explained here. That kind of work is fairly fussy, and the devil is in the details.

How to deal with a user with multiple email addresses and how to merge them comes up here with some frequency, and there is a move afoot to do that. It doesn’t really happen that often in the process of importing, in my experience.

If you have a budget for this work, post in #marketplace and someone will likely be available to help.


(Walker Blackwell) #5

Thank you for the replies. I will investigate the pre-merge first.

No budget as of yet for it, but possibly down the road.

cheers!
Walker


(Marius) #7

finaly works for me also.
but:
- "templates/import/mysql-dev.template.yml"
must be:
- "templates/import/mysql-dep.template.yml"

also on:

nano script/import_scripts/bbpress.rb

you must also add user,pass and database name, data base prefix if is not default one wp_ (of the imported mysql database) for this to work

here:

BB_PRESS_HOST ||= ENV['BBPRESS_HOST'] || "localhost"
BB_PRESS_DB ||= ENV['BBPRESS_DB'] || "bbpress"
BATCH_SIZE ||= 1000
BB_PRESS_PW ||= ENV['BBPRESS_PW'] || ""
BB_PRESS_USER ||= ENV['BBPRESS_USER'] || "root"
BB_PRESS_PREFIX ||= ENV['BBPRESS_PREFIX'] || "wp_"

like this works in my case.


(Vanina Yordanova) #8

@lead4good, really good explanation!!!
But I have some question!

My case:

I am not gonna use docker container. I want to instal Discourse on ubunto and migrate all posts from lets say local drupal website to discourse. So I assume I need to import the Drupal script. Thats is good! BUT…!
The re is existing Drupal website on an inline server. The idea is to recreate the website using WordPress, migrate all users to Wordpress and than use SSO so they can directly login to the Discourse Forum.

I can easily migrate users from Drupal to Wordpress and enable SSO, but how to move the posts of the users and assigned them to specific users, since I am not migrate anybody to Discourse???


(Jay Pfaffman) #9

The import script will import the users to discourse. The SSO stuff should just work out


(Tumi) #10

Pls tell me How , any good plugin for begginer?


(Tumi) #11

there is still no importer for ip.board?


(Jay Pfaffman) #12

I have an ipboard importer. It’s not quite ready for prime time. The next time I get a job to use it I plan to submit a PR, but until then it’ll be hard to make time for that.


(Vanina Yordanova) #13

Use this import script . You just need to add your database credientials and include it into discourse/import_scripts in your Discourse instance. It works perfectly.


(Anuj Saini) #14

Newbie here, in which directory I have to put yourbackup.sql file ?


(Jay Pfaffman) #15

In that example, you’d put it in /backup


(Anuj Saini) #16

I have created “backup” directory in root and placed “yourbackup.sql” in it, getting following error.

root@jkcf-new:~# docker exec -it mysql bash
root@9a490d9238f7:/# mysql -u$MYSQL_USER -p$MYSQL_PASSWORD $MYSQL_DATABASE < /backup/yourbackup.sql
bash: /backup/yourbackup.sql: No such file or directory

(Anuj Saini) #17

Finally working for me, used mysql remote connection. Before following procedure in the post enable mysql remote access How Do I Enable Remote Access To MySQL Database Server? - nixCraft


#20

Hi and thanks for the How to!

I achieved a migration and the new forum works fine, but I have a question… When I list the dockers currently running I have “import” (running for 5 weeks) and “app” (Exited (10) 5 weeks ago). So, it means my discourse forum is runing on the “import” docker… Should I stop it and relaunch “app” one? Will it create any issues? What are the commands to do it in a clean way (and also remove the import docker)?


(Carl Moebis) #21

I’ve run this several times over the last couple of weeks and it worked. I restored a fresh snapshot, updated discourse and went through the instructions again. Now at the very last step I get this:

su discourse -c "bundle exec ruby script/import_scripts/bbpress.rb"

loading existing groups...

loading existing users...

loading existing categories...

loading existing posts...

loading existing topics...

/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/mysql2-0.5.1/lib/mysql2/client.rb:90:in `connect': Authentication plugin 'caching_sha2_password' cannot be loaded: /usr/lib/mysql/plugin/caching_sha2_password.so: cannot open shared object file: No such file or directory (Mysql2::Error)

from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/mysql2-0.5.1/lib/mysql2/client.rb:90:in `initialize'

from script/import_scripts/bbpress.rb:19:in `new'

from script/import_scripts/bbpress.rb:19:in `initialize'

from script/import_scripts/bbpress.rb:510:in `new'

from script/import_scripts/bbpress.rb:510:in `&lt;main&gt;'

(Carl Moebis) #22

I will answer my own question, since I have found a solution:

Go back into your mysql container:

docker exec -it mysql bash

Enter mysql:

mysql -uuser -ppass

Then run:

ALTER USER 'username' IDENTIFIED WITH mysql_native_password BY 'pass';
exit
exit
./launcher enter import
su discourse -c "bundle exec ruby script/import_scripts/bbpress.rb"

(Carl Moebis) #23

Ok new problem, any idea why some attachments/inline images come over and other do not? Most are still linked or refer to the old URL, some with Lightbox effect, some without. We’ve done several tests by attaching images to bbpress posts in different ways to see what effects it, no rhyme or reason.