phpBB 3 Importer (old)


(Neil Lalonde) #1

NOTE: This guide is out-dated. Follow this guide instead!


I committed a script to import from a phpBB 3 to Discourse, similar to our Drupal, Kunena, and bbPress importers. Contributions and fixes are welcome!

Thanks to ruby-bbcode and @haiku’s ruby-bbcode-to-md, some bbcode can optionally be converted to markdown before importing into Discourse. I made a fork that does some conversion that is specific for Discourse. The script will also handle some other phpbb-specific markup, like smileys and links that use <!-- m -->, etc.

Usage:

Edit the bbpress3.rb file to change the MySQL connection info (PHPBB_DB, and Mysql2::Client.new(...)) so it connects to a local copy of the phpBB database.

To convert phpBB’s bbcode to markdown before importing, install the ruby-bbcode-to-md gem. In a temp directory:

git clone https://github.com/nlalonde/ruby-bbcode-to-md.git
cd ruby-bbcode-to-md
gem build ruby-bbcode-to-md.gemspec
gem install ruby-bbcode-to-md-0.0.13.gem

Run the import, from the discourse project directory:

ruby script/import_scripts/phpbb3.rb bbcode-to-md

Is migrating from phpBB to Discourse a good idea
Upgrading an old phpBB client forum to Discourse?
phpBB conversion and general (Linux / Docker / Ruby / GEMs) help
phpBB 3 Importer goes slow
Can't start PHPBB3 Importer
Can't start PHPBB3 Importer
(InsaneMosquito) #2

To confirm what I think I’m reading, we need to have a copy of the phpbb database and mysql on the same server we are running Discourse on, correct?


(Neil Lalonde) #3

That’s how I’ve been doing it, but if you can connect to a remote mysql server, then that will work too. Just update the connection info to whatever works in the script.

    @client = Mysql2::Client.new(
      host: "localhost",
      username: "root",
      #password: "password",
      database: PHPBB_DB
    )

Connecting to the live production database is probably a bad idea, since data will be changing.


(InsaneMosquito) #4

If the PHPBB board (gah, to many 'B’s) is closed for maintenance though, that shouldn’t be a problem as very few people with specific privileges would be able to do anything on the board at the time.


(Rikki Tooley) #5

I’m glad you found what I did useful… I believe this is the first time code I wrote has been adopted by an open-source project :smiley:


#6

Congrats! It really helped me converting our forum over so my personal thanks for all your hard work.


#7

Neil, as a creator of many import scripts, cleanup scripts and scheduled tasks for processes, I just want to say that there is a special rung in hell for import processes.

Seriously though, I give you much respect for doing ‘invisible-to-end-user’ work.


(Wint) #8

Probably a noobish question but I’ve edited the file and scp’d it to my server, where is the script/import_scripts folder if you used the docker setup?

Also, does this take care of not spamming users when new posts are created?


(Gerhard Schlager) #9

Great script. Any chance that it could import polls as well?
Or should I wait and hope that the official importer will support this?


(Rikki Tooley) #10

@gerhard This *is* the official importer :stuck_out_tongue:


(Marty ) #11

I’m trying out the phpbb3.rb conversion script and am having a little trouble. I suspect it’s a basic ruby thing… I’ve new to ruby.

What I’ve done:

  1. install discourse locally with discourse_docker
  2. Signed into discourse and created my admin account. Things look good, basically empty forum.
  3. Entered the discourse conatiner
  4. Edited phpbb3.rb to set the mysql setting
  5. installed mysql-client, libmysql-dev,
  6. gems install mysql2

I’m getting an error about missing roles in my DB. Does anyone have a suggestion? I know the stack trace suggest that I create a DB, but I already have one :slight_smile: The stack trace is below

root@ubuntu-app:/var/www/discourse/script/import_scripts# ruby phpbb3.rb 
2014-08-03T19:54:58Z 1990 TID-otwl8st54 INFO: Sidekiq client with redis options {:url=>"redis://localhost:6379/0", :namespace=>"sidekiq"}
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.4/lib/active_record/connection_adapters/postgresql_adapter.rb:898:in `rescue in connect': FATAL:  role "root" does not exist (ActiveRecord::NoDatabaseError)
Run `$ bin/rake db:create db:migrate` to create your database
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.4/lib/active_record/connection_adapters/postgresql_adapter.rb:888:in `connect'
	from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-
...
	from /usr/local/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /var/www/discourse/script/import_scripts/base.rb:19:in `initialize'
	from phpbb3.rb:14:in `initialize'
	from phpbb3.rb:295:in `new'
	from phpbb3.rb:295:in `<main>'

(Gerhard Schlager) #12

Try running it with sudo:
sudo -E -u discourse ruby phpbb3.rb


(Marty ) #13

@gerhard Thank you for the advice.

I got a different error, same stack. Below is the top of the stack. Seem like it’s looking for a discourse_development db?

root@ubuntu-app:/var/www/discourse/script/import_scripts# sudo -E -u discourse ruby phpbb3.rb 
2014-08-03T21:36:25Z 150 TID-oumvhi6j8 INFO: Sidekiq client with redis options {:url=>"redis://localhost:6379/0", :namespace=>"sidekiq"}
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.4/lib/active_record/connection_adapters/postgresql_adapter.rb:898:in `rescue in connect': FATAL:  database "discourse_development" does not exist (ActiveRecord::NoDatabaseError)
Run `$ bin/rake db:create db:migrate` to create your database

(Jens Maier) #14

You need to set the Rails environment:
RAILS_ENV=production sudo -E -u discourse ruby phpbb3.rb

Also, you can avoid the repetitive sudo cruft by becoming the discourse user instead of editing and running these scripts as root. Just run su - discourse after entering the container.


(Marty ) #15

@elberet Sweet! Thanks that was it.

Is it work adding some documentation around this for ruby newcomers like me? I could write up a README.md for discourse/script/import_scripts?


(Marty ) #16

I found one minor issue:

creating users
Skipping user id 1 because email is blank
      641 / 1041 ( 61.6%)    Skipping user id 68191 because email is blank
     1043 / 1041 (100.2%)    
creating categories
    Admin
    The Kiteboarding
    The Gear
    Everything Else
    Snowboarding/Ski
    Other sports - MTB, Wake, SUP, Surf and anything else
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.4/lib/active_record/validations.rb:57:in `save!': Validation failed: Category Name is too long (maximum is 50 characters) (ActiveRecord::RecordInvalid)

I think this can be fixed by changing the mysql select to trim category names to 50 chars. I’ll put that in a pull request.

https://github.com/discourse/discourse/pull/2618


(Gerhard Schlager) #17

I’ve tried running the importer again - this time using bbcode-to-md as parameter.
I built and installed the gem using the instructions in the first post.

Now, when I’m executing ruby script/import_scripts/phpbb3.rb bbcode-to-md I get the following error:

/usr/local/lib/ruby/gems/2.0.0/gems/bundler-1.6.5/lib/bundler/runtime.rb:34:in `block in setup': You have already activated i18n 0.6.11, but your Gemfile requires i18n 0.6.9. Prepending `bundle exec` to your command may solve this. (Gem::LoadError)

I know why it’s happening. When I install the ruby-bbcode-to-md-0.0.13.gem it also installs some dependencies which are the reason for this error. Am I doing something wrong during the installation or how can I fix this?


(Kane York) #18

Have you tried running bundle exec script/import_scripts/phpbb3.rb bbcode-to-md?

Note that I have not done imports.


(Gerhard Schlager) #19

bundle exec script/import_scripts/phpbb3.rb bbcode-to-md fails with:

bundler: not executable: script/import_scripts/phpbb3.rb

I tried using bundle exec ruby script/import_scripts/phpbb3.rb bbcode-to-md too which results in the following error:

/var/www/discourse/script/import_scripts/base.rb:11:in `require': cannot load such file -- ruby-bbcode-to-md (LoadError)
        from /var/www/discourse/script/import_scripts/base.rb:11:in `<top (required)>'
        from script/import_scripts/phpbb3.rb:1:in `require'
        from script/import_scripts/phpbb3.rb:1:in `<main>'

Here are the steps to reproduce this:

  • Bootstrap a new Docker image and log in with ./launcher ssh app
  • Install ruby-bbcode-to-md as root:
    cd /tmp
    git clone GitHub - nlalonde/ruby-bbcode-to-md: Convert BBCode to Markdown.
    cd ruby-bbcode-to-md
    gem build ruby-bbcode-to-md.gemspec
    gem install ruby-bbcode-to-md-0.0.13.gem
  • Install the mysql2 gem as root:
    apt-get update && apt-get install libmysqlclient-dev
    gem install mysql2
  • Change the user and execute the importer:
    su - discourse
    cd /var/www/discourse
    RAILS_ENV=production ruby script/import_scripts/phpbb3.rb bbcode-to-md

It works without the bbcode-to-md, but fails when that parameter is used. Here’s the complete error message with stacktrace:

/usr/local/lib/ruby/gems/2.0.0/gems/bundler-1.7.0/lib/bundler/runtime.rb:34:in `block in setup': You have already activated i18n 0.6.11, but your Gemfile requires i18n 0.6.9. Prepending `bundle exec` to your command may solve this. (Gem::LoadError)
        from /usr/local/lib/ruby/gems/2.0.0/gems/bundler-1.7.0/lib/bundler/runtime.rb:19:in `setup'
        from /usr/local/lib/ruby/gems/2.0.0/gems/bundler-1.7.0/lib/bundler.rb:121:in `setup'
        from /usr/local/lib/ruby/gems/2.0.0/gems/bundler-1.7.0/lib/bundler/setup.rb:7:in `<top (required)>'
        from /usr/local/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:135:in `require'
        from /usr/local/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:135:in `rescue in require'
        from /usr/local/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:144:in `require'
        from /var/www/discourse/config/boot.rb:6:in `<top (required)>'
        from /usr/local/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/local/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /var/www/discourse/config/application.rb:1:in `<top (required)>'
        from /usr/local/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/local/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /var/www/discourse/config/environment.rb:2:in `<top (required)>'
        from /usr/local/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/local/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /var/www/discourse/script/import_scripts/base.rb:19:in `initialize'
        from script/import_scripts/phpbb3.rb:14:in `initialize'
        from script/import_scripts/phpbb3.rb:295:in `new'
        from script/import_scripts/phpbb3.rb:295:in `<main>'

As I said in my previous post I think it’s because of the ruby-bbcode-to-md gem, which installs a bunch of dependencies:

root@docker-app:/tmp/ruby-bbcode-to-md# gem install ruby-bbcode-to-md-0.0.13.gem
Fetching: i18n-0.6.11.gem (100%)
Successfully installed i18n-0.6.11
Fetching: thread_safe-0.3.4.gem (100%)
Successfully installed thread_safe-0.3.4
Fetching: tzinfo-1.2.2.gem (100%)
Successfully installed tzinfo-1.2.2
Fetching: minitest-5.4.0.gem (100%)
Successfully installed minitest-5.4.0
Fetching: activesupport-4.1.4.gem (100%)
Successfully installed activesupport-4.1.4
Successfully installed ruby-bbcode-to-md-0.0.13
6 gems installed

I’m no Ruby expert, so I’m not sure how to solve this. This looks like dependency hell to me.
@neil Could you please take a look at this? Thanks.


Can't start PHPBB3 Importer
(Neil Lalonde) #20

Try this instead:

As discourse user, add to end of /var/www/discourse/Gemfile:

gem 'mysql2'
gem 'ruby-bbcode-to-md', path: '/tmp/ruby-bbcode-to-md'

As root:

cd /var/www/discourse
sudo -E -u discourse bundle install --no-deployment --path vendor/bundle

As discourse user:

cd /var/www/discourse
RAILS_ENV=production bundle exec ruby script/import_scripts/phpbb3.rb bbcode-to-md