Migrate a phpBB3 forum to Discourse

[Discourse]

So, you discovered Discourse and want to know how to migrate from your existing phpBB3 forum?

Great! Then keep on reading. This guide will show you how to use the phpBB3 import script for importing from phpBB 3.0 up to 3.3.

What data can be imported?

• Users
  • Avatars (optional)
  • Anonymous users (either as user “system” or as suspended users)
  • Password hashes, which can be used with the migratepassword plugin (optional)
  • User profile data (birth date, website, location, custom fields, etc.)
• Groups
• Categories and Forums
• Topics and Posts
  • Polls and votes (optional)
  • Smilies
  • BBCodes
  • Internal links to topics and posts
• Sticky topics and (global) announcements
• Private Messages (optional)
• Attachments (optional)
• Bookmarks (optional)
• Permalinks for imported categories, topics and posts (optional)

1. Importing using Docker container

This is the recommended way for importing content from your phpBB3 forum into Discourse.

1.1. Installing Discourse

Install Discourse by following the official installation guide.
Afterwards it’s a good idea to go to the Admin section and configure a few settings:

• Enable login_required if imported topics shouldn’t be visible to the public
• Enable hide_user_profiles_from_public if user profiles shouldn’t be visible to the public.
• Disable download_remote_images_to_local if you don’t want Discourse to download images embedded in posts.
• Enable disable_edit_notifications if you enabled download_remote_images_to_local and don’t want your users to get lots of notifications about posts edited by the system user.
• Change the value of slug_generation_method if most of the topic titles use characters which shouldn’t be mapped to ASCII (e.g. Arabic). See this post for more information.

The following steps assume that you installed Discourse on Ubuntu and that you are connected to the machine via SSH or have direct access to the machine’s terminal.

1.2. Preparing the Docker container

Copy the container configuration file app.yml to import.yml and edit it with your favorite editor.

cd /var/discourse
cp containers/app.yml containers/import.yml
nano containers/import.yml

Add - "templates/import/phpbb3.template.yml" to the list of templates. Afterwards it should look something like this:

templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
  #- "templates/web.ssl.template.yml"
  #- "templates/web.letsencrypt.ssl.template.yml"
  - "templates/import/phpbb3.template.yml"

That’s it. You can save the file, close the editor and build the container.

/var/discourse/launcher stop app
/var/discourse/launcher rebuild import

Building the container creates an import directory within the container’s shared directory. It looks like this:

/var/discourse/shared/standalone/import
├── data
├── mysql
└── settings.yml

1.3. Configuring the importer

You can configure the importer by editing the example settings.yml file that has been copied into the import directory.

nano /var/discourse/shared/standalone/import/settings.yml

The settings file is well documented and comes with sensible defaults, but here are a few tips anyway:

1.3.1. Connecting to a remote database

Change the database host, port, username, password and schema if you want to connect to a remote database. Make sure that your MySQL server allows remote connections.

1.3.2. Connecting to internal database

The Docker container comes with a MariaDB server which is compatible with MySQL. You don’t need to change any database settings if you are going to use it. All you need is a database dump which you can create in multiple ways. Here are two of them:

• Create a database backup in phpBB’s Administration Control Panel
  

  

• Connect to the database host and create a database dump:

  # replace the last parameter "phpbb" with the database schema name
  mysqldump --user root --password --result-file phpbb_mysql.sql phpbb
  

Copy the database dump into the /var/discourse/shared/standalone/import/data/ directory. You need to extract the file if you created a gzip or bzip2 compressed backup and make sure that the file is named phpbb_mysql.sql

1.3.3. Other settings

• Change the table_prefix in case your phpBB forum isn’t using the default naming for database tables.
• Make sure the phpbb_base_dir points to /shared/import/data

1.4. Copying attachments and images

You can skip this step if you don’t want to import attachments, avatars or smilies.

Connect to your phpBB3 host using FTP or SCP and download the following directories into the /var/discourse/shared/standalone/import/data directory:

• Attachments are usually stored in the files directory. You can make sure by checking the path in the Administration Control Panel.
  

  

• Avatars are usually stored in the images/avatars directory. You can make sure by checking the path in the Administration Control Panel.
  

  

• Smilies are stored in the images/smilies directory.

The import directory should look like this if you downloaded all of those directories:

/var/discourse/shared/standalone/import
├── data
│   ├── files
│   ├── images
│   │   ├── avatars
│   │   │   ├── gallery
│   │   │   └── upload
│   │   └── smilies
├── mysql
└── settings.yml

1.5. Executing the import script

Tip: It’s a good idea to start the import inside a tmux or screen session so that you can reconnect to the session in case of SSH connection loss.

Let’s start the import by entering the Docker container and launching the import script inside the Docker container.

/var/discourse/launcher enter import
import_phpbb3.sh # inside the Docker container

Depending on the size of your forum it’s now time for some or
The import script will show you a message like this when it’s finished: Done (00h 26min 52sec)

And while you are waiting for the import to finish you can take a look at the FAQ.

Tip: You can abort the import anytime you want by pressing Ctrl+C
When you restart the import it will continue where it left off.

You can exit and stop the Docker container after the import has finished.

exit # inside the Docker container
/var/discourse/launcher stop import

1.6. Starting Discourse

Let’s start the app container and take a look at the imported data.

/var/discourse/launcher start app

Discourse will start and Sidekiq will begin post-processing all the imported posts. This can take a considerate amount of time. You can watch the progress by logging in as admin and visiting http://discourse.example.com/sidekiq

1.7. Clean up

So, you are satisfied with the result of the import and want to free some disk space? The following commands will delete the Docker container used for importing as well as all the files used during the import.

/var/discourse/launcher destroy import
rm /var/discourse/containers/import.yml
rm -R /var/discourse/shared/standalone/import

1.8. The End

Now it’s time to celebrate and enjoy your new Discourse instance!

---

2. Importing using development environment

The following instructions are for users who want to import using a development environment.

1. Setup your development environment by following the guides for Ubuntu or Mac OS X.
   The following instructions assume that you are using Ubuntu.

2. Make sure that neither Discourse nor Sidekiq are running.

3. Install some dependencies:

   sudo apt-get update
   sudo apt-get install libmysqlclient-dev
   cd ~/discourse
   IMPORT=1 bundle install
   git checkout Gemfile Gemfile.lock
   

4. Configure your import. There’s an example settings file at ~/discourse/script/import_scripts/phpbb3/settings.yml

5. Start your import (change the path to your settings file if you put your custom settings somewhere else):

   cd ~/discourse/script/import_scripts
   IMPORT=1 bundle exec ruby phpbb3.rb phpbb3/settings.yml
   

6. Wait until the import is done. You can restart it if it slows down to a crawl.

7. Start your Discourse instance:

    cd ~/discourse
    bundle exec rails server
    bin/ember-cli
   

8. Start Sidekiq and let it do its work: bundle exec sidekiq
   Depending on your forum size this can take a long time. You can monitor the progress at http://localhost:3000/sidekiq

---

3. Roadmap

Here’s a list of things that are still missing from the importer (in no particular order) :

• Add support for the cakeday plugin
• Imported text that looks like Markdown should be escaped
• Close topics that are closed in phpBB3
• Import unapproved posts as hidden posts
• Import read status for each post and private message
• Improve the BBCode to Markdown converter (ruby-bbcode-to-md)
• Support custom patterns for internal links to topics and posts (SEO optimized URLs)
• Add support for more database sources: MS SQL Server, Oracle, PostgreSQL

Feel free to start your favorite Ruby IDE and help making the importer even better.

---

4. FAQ

4.1. I have a heavily modified forum. Will the import script still work?

Maybe. It depends on what changes those mods made to the database. You’ll have to give it a try.

4.2. Why is the import process so slow?

Importing is CPU-bound. You should use CPUs with fast single-core speed if you are in a hurry. You get an import speed of about 400–600 posts/minute on a typical DigitalOcean server. Bare metal servers, and probably also your desktop computer, are usually about twice as fast.

4.3. I have a large forum with lots of users and posts. How can I minimize the downtime during the import?

You can do incremental imports in order to shorten the downtime.
But be warned: There’s a small risk of data loss when you do an incremental import, because existing posts and users aren’t updated during incremental imports. You’ll lose post edits that occurred between imports as well as changes made to user profiles.

4.4. Can I run the Docker based import on my desktop computer or a staging server?

Of course. You can move your Discourse instance to a different server after the import has finished. Take a look at this How-To for instructions.

4.5. Does the script send any emails during the import?

Sending of emails is disabled during the import and Discourse holds off on sending summary emails to imported users for some time. This can be configured with the site setting default_email_digest_frequency which defaults to 7 days.

4.6. Which version of Discourse should I use for importing?

It is recommended to always use the tests-passed or beta channel of Discourse for imports since the import script is being constantly improved. So, make sure that the value of version is set accordingly in the app.yml and import.yml configuration files. Everything should be fine if you followed the official installation guide.

4.7. I enabled the import of passwords. Why can’t my users login with their old passwords?

Make sure that you enabled the migratepassword plugin in app.yml by following the How-To for installing plugins. Also, users won’t be able to login with their old password if it’s considered insecure. By default Discourse requires a minimum password length of 10 chars (for admins it’s 15) and the password must not be on the list of common passwords. But users can always reset their password by clicking the “I forgot my password” link on the login dialog or directly by visiting https://discourse.example.com/password-reset

4.8. The import was successful, but the user list is empty. What is wrong?

The list of users won’t show up right away. It is generated by a background task which is executed by Sidekiq. You’ll probably have to wait until all the post-processing of the imported data is finished.

4.9. I have custom smilies in my forum. Are they imported?

Yes. You can map them to Emojis in the settings file, otherwise they’ll get imported as images. All the default smilies (except and ) already have an Emoji mapping.

---

Last Reviewed by @cocococosti on 2022-07-22T04:00:00Z

Last edited by @JammyDodger 2024-05-27T14:58:09Z

Check document

Perform check on document:

[Canapin (Coin-coin le Canapin)]

Greetings,
I’ve been stuck on the nested quote issue for a long time. As discourse handles the [quote] syntax the same way it uses >, I tried just to add a new line before and after each [quote] tags.
I deactivated the bbcode_to_md function to be sure that the importer doesn’t touch the [quote] tags.
I added this line at the end of process_lists in text_processor.rb:

 text.gsub!(/(\[\/?quote.*?\])/mi, "\n\1\n")

But it simply removed all the quotes from the import.
What am I doing wrong?

[gerhard (Gerhard Schlager)]

Try this one instead:

text.gsub!(/(\[\/?quote.*?\])/mi) { |q| "\n#{q}\n" }

[Canapin (Coin-coin le Canapin)]

Thank you for your reply. For what I remember, I previously tried a regex with single quotes but the \n didn’t work as it created plain text \n instead of actually creating new lines…

I don’t know ruby and I tried some gsub on onlines ruby compilers. I achieved to successfully adding new lines before and after quote after gathering some useful info here: ruby - How do you replace with a newline using gsub? - Stack Overflow

They say you have to use double quotes " in order to create actual new lines, and double backslashes \\ to write a regex group content. Do I need to do the same in text_processor.rb ?

[gerhard (Gerhard Schlager)]

Ah, yeah, that gets ugly really fast with all the escaping. I’ve updated my answer above.

[Canapin (Coin-coin le Canapin)]

Your suggestion did the trick. Thousand thanks to you.

[allu (Allen)]

Btw, here is an issue I stumbled across while working on a migration from phpbb 3.0.11 to Discourse. It appears that the phpbb importer reads the user from the post_username instead of user_id in the posts table. In my case, this data does not match, causing many posts to be imported as “Guest”:

[gerhard (Gerhard Schlager)]

The imported passwords are probably too short.

@gerhard

4.7. I enabled the import of passwords. Why can’t my users login with their old passwords?
Make sure that you enabled the migratepassword plugin in app.yml by following the How-To for installing plugins. Also, users won’t be able to login with their old password if it’s considered insecure. By default Discourse requires a minimum password length of 10 chars (for admins it’s 15) and the password must not be on the list of common passwords. But users can always reset their password by clicking the “I forgot my password” link on the login dialog or directly by visiting https://discourse.example.com/password-reset

---

Did you migrate to phpBB3 from a different forum software some time in the past? post_username shouldn’t have a value unless it was posted by an anonymous user. This problem has come up a few times, so I’d like to add a workaround to the import script. Could you help me by looking up a few things in the database?

• What’s the value of phpbb_posts.poster_id for the post you’ve shown in the screenshot?
• What’s the value of phpbb_users.user_type for that user (phpbb_users.user_id = phpbb_posts.poster_id)?

If you want to fix the problem right now, take a look at the following post. The solution will probably work for you as well.
https://meta.discourse.org/t/importing-from-phpbb3/30810/224

[allu (Allen)]

Another requirement I have is to import my user custom fields from phpBB to Discourse. I took a look at the discourse db and imo the most straightforward (but not so clean) solution would be to create all required custom fields in the Discourse admin backend and then import my data via postgres copy. Beforehand, I would prepare my import file so I have the correct Discourse user id in the user_id field. Basically, I would import exactly this structure:

Would that work or are other things going on behind the scenes which would become inconsistent?

[gerhard (Gerhard Schlager)]

That will most likely work, but you’ll have to try it out to know for sure.
But, you could also add a few lines of code to the import script and improve it for everyone by sending a pull request

[sukria (Alexis Sukrieh)]

Hi there, first of all thanks a lot for this thread, and for Discourse itself, it’s a great tool!

I’m running a phpBB forum (version 3.2.0) and I’m looking into migrating it to Discourse. The initial post of this thread states that “Importing from phpBB 3.2 is not supported”.

I wanted to know if there is any plan regarding support for this phpBB version? Or if there’s any hint/advice I could use to migrate my content to Discourse.

Thanks a lot for your comments.

[gerhard (Gerhard Schlager)]

phpBB 3.2 uses a XML based format for storing posts which is completely different from previous versions. (And if you upgraded from 3.1 or below, you’ll probably see a mixture of old and new storage format, until a background process converted all posts to the new format.)

I’m not aware of an easy way to get the import script working again for version 3.2. You’d need to write a conversion from XML to Markdown.

I’d really love to add support for the latest phpBB version, but it could take a while until I find some spare time.
That is, unless a customer pays us for it or you or someone else implements it and sends a pull request

[sukria (Alexis Sukrieh)]

Thanks for the reply, I see!

I’ll look into what I can do and let you know if I have valuable bits to share.

Have a nice day

[Jean-Michel25 (Jean-Michel)]

Hello
Yes it fix definitively our issue !

[marianord (Mariano Rodriguez)]

Hello, I tried the import step by step and I have this error when I run the importer.

root@clubcorsavenezuela-import:/var/www/discourse# import_phpbb3.sh
Loading database dump into MySQL...
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

The dump of the database is in /var/discourse/shared/standalone/import/data

This is my settings file:

# This is an example settings file for the phpBB3 importer.

database:
  type: MySQL # currently only MySQL is supported
  host: localhost
  port: 3306
  username: root
  password:
  schema: phpbb
  table_prefix: phpbb_ # Change this, if your forum is using a different prefix. Usually all table names start with phpbb_
  batch_size: 1000 # Don't change this unless you know what you're doing. The default (1000) should work just fine.

import:
  # WARNING: Do not activate this option unless you know what you are doing.
  # It will probably break the BBCode to Markdown conversion and slows down your import.
  use_bbcode_to_md: false

  # This is the path to the root directory of your current phpBB installation (or a copy of it).
  # The importer expects to find the /files and /images directories within the base directory.
  # You need to change this to something like /var/www/phpbb if you are not using the Docker based importer.
  # This is only needed if you want to import avatars, attachments or custom smilies.
  phpbb_base_dir: /shared/import/data

  site_prefix:
    # this is needed for rewriting internal links in posts
    original: bwl.com.ve    # without http(s)://
    new: https://clubcorsavenezuela.com       # with http:// or https://

  # Enable this, if you want to redirect old forum links to the the new locations.
  permalinks:
    categories: true  # redirects   /viewforum.php?f=1            to  /c/category-name
    topics: true      # redirects   /viewtopic.php?f=6&t=43       to  /t/topic-name/81
    posts: false      # redirects   /viewtopic.php?p=2455#p2455   to  /t/topic-name/81/4

  avatars:
    uploaded: true  # import uploaded avatars
    gallery: false   # import the predefined avatars phpBB offers
    remote: false   # WARNING: This can considerably slow down your import. It will try to download remote avatars.

  # When true: Anonymous users are imported as suspended users. They can't login and have no email address.
  # When false: The system user will be used for all anonymous users.
  anonymous_users: true

  # Enable this, if you want import password hashes in order to use the "migratepassword" plugin.
  # This will allow users to login with their current password.
  # The plugin is available at: https://github.com/communiteq/discourse-migratepassword
  passwords: false

  # By default all the following things get imported. You can disable them by setting them to false.
  bookmarks: true
  attachments: true
  private_messages: false
  polls: true

  # When true: each imported user will have the original username from phpBB as its name
  # When false: the name of each imported user will be blank unless the username was changed during import
  username_as_name: false

  # Map Emojis to smilies used in phpBB. Most of the default smilies already have a mapping, but you can override
  # the mappings here, if you don't like some of them.
  # The mapping syntax is: emoji_name: 'smiley_in_phpbb'
  # Or map multiple smilies to one Emoji: emoji_name: ['smiley1', 'smiley2']
  emojis:
    # here are two example mappings...
    smiley: [':D', ':-D', ':grin:']
    heart: ':love:'
~
~
~
~

[notriddle (Michael Howell)]

Can you try connecting to mysql using the mysql -uroot command?

[marianord (Mariano Rodriguez)]

I solved it, I had some issues with the mysql credentials. Thanks

[MJ94]

Hello,

Do you have any information on how we may migrate an existing phpBB 3.2.0 forum to Discourse? If not, do you know if/when an option may be available?

Thank you,
Michael

[pfaffman (Jay Pfaffman)]

@MJ94

If not, do you know if/when an option may be available?

Not until someone decides to do it for themselves, hires someone to do it, or a discourse.org customer wants it.

My Discourse Migration page describes imports in general. If you have a budget and want an estimate, I’d need to see a significant sample of what raw posts look like now to determine what would be involved to convert them from whatever they look like into something that Discourse can handle.

[funtent (M)]

Add me to the 3.2 -> Discourse waiting list pile.