Migrate a phpBB3 forum to Discourse

Is it possible to import phpBB forums into an existing Discourse forum, ie to merge them? I assume it is based on the settings about mapping categories.

Any issues to be aware of?

Are users merged if they have the same email address?

yes. if you don’t do your own cat mapping in the settings file, new cats will be created as needed during the import. if there is an existing cat in discourse already matching a cat name from phpbb, the cats will be merged.

oh, yes. here is a thread detailing what i’ve encountered. i have some tips and also a repo with my combined updates to the importer.

i can’t remember the exact conditions, but yes, users get merged automatically during the import.

I’m curious on if it is possible to merge a phpbb3 forum into Discourse while keeping all of the imported topics read-only. Searched through the thread, but did not see this mentioned. Thanks.

You could modify the import script to archive all topics created, or it might be easier to just do it in bulk after the script ran. See Administrative Bulk Operations for examples.

Hello,

I seem to be running into an issue when starting the importer. Seems to happen during user import or shortly after. I have included the stack trace and a quick 20 second video of the output

Please advise

0x00007fe87a88bb10 /var/www/discourse/script/import_scripts/phpbb3/importers/user_importer.rb:87>}
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/validations.rb:80:in `raise_validation_error': Validation failed: Name can't be blank (ActiveRecord::RecordInvalid)
        from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/validations.rb:53:in `save!'
        from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:302:in `block in save!'
        from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
        from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/connection_adapters/abstract/database_statements.rb:314:in `transaction'
        from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
        from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:302:in `save!'
        from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/suppressor.rb:54:in `save!'
        from /var/www/discourse/script/import_scripts/base.rb:361:in `block in create_user'
        from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
        from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
        from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
        from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
        from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
        from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
        from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
        from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/transactions.rb:209:in `transaction'
        from /var/www/discourse/script/import_scripts/base.rb:360:in `create_user'
        from /var/www/discourse/script/import_scripts/base.rb:278:in `block in create_users'
        from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.1.1/lib/patches/db/mysql2/alias_method.rb:8:in `each'
        from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.1.1/lib/patches/db/mysql2/alias_method.rb:8:in `each'
        from /var/www/discourse/script/import_scripts/base.rb:266:in `create_users'
        from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:106:in `block in import_anonymous_users'
        from /var/www/discourse/script/import_scripts/base.rb:948:in `block in batches'
        from /var/www/discourse/script/import_scripts/base.rb:947:in `loop'
        from /var/www/discourse/script/import_scripts/base.rb:947:in `batches'
        from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:293:in `batches'
        from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:102:in `import_anonymous_users'
        from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:33:in `execute'
        from /var/www/discourse/script/import_scripts/base.rb:47:in `perform'
        from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:22:in `perform'
        from script/import_scripts/phpbb3.rb:35:in `<module:PhpBB3>'
        from script/import_scripts/phpbb3.rb:16:in `<module:ImportScripts>'
        from script/import_scripts/phpbb3.rb:15:in `<main>'

EDIT:

Looks like it has something to do with anonymous users, I think I’m going to look into my database

 def map_anonymous_user(row)
      username = row[:post_username]

      {
        id: @settings.prefix(username),
        email: "anonymous_#{SecureRandom.hex}@no-email.invalid",
        username: username,
        name: @settings.username_as_name ? username : "",
        created_at: Time.zone.at(row[:first_post_time]),
        active: true,
        trust_level: TrustLevel[0],
        approved: true,
        approved_by_id: Discourse.system_user.id,
        approved_at: Time.now,
        post_create_action:
          proc do |user|
            row[:user_inactive_reason] = Constants::INACTIVE_MANUAL
            row[:ban_reason] = "Anonymous user from phpBB3" # TODO i18n
            suspend_user(user, row, true)
          end,
      }
    end

EDIT

I managed to get it to get past the error by changing this line in the map_anonymous_user method. Turns out the default was to set the username to an empty string, so I just put a word in it.

changed this

name: @settings.username_as_name ? username : "",

to this

name: @settings.username_as_name ? username : "Anonymous ",

Can I delete old posts before running an incremental import? I successfully ran the import locally in a dev install, and restored it to my server running a docker install, and my plan is to run an import there for new posts. But even ignoring the posts that have already been imported, running the import on the server is taking hours. My forum is pretty big (1.3m posts) and my server’s disk is slow, so just loading the .sql file takes hours.

Will it cause problems if I do something like DELETE FROM phpbb_posts WHERE post_id < 2926807 locally and then dump and upload that to the server and run the import as normal?

I added an IMPORT_AFTER setting to some scripts to have it pulll in only data after some date. So I’d modify the SQL that pulls in the posts to just get ones after a day before the last data dump.

But if you have a really slow disk, you likely are going to have trouble hosting Discourse. Is performance acceptable with the data you have?

Yes, for normal use I’ll be using a faster disk, but it isn’t big enough to host multiple copies of the database at the same time, so I’m routing /shared/import to a larger, slower disk in the import.yml volumes section. The trick worked fine for restoring the initial backup, I rerouted /shared/tmp to the slow disk after restore failed due to running out of space even though I had 18G free and a 6G database (I think I saw you mention elsewhere that restore creates 3 copies of the database while it’s running). But at that stage downtime wasn’t a concern.

So day-to-day, size and speed aren’t an issue, my forum is pretty quiet it’s just old.

I’ll take a look at modifying the SQL in the scripts. Thanks!

In the settings.yml the content for connecting to mysql is this:

Where can i put the database name?

The database name should go in the schema param. The name of that param is a bit confusing, yes. You can see here in the code, it is passed as the database name.

Hi! I have made a brief video talking about the issue we are having with the import. We have about 650,000 posts we are bringing over.

Thank you thank you!

https://www.loom.com/share/1f66315779af4cf7b286c8541d4f3f09?sid=11a46d3c-8510-43a1-82e9-1a3524cbb365

Hi. Is there a way to execute the import row by row. We are having problems with the script completing. Some things work, like users and avatars, others don’t. I have reimaged my server about 4 times because I read somewhere that after an import errors out, a clean install needs to be done.

Other details are in the video I posted. Any assistance is appreciated.

Hi. I’m about 18 hours into migrating and am no closer to getting the migration to work. Could you please see my video and advise if you have any tips or tricks or if there is something missing s configured. We just need posts, topics, private messages, categories. Avatars and users are working. Server is clean UBUNTU 22.04.

An easier solution is to make a backup after you create the site so that you can restore that one. Another way to get a clean database is to do a rake db:drop db:create db:migrate, but you have to do a few things to make that possible. If you can’t guess those things by the error messages, then the backup/restore solution is the way to go, and may be faster anyway.

You can add some debugging puts statements in there, though the structure of the phpbb3 script is a bit confusing for people who don’t know Ruby well.

Also, you can run the script multiple times and it’ll skip over the things it’s done already, so you don’t really need to wipe the database until you have changed something that requires a fresh start (i.e., it’ll do something different with the data imported already).

That’s beyond what I can do for free. If you have a budget you can contact me directly or post in marketplace.

The import script doesn’t appear to be working. The issue is in the video. If someone from discourse migrations can look into it, it would be appreciated.

If support is fee based, could you guys list the cost of migrating? My understanding is that this is where we go to ask questions on how to get the script working. I’m ok with fees, I just thought this was an open source project and this was the place we go to ask questions.

I’m saying that I’m a volunteer here, and I’m not willing to watch a video for free, largely because of my irrational disdain for video. But I’m not the only one here. Lots of people love watching videos.

So every time it runs it creates duplicates of the users it created on the previous run? I can’t imagine how that could be the case. Are you sure? That’s not how it’s supposed to work.

It will appear to import the stuff again, but it goes much faster since it’s not really doing anything since things that have been imported already have an import_id custom field set that keeps it from being imported again. If it’s duplicating data on each run then something very strange is wrong.

It is. Sometimes people need or want more help than you can get for free.

Got it. Could you please posts your fees for migrating. I understand the script doesn’t seem to work and it requires paid support. I’m totally ok with that.

Migrations are quite complex. Every one of them has different challenges, as every forum’s data is unique. Debugging a migration issue can take hours, and requires an in depth look into the logs, the DB, the code, etc. Difficult to do by just looking at these posts, sorry I can’t be of more help! Here’s our migrations FAQ: Migrating to Discourse | Discourse - Civilized Discussion

Thank you for the link. I think I figured out the issue and the import script is now running. I’m at about 100000/666357 posts right now. Chat GPT has been helpful in identifying some of the permission-based errors and such.