Migrate a phpBB3 forum to Discourse

pfaffman · December 22, 2021, 2:35am

That’s all pretty typical.

I think that the changes you suggest likely won’t break any other imports. Often there are a set of changes like these and then a bunch of other changes that are specific to the import and figuring which are which and testing again with only those would be a bunch of work so a PR doesn’t get created.

Glad you got it done!

Canapin · December 22, 2021, 12:57pm

Thank you Constanza, very helpful link.

Thanks for the information Jay. I’ll do PR later after I’m done with the migration if it can help to have an even better phpBB3 migration tool.

The current script also ignores [size=XXX] tags which I encountered during other migrations.
On my current migration, I did a quick script in my importer and used the value of XXX to replace these tags with content and <big>content</big> since they are supported by Discourse.
But that was a personal wish and it might be a more proper solution (generally speaking) to simply get rid of these [size] tags, like the import script already does with the [color] tags.

gerhard · December 22, 2021, 5:48pm

Thanks @Canapin for all your feedback. A PR is definitely welcome!

Also, a small teaser: We are working on a solution which should make all imports – not just phpBB – a lot better, faster, easier to customize and get rid of those pesky problems with the BBCode conversion…

Canapin · January 1, 2022, 2:01pm

While importing my posts, I got a bunch of these errors (not for all posts):

   251491 / 251672 ( 99.9%)  [14140 items/min]  Exception while creating post 354629. Skipping.
undefined method `[]' for nil:NilClass
/var/www/discourse/script/import_scripts/phpbb3/importers/post_importer.rb:66:in `block in map_first_post'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/core_ext/object/try.rb:15:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/core_ext/object/try.rb:15:in `try'
/var/www/discourse/script/import_scripts/base.rb:576:in `create_post'
/var/www/discourse/script/import_scripts/base.rb:523:in `block in create_posts'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rack-mini-profiler-2.3.3/lib/patches/db/mysql2/alias_method.rb:8:in `each'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rack-mini-profiler-2.3.3/lib/patches/db/mysql2/alias_method.rb:8:in `each'
/var/www/discourse/script/import_scripts/base.rb:510:in `create_posts'
/var/www/discourse/script/import_scripts/phpbb3/importer.rb:192:in `block in import_posts'
/var/www/discourse/script/import_scripts/base.rb:874:in `block in batches'
/var/www/discourse/script/import_scripts/base.rb:873:in `loop'
/var/www/discourse/script/import_scripts/base.rb:873:in `batches'
/var/www/discourse/script/import_scripts/phpbb3/importer.rb:254:in `batches'
/var/www/discourse/script/import_scripts/phpbb3/importer.rb:188:in `import_posts'
/var/www/discourse/script/import_scripts/phpbb3/importer.rb:38:in `execute'
/var/www/discourse/script/import_scripts/base.rb:47:in `perform'
/var/www/discourse/script/import_scripts/phpbb3/importer.rb:22:in `perform'
script/import_scripts/phpbb3.rb:33:in `<module:PhpBB3>'
script/import_scripts/phpbb3.rb:14:in `<module:ImportScripts>'
script/import_scripts/phpbb3.rb:13:in `<main>'

But I couldn’t figure out what causes this.

Any clue?

How can I check if I have actually post missing? Couldn’t find how to output the data (post content for example) corresponding to “Exception while creating post 354629”

I can’t recall having this error in my other import tests the previous days, but maybe I just didn’t pay attention… Could it be related to this known issue?

251490 / 251672 ( 99.9%)  [14140 items/min]  Parent post 337703 doesn't exist. Skipping 354628: vw-camper est malade !?

I ran the import script twice as it should, regarding this particular issue.

gerhard · January 3, 2022, 2:33pm

Make sure that you set tag_mappings: {} in your settings.yml file if you aren’t using this feature.

Canapin · January 3, 2022, 10:41pm

I feel dumb! I even thought doing about that, but I was so much sure that I didn’t encounter this error before that I didn’t try…

It resolved the issue, thank you very much.

I supposed you can’t say much at this time, but any clue about how it will work, and when some useable import scripts will be useable using this new thing?
I might migrate another phpBB this year, so that could be very interesting for me to know a bit more.

Sebastien_Guilbaud · January 13, 2022, 7:15am

Hi, i’m testing a phpbb 3.2 migration to discourse for a decently sized forum (30k topics / 600k posts) and almost everything works fine, apart for a couple of topics not imported (“Parent post xx doesn’t exist”), not solved with multiple runs, but it’s not an issue.
My main problem is that subsequent imports with fresh data (verified with the sha256sum in import/mysql/imported file) do not import fresh posts in Discourse. I’m facing an exception early in the process, dunno if it’s linked :

Failed to map post with ID 6815

BIGINT UNSIGNED value is out of range in '(`phpbb_prod`.`o`.`poll_option_total` - (select count(distinct `phpbb_prod`.`v`.`vote_user_id`) from `phpbb_prod`.`phpbb3_poll_votes` `v` join `phpbb_prod`.`phpbb3_users` `u` join `phpbb_prod`.`phpbb3_topics` `t` where ((`phpbb_prod`.`u`.`user_id` = `phpbb_prod`.`v`.`vote_user_id`) and (`phpbb_prod`.`v`.`topic_id` = `phpbb_prod`.`t`.`topic_id`) and (`phpbb_prod`.`v`.`poll_option_id` = `phpbb_prod`.`o`.`poll_option_id`) and (`phpbb_prod`.`t`.`topic_id` = `phpbb_prod

Is this an issue with embedded polls in posts, from the SQL displayed ?

have a nice day !

gerhard · January 13, 2022, 7:39pm

I can’t say much yet as it’s still under development. Plans can change… But “this year” is a good bet.

gerhard · January 13, 2022, 7:40pm

BIGINT UNSIGNED value is out of range

Hmm, are there that many anonymous votes in this poll?

You will need to fiddle with this SQL:

github.com

discourse/discourse/blob/f82f0e11798fea1130c91a3958a26dd52453e3f7/script/import_scripts/phpbb3/database/database_3_0.rb#L115-L129


      
          def fetch_poll_options(topic_id)
            query(<<-SQL)
              SELECT o.poll_option_id, o.poll_option_text, o.poll_option_total AS total_votes,
                o.poll_option_total - (
                  SELECT COUNT(DISTINCT v.vote_user_id)
                    FROM #{@table_prefix}poll_votes v
                      JOIN #{@table_prefix}users u ON (v.vote_user_id = u.user_id)
                      JOIN #{@table_prefix}topics t ON (v.topic_id = t.topic_id)
                    WHERE v.poll_option_id = o.poll_option_id AND v.topic_id = o.topic_id
                ) AS anonymous_votes
              FROM #{@table_prefix}poll_options o
              WHERE o.topic_id = #{topic_id}
              ORDER BY o.poll_option_id
            SQL
          end

Something like this maybe? mysql - BIGINT UNSIGNED VALUE IS out of range My SQL - Stack Overflow
Please let me know when you’ve found a solution or create a PR with a fix. I’d appreciate it.

Sebastien_Guilbaud · January 14, 2022, 9:03am

Polls are from the very beginning of the forum, I wasn’t in charge at this time, but I think anonymous users were never allowed to post/answer polls

With

 SET sql_mode = 'NO_UNSIGNED_SUBTRACTION';

I’m able to query all polls votes, and for 2 of them, I have negative “anonymous_votes”. It seems this forum somehow allowed anonymous voting, actually

Thanks @gerhard for the tip, I’m going to dig deeper…

ePirat · January 17, 2022, 8:04pm

I am currently looking into migrating a phpBB forum to Discourse, the problem is that our phpBB is using Postgres instead of MySQL, so I would like to adapt the script to work with that too.

Could anyone give me some pointers about the best way to go about this? As ideally I would like for everyone to benefit from this work, so would like it to use an upstream-accepted approach instead of just quick and dirty hacking it to work.

From what I can tell it seems to be mostly a matter for adding the proper SQL statements for the PostgreSQL schema using the right DB adapter, however I am not sure how the whole orchestration for the Docker container needs to be adapted to properly spin up a PostgreSQL instance, if needed by the DB type specified in the config, to import the DB dump into and execute the statements against.

gerhard · January 17, 2022, 8:16pm

There’s a bulk import script for phpBB running on postgresql. Did you see it? discourse/script/bulk_import/phpbb_postgresql.rb at main · discourse/discourse · GitHub

However, I don’t think it has been used recently, so it’s probably broken and might not work with current phpBB versions because the Markdown storage format changed.

Whatever path you choose, I wouldn’t spend too much time on gold plating your solution unless you really want to. The current import scripts will be deprecated sometime this year…

ePirat · January 17, 2022, 8:20pm

Thanks, I wasn’t aware of that! And thanks for the heads-up that the scripts are going to be deprecated, so I guess it might not be worth upstreaming such a major feature for those if they will anyway be deprecated soon? Is there a replacement planned for them?

gerhard · January 17, 2022, 8:23pm

Yes, of course there will be a replacement.

ePirat · January 17, 2022, 8:30pm

Another important thing I was wondering, do I need to set anything other than disable_edit_notifications to ensure none of the imported users are emailed by Discourse for anything that I import? As currently the instance is not public and we are just experimenting with the import and probably needing severals tries until it is good enough, I want to avoid any emails being sent.

gerhard · January 17, 2022, 8:32pm

You are looking for the disable_emails site setting…

Canapin · January 27, 2022, 3:56pm

While [b]text[/b] is supported inline on Discourse, it’s not interpreted for block of text that have new lines.

[b]For example, if I have this text in my post…

And I continue my line after an empty line, I close the bbcode and the text won’t be converted to bold text[/b]

Same for [i] and probably other tags.

I fail to find a clean solution for this in my import. Any idea?

Steven · January 27, 2022, 4:11pm

This is not an ideal solution, but I have an idea for a plan B or C at least.

After the import, you could remap [b] to  and [/b] to  since  works on block here

test

<b>test

test</b>

But on a large forum, it will take a while

Canapin · January 27, 2022, 4:25pm

Good idea!

My forum isn’t imported yet, so I can still work on the importer.

Wouldn’t use  and  have more sense, notwithstanding their semantic “property”?

pfaffman · January 27, 2022, 4:44pm

That’s great news!

I think that’s a good thing to try first.

Topic		Replies	Views
phpBB 3 Importer (old) Feature	90	16886	October 8, 2014
My phpBB Migration Journey (postgresql) Migration	23	1553	March 30, 2023
Migrate an SMF2 forum to Discourse Sysadmins how-to	95	13202	April 7, 2024
Use an import script that requires MySQL Sysadmins how-to	51	12457	December 12, 2023
Migrate a vBulletin 4 forum to Discourse Sysadmins how-to	160	28877	July 27, 2024

Migrate a phpBB3 forum to Discourse

Related topics