Importing / migrating from phpBB3

I have rebuilt import, rebuilt app, and still connection refused. :frowning:

You need to add the database dump and files from phpBB again. You probably deleted them during the reset.

2 Likes

Yes, i have deleted those. But wont i need to restore my stuff first? I had set up a lot of things before the import so i wouldnt have to do it again when i would have to redo an import. Wouldnt it now just import into a vanilla discourse instance?

And then you’ll restore your backup that should be in /var/discourse/shared/backups/default

aaand here I am again, sorry but I now have another issue…

this time it is: smilies! I have put each smiley in the settings.yml like this:

happy: [’:D’,’:-D’]
woo: ‘:woo:’
etc.

I have put the smilies images in the /var/discourse/shared/standalone/import/images/smilies dir.
when importing I didn’t see any errors regarding smilies not found or something like that.

still, smilies haven’t been mapped to emoji, and in the posts they have been converted into images.

what have I done wrong?

thank you once again for your help and insights!

edit: :exploding_head: ofcourse… I have to map them from phpbb3 not phpbb2…
i think that solves it but I still have to test if this was the problem though.

edit2: I have done a new import now with a new phpbb3 database dump that had the smilies inserted. Still no smilies. they have been converted to images and they are not in the emoij set. What can be the issue?

I have finally managed to get the smilies mapped to emoji.
Since it was a lot of stumbling for me with having over 150 custom smilies which all had different image-names and different smiley codes, here is a quick extended how-to for others like me.

understanding what the importer does with smilies
What I thought would happen, is that when you add the smiliy codes in the importer and put the images in the designated image file, they will automatically also be added into the emoji folder. It doesn’t do that. So, you will need to manually import your smiley images into the emoji.
When importing them, they need to have the name of the smiley code you will actually be using. So for example if you had a smiley which image filename is “cheery_icon0.gif” that is displayed when users typed :cheer:, you will have to rename that image into cheer.gif and upload it to the emoji’s. (in admin cp > customize > emojis)

Now, it gets even more interesting when you have a bunch of smilies that in phpbb displayed with things like

<;-)
%-)
:3 

So for instance I would have a smiley

 code : <:-) and was named "_1partyguyhat.gif"

I first had to decide what the new code would be in discourse, since one can’t name files with ‘<:-)’ . Then rename the gif into that code, and then add the corresponding mapping into the settingsfile.

so for me for that particular smiley that was:

party_hat: '<:-)'

what then happens when importing, all instances in a post when someone has typed <:-) are translated into :party_hat:.
It will then use the party-hat emoji to render the smiley when that is available in the emoji’s.

tips for when you have 120 smilies to convert :wink:

  • create a test post in your phpbb3 instance with ALL the smilies in it
  • when importing, you can quickly scan that post and see whether any text is left, or with the [edit] option, see whether any have been translated into images instead of just the emoji-code. (Believe me that can happen when you forget a : or mistake a ; for a : )

troubleshooting settings.yml

  • you need to uncomment the line emojis . (I had totally looked over that)
  • format should be 4 spaces in front of a smiley code
  • you need to add all codes, even when you already had a smiley coded :cheer: and you put in a cheer.gif in the emojis. If you don’t, it will still be translated into an imagefile instead of an emoji-code.
  • if you happen to have a smiley which is coded :yes: or :no:, you need to comment those because otherwise it will parse into boolean values :true: or :false:. In that case you would need to do something like this: "yes": ':yes:' to code that particular emoji.
  • edit: oh and another fun fact I forgot: if you use any - in emojis like ‘party-hat’ , when you upload the image to discourse it will convert that into an underscore so it will be named ‘party_hat’. So don’t use ‘-’ only ‘_’.

I hope this helped someone, I know I have spent almost 2 weeks on this before I finally had them all imported right.

8 Likes

I seem to be the only one posting here…

So, right when I am about to launch my new discourse forum, we discovered a very strange issue! I’m stumped with this issue, I’m hoping someone here can shed light on it…

Case: I have imported 2 databases into discourse from phpbb3. First one is an archive, which is basically the current forum we had cut in two 15 years ago and then frozen. The other half of the forum stayed active and grew. My hope was that the importer would merge them on import, which it did! :smiley:

But now I discovered a case of one user, and I need to know whether this is only one case, or if it has impacted others, and then how to be able to search for other cases.

The user had created two accounts. Both on the current forum. Let’s say old account was named OldName and new is NewName.

On the imported discourse, there are now two users. One called OldName and the one that should have been NewName is renamed to OldName1 Also, the posts of OldName have been merged into OldName1.

I have no clue why this only happened to this user since there are other users with double accounts and they don’t seem to be affected by this.

Does anyone know what might have happened here?
Is there a way to search for other cases?

1 Like

But probably not how you think. If you merely ran the importer twice, all of the import_id custom fields conflicted, so the user with userid 123 on the first database is now the same user as userid 123 in the new database. And the same goes for topics and posts.

I’m pretty sure that every userid (and topic and post, and probably category) that exists in both the old and new databases is merged with the old one.

You’ll need to do something to see that the import_ids don’t conflict (maybe delete the old ones, maybe change the import script to make them unique for each import).

4 Likes

well… they ARE the same anyhow??
i mean it’s basically the same forum, only one part went on, and the other part stayed frozen in time. So user ID 123 is still user ID 123, and post and topic id’s as well?

what do you mean by import_id custom fields?

the forum looks ok, all topic, posts etc in place. that they are merged is the general idea?

I can’t find any other issues with other users/posts/topics except that one use case?

to explain perhaps better what is the case.

start was one phpbb forum. At some point in time I cloned it. one clone remained closed and became the archive. The other part, I deleted the oldest posts (that were now in the archive) and kept open. In that one, the users, topics, posts etc just increased and never were changed (ID-wise). So what I thought is that the import script will just see the archive and current forum as two parts of the same database, because they are… and treats it as an incremental import. And it has?

So, please help me understand now what’s gone wrong in your eyes because what you point out is exactly what I wanted it to do…

If the second forum was a superset of the first one, then I don’t see why you imported them both, but as long as the new forum doesn’t use any of the same user, post, or topic ids as the first one you should be ok.

When a user, topic, post is imported a XxxCustomField is created with name “import_id” so that you can tell the mapping from the old forum to the new one.

Some importers will also create an import_username custom field if the username changes. You can do something like

cd /var/discourse
rails c
u=User.find_by_username('username-youwant')
u.custom_fields

to see them.

1 Like

because I wanted the archived part merged back into the whole forum again, instead of having two “halves” of one forum. Now I don’t need to upkeep a separate ancient phpbb2 archive that I had to convert to discourse in some way or another… now its whole again :slight_smile: (we did the archive in an effort to reduce server load back in those days).

I tried your query, thank you,
for the usecase in my post I got {"import_id"=>"21293"} for NewName
and for OldName1 it did
{"import_id"=>"8800", "import_username"=>"OldName"}
so, wow it shows what it did there… Any reason why it decided that import_id8800 belonged to import username NewName? since they have different import_ID’s

Is there a way to do a more generic search because I don’t know what users are affected, that’s what I’m trying to figure out, so I don’t know the username-youwant part.

One way to do this would be:

  • Restore to the backup taken right after completing the archive import.
  • Wipe the import_id values from posts, topics, categories, and users.
  • Perform the current-forum import.

When you do this, users should get matched & combined by email.

4 Likes

Right. They have different import_ids, but the same username. If you want them to be the same user, they’ll need to have the same username (or you can contrive for them to have the same import_id).

Maybe search for

User.where("username like '%1'")

That’ll also catch people who just happened to want to end their username with a 1. .

And that will work perfectly . . . unless their email addresses changed.

However you got about it, you’ll have some stuff like this to resolve by hand.

1 Like

Quite useful. I am moving my gaming forum to discord. The PHPbb hosting (this one: https://www.cloudways.com/en/phpbb-hosting.php ) that I was using doesnt support apps other than PHP. So, I am in the process of exploring another platform for that.

I try to migrate a phpBB 3.2.9 forum to discourse. I have about 2.800 users and nearly 36.000 posts. For the import I use the Git repo of @gerhard with the phpbb3 branch.

The migration of the users and the PNs work fine. But with the posts I have a problem. There are only about 2.800 posts migrated. I got messages like the following one:

    12201 / 35281 ( 34.6%)  [68323 items/min]  Failed to map post with ID 14742
undefined method `name' for nil:NilClass
/var/www/discourse/script/import_scripts/phpbb3/support/bbcode/xml_to_markdown.rb:57:in `visit'
/var/www/discourse/script/import_scripts/phpbb3/support/bbcode/xml_to_markdown.rb:25:in `convert'
/var/www/discourse/script/import_scripts/phpbb3/support/text_processor.rb:41:in `process_raw_text'
/var/www/discourse/script/import_scripts/phpbb3/support/text_processor.rb:68:in `process_post'
/var/www/discourse/script/import_scripts/phpbb3/importers/post_importer.rb:35:in `map_post'
/var/www/discourse/script/import_scripts/phpbb3/importer.rb:130:in `block (2 levels) in import_posts'
/var/www/discourse/script/import_scripts/base.rb:491:in `block in create_posts'
/usr/local/lib/ruby/gems/2.6.0/gems/rack-mini-profiler-1.1.4/lib/patches/db/mysql2.rb:8:in `each'
/usr/local/lib/ruby/gems/2.6.0/gems/rack-mini-profiler-1.1.4/lib/patches/db/mysql2.rb:8:in `each'
/var/www/discourse/script/import_scripts/base.rb:490:in `create_posts'
/var/www/discourse/script/import_scripts/phpbb3/importer.rb:128:in `block in import_posts'
/var/www/discourse/script/import_scripts/base.rb:870:in `block in batches'
/var/www/discourse/script/import_scripts/base.rb:869:in `loop'
/var/www/discourse/script/import_scripts/base.rb:869:in `batches'
/var/www/discourse/script/import_scripts/phpbb3/importer.rb:186:in `batches'
/var/www/discourse/script/import_scripts/phpbb3/importer.rb:122:in `import_posts'
/var/www/discourse/script/import_scripts/phpbb3/importer.rb:33:in `execute'
/var/www/discourse/script/import_scripts/base.rb:47:in `perform'
/var/www/discourse/script/import_scripts/phpbb3/importer.rb:22:in `perform'
./script/import_scripts/phpbb3.rb:33:in `<module:PhpBB3>'
./script/import_scripts/phpbb3.rb:14:in `<module:ImportScripts>'
./script/import_scripts/phpbb3.rb:13:in `<main>'
    12202 / 35281 ( 34.6%)  [68322 items/min]  Failed to map post with ID 14744
undefined method `name' for nil:NilClass

I am not a Ruby expert. So I do not realy know how I can debug this. It looks like, that there is missing a method. But how to I get them and why are nearly 2.800 posts migrated?

Any ideas, what creates this problem and of course, how I can fix this?

Thank you!

1 Like

Sometimes there are some gremlins in code or DB goblins.

If your migration halts on error (sound like it does in this case), what I do is look at the error message and go into the offending file in the script and wrap the error in a simple:

begin
   #offending line or lines of ruby code
rescue
   puts "error! in <offending routine> variable #{variable} blab blah my great message"
end

I recently migrated over 1M posts, and trapping errors like in the example above and “moving on” was a lifesaver.

Every migration is different, not that I have so much experience like others, but I have done one about 20 different ways and 100 different times, LOL, and my experience is that the begin rescue end construct is very important.

Of course, you can dig deeper and see if there is a bigger problem later; but sometimes you just want to keep pluggn’ and migrat’n along :slight_smile:

2 Likes

Welcome @TBauer :wave:

Maybe you overlooked this warning in the first post.

2 Likes

@TBauer seems to be using this experimental branch, so I guess this is fine. It’s still not supported, but I’d like to know what’s causing this error anyway.

@TBauer could you please send me the content of the post_text column in the phpbb_posts table for post with ID 14742 in a PM or post it here?

I’m guessing you recently upgraded to phpBB 3.2 and it hasn’t finished converting the old BBCode format to the new one. That’s why it might fail after 2800 posts. If that’s the case, you might want to run the phpBB 3.2+ Text Reparser in a CLI. The import script currently doesn’t support mixing old and new storage formats for BBCodes and the BBCode to Markdown conversion should work a lot better with the new format. :wink:

4 Likes

No, the migration does not halt. The script runs until it regular ends. It only does not migrate a lot of posts. In this case I got the messages like above. I checked a view posts in the database I it looks like that these posts are all old ones without XML-Code.

@Helmi: of course I read this. That’s the reason why I used the experimental branch of the code. :wink:

1 Like

Maybe that’s the solution. I will try this first. I realized, that the failed posts are in BBCode and not in the XML-format… So next try to convert after reparse…

2 Likes