Migrate an SMF2 forum to Discourse

[Zup]

Sorry to reply again. I’m struggling to figure out why permalinks aren’t being created.

To recap:
I’m using the default ugly URLs in SMF2 on the toplevel root domain. Discourse is installed on a subdomain of that root domain. I’ve used the smf2.rb importer in the latest Discourse repo to convert SMF2 to Discourse (following your guide), but there are no permalinks in Discourse after import finished.

Are there any additional steps needed, such as certain settings or plugin(s) running on either SMF2 or Discourse?

[pfaffman (Jay Pfaffman)]

@Zup

I’m using the default ugly URLs in SMF2

It looks like that function is for pretty URLs? So you’d need to change it accordingly (unless I missed something in the code).

[Zup]

I’m a bit confused because Marco didn’t use pretty URLs but permalinks were created for him and he used the same import script.

This is the pretty URLs function in the import script: https://github.com/discourse/discourse/blob/master/script/import_scripts/smf2.rb#L674

I’m hoping for some help to get this working for SMF’s default “ugly” URLs… any direction would be greatly appreciated. Thanks guys.

[pfaffman (Jay Pfaffman)]

I don’t know. But my best free answer is that I think what you want to look at is line 222 that says:

           Permalink.find_or_create_by(url: "forums/index.php/board,#{c["id_board"]}.0.html", category_id: category.id)

You’d change that permalink (or maybe the couple other ones?) to whatever ugly ones look like? Though, that looks pretty ugly to me. You might also adjust the permalink_normalizations at the top of the file.

[Zup]

Sorry Jay for replying to you again, but line 222 where? Line 222 of smf2.rb is only:

if parent

[pfaffman (Jay Pfaffman)]

Well, you and I are looking at different versions, then. I’m not sure why that might be. But the thing to look for is those lines about Permalinks.

[Zup]

I’m confused because the version I linked seems to be the most recent “official” version… but you’re the second person who referenced a different version. What am I missing? Where can I get your version?

[pfaffman (Jay Pfaffman)]

Aha! Sorry. It looks like I clicked the smf1 script. So look at smf1.rb1 that’s right there with the smf2.rb script you’re looking at.

You can either look at the smf1 script or use make_prettyurl_permalinks as a model to create a make_uglyurl_permalinks. But that is reading from the pretty URL table, so it might be easier to use smf1.rb as a model, but I’d have to look at the database to tell for sure. No. That won’t help much either unless you modify the create_xxx functions.

Sorry. I don’t see a fix that can be explained here.

[JustinZ (Justin)]

I’m up to here, but cannot seem to get past. I have even created a new user on the MySQL server with a native password and tried but to no avail.

su discourse -c "bundle exec ruby script/import_scripts/smf2.rb /shared/smf2 -t UTC"
Loading existing groups...
Loading existing users...
Loading existing categories...
Loading existing posts...
Loading existing topics...
Traceback (most recent call last):
        7: from script/import_scripts/smf2.rb:701:in `<main>'
        6: from script/import_scripts/smf2.rb:28:in `run'
        5: from script/import_scripts/smf2.rb:28:in `new'
        4: from script/import_scripts/smf2.rb:62:in `initialize'
        3: from script/import_scripts/smf2.rb:274:in `create_db_connection'
        2: from script/import_scripts/smf2.rb:274:in `new'
        1: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/mysql2-0.5.3/lib/mysql2/client.rb:90:in `initialize'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/mysql2-0.5.3/lib/mysql2/client.rb:90:in `connect': RSA Encryption not supported - caching_sha2_password plugin was built with GnuTLS support (Mysql2::Error)

[JustinZ (Justin)]

OK after MANY hours back and forth trying to get this to work I have identified the issue and am posting it here for any others that might get stuck like I did.

The MySQL docker container is version 8, this means for some reason the mariadb library that the MySQL template draws in doesn’t work.

I did not include the MySQL template file in the import docker container config file, this was the first change.

I built the import container, entered into it with ./launcher enter import

Then:

echo "gem 'mysql2'" >> Gemfile
wget https://dev.mysql.com/get/mysql-apt-config_0.8.17-1_all.deb
dpkg -i mysql-apt-config_0.8.17-1_all.deb

This has an interactive prompt and you can select the defaults (MySQL 8 with tools)

Then I installed the normal MySQL 8 library and continued the build:

apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y libmysqlclient-dev
su discourse -c 'bundle config unset deployment'
su discourse -c 'bundle install --no-deployment --path vendor/bundle --jobs 4 --without test development'

Once this was all done the rest was the same, the import is now running.

The Discourse devs may want to adjust the import scripts to account for this, at least it’s now on the forums for others to see if they get stuck like I did.

[Zup]

@miligraf

1. After the import is finished, the user needs to destroy the import container because if not, that’s the one that will run on server boot (instead of the app container).
2. If you’re planning on using S3 for uploads - then set it up before running the import!

1. Destroy according to these instructions? how-to-migrate-import-from-smf2-to-discourse/90129#cleanup-5

2. I’ve manually transferred the attachments directory from smf2 to Discourse prior to importing, then ran the importer once, but uploads aren’t appearing in posts in Discourse. Any ideas why?

3. I’ve already run the importer once without S3 enabled, but I want to use S3. What should I do?

[miligraf]

1. Yes
2. Hmm…I’d try enabling S3 and re-importing.
3. Enable S3 before starting the import.

[Zup]

I already ran the importer once without S3 though. Would uploads from topics that were already imported be transferred to S3 during a delta import?

[miligraf]

It’s best if you just start over and enable S3 before importing.

[Zup]

Getting this error upon creating users:

oxipng worker: oxipngnot found; please provide proper binary or disable this worker (--no-oxipng argument or:oxipng => false through options)

Then after that it begins creating posts.

• Could this be why attachments aren’t appearing in Discourse?
• Where should oxipng be installed?

Update: I’m confused why I’m getting this oxipng error. I can’t find anything about it anywhere in Discourse, and I’m experiencing this issue only upon running the smf2 import script. Could it possibly be related to this? https://meta.discourse.org/t/faster-and-smaller-uploads-in-discourse-with-rust-webassembly-and-mozjpeg-blog

[Falco]

That is just a warning from the image optimization library we use at server side, shouldn’t block anything.

[Zup]

So I thought my import issues were possibly related to S3 issues, but turns out S3 seems to be working just fine. The issue I’m having is that after running the import script (with S3 enabled) and rebuilding the import container, it seems all topics in Discourse are lacking the uploads that are attachments in smf2 topics. In other words, there’s no visual cue that there is even an upload in a Discourse topic, which is clearly seen as an attachment in its smf2 topic equivalent. It’s the same result when I import with S3 disabled. A bit out of ideas here. Any thoughts?

[pfaffman (Jay Pfaffman)]

It may be that the images are attached to the post, so the import script knows to import them, but that they are not referenced in the post so that they don’t appear. You need to modify the post text to include a link to the images. I’ve seen that, I think with a different forum.

[Zup]

You mean modify the post on smf2 to include links to the attached files before importing to Discourse?

[pfaffman (Jay Pfaffman)]

Well, that could be a way.

But I mean to modify the import script to append the attachment to the raw post. Something like

discourse/discourse/blob/main/script/import_scripts/vbulletin.rb#L607-L627