I’m trying to import from a Vanilla forum using the instructions posted here. However, I get the following error when I run the vanilla.rb import script: Loading existing groups... Loading existing users... Loading existing categories... Loading existing posts... Loading existing topics... parsing file... reading file... Traceback (most recent call last): 5: from script/import_scripts/vanilla.rb:254:in ` ' 4: from /var/www/discourse/script/import_scripts/base.rb:47:in `perform' 3: from script/import_scripts/vanilla.rb:17:in `execute' 2: from script/import_scripts/vanilla.rb:37:in `parse_file' 1: from script/import_scripts/vanilla.rb:72:in `read_file' script/import_scripts/vanilla.rb:72:in `gsub': invalid byte sequence in UTF-8 (ArgumentError) I’ve tried changing the MySQL database character set to UTF8 following the instructions here and then re-exporting the porter file, but that didn’t resolve the issue. Any suggestions?

Oh. It’s aweful. You best bet, based on a vague recollection of a single time I did this over a year ago, is to play around with as many different conversions as you can until you can finally hit on one that works for all or most data. I think that I did a bunch of one-by-one transformations that ended up being a waste of time when I stumbled on some conversion that worked for all (most?) data. Here is what I did. Use at your own risk. (This was vbulletin, FWIW). def char_map(raw_original) raw = raw_original.dup debug = false # (raw.length > 50) # windows 1252 all = '' win_encoded = '' ### WIN1252 encoding win_encoded = '' begin win_encoded = raw.force_encoding('utf-8').encode("Windows-1252", invalid: :replace, undef: :replace, replace: "" ).force_encoding('utf-8').scrub rescue => e puts "\n#{'-'*50}\nWin1252 failed for \n\n#{raw}\n\n" win_encoded = '' end ### ISO 8859 encoding iso_encoded = '' if all.length == 0 && win_encoded.length > 0 && win_encoded != raw all = (debug ? "Win1252--" : '') + win_encoded else all = raw end all = old_char_map(all) all end

That code is used within the import script or on the server/database side?

In the import script. I don’t like to mess with the database. Somewhere you call this function on raw to fix raw (and maybe titles?).

Ok, thanks very much! This should give me a huge head-start in debugging this.

We got this to work by adding a simple command to encode the file as UTF-8 while reading it, using something like encode"UTF-8" on lines 76-80 of the vanilla.rb import script. I’m just waiting for confirmation on the exact syntax from the guy who did it via command line. I will update this when it I have it.

Here is what he used to fix this, starting on line 76 of vanilla.rb def read_file puts "reading file..." string = [File.read](http://file.read/)(@vanilla_file) .force_encoding('UTF-8').encode("UTF-8").gsub("\\N", "") .force_encoding('UTF-8').encode("UTF-8").gsub(/\\$\n/m, "\\n") .force_encoding('UTF-8').encode("UTF-8").gsub("\\,", ",") .force_encoding('UTF-8').encode("UTF-8").gsub(/(?<!\\)\\"/, '""') .force_encoding('UTF-8').encode("UTF-8").gsub(/\\\\\\"/, '\\""') [StringIO.new](http://stringio.new/)(string) end

This doesn’t work. Same error.

You need to Google about encoding and figure out your to fix your broken encoding.

Hi @ddeveloper, I worked through this process just a couple of months ago (and I am not a developer :slightly_smiling_face:) and managed to successfully migrate a self-hosted Vanilla forum to self-hosted Discourse. One thing that was key for me, was making sure when doing the data export with Vanilla Porter, to select “Vanilla 2” as the Source Forum Type in the first drop-down menu. I used the Vanilla Porter 2.6 version available for download as a zip file here: https://open.vanillaforums.com/discussion/38882/vanilla-porter-2-6-rc1 instead of the 2.5 version linked in Migrate a Vanilla forum to Discourse. If I remember correctly, I didn’t hit the UTF-8 error again when using the newer Vanilla Porter script and the “Vanilla 2” forum type. If those two suggestions don’t make a difference for your import, please provide a few details about the steps you’ve taken so far and exactly what you’re seeing. Sometimes there are slight variations to “same error” that can make a big difference when troubleshooting.

I’ve followed the same guide except using the porter version 2.6. I will the export file from version 2.6 and update here.

Error importing from vanilla: invalid byte sequence in UTF-8

Product Support Migration

Topic		Replies	Views
Migrate a Vanilla forum to Discourse Sysadmins how-to	44	17031	January 30, 2023
[Paid] Need a Vanilla 2 Import tool Marketplace	66	11710	March 3, 2015
Error when importing from Vanilla Migration	4	2075	September 28, 2015
Migrate/Convert WP Posts to Discourse Topics Development	3	831	November 4, 2021
Migrating vBulletin 5 database - Import script errors Migration vbulletin5	46	3466	March 8, 2023

Error importing from vanilla: invalid byte sequence in UTF-8

Related topics