I’m trying to import from a Vanilla forum using the instructions posted here. However, I get the following error when I run the vanilla.rb import script: Loading existing groups... Loading existing users... Loading existing categories... Loading existing posts... Loading existing topics... parsing file... reading file... Traceback (most recent call last): 5: from script/import_scripts/vanilla.rb:254:in ` ' 4: from /var/www/discourse/script/import_scripts/base.rb:47:in `perform' 3: from script/import_scripts/vanilla.rb:17:in `execute' 2: from script/import_scripts/vanilla.rb:37:in `parse_file' 1: from script/import_scripts/vanilla.rb:72:in `read_file' script/import_scripts/vanilla.rb:72:in `gsub': invalid byte sequence in UTF-8 (ArgumentError) I’ve tried changing the MySQL database character set to UTF8 following the instructions here and then re-exporting the porter file, but that didn’t resolve the issue. Any suggestions?

@dpkoch Did you ever figure this out?

You can google stuff about UTF-8 encoding. You need to do something that will coerce the table into UTF-8. The time that I did it, there were further complications because some rows were in one format and others in another format. I think that I did some nonsense where I coerced things on a value-by-value basis.

Sounds awful… We’ll have to tinker with the table encoding and see what happens. Thanks!

Oh. It’s aweful. You best bet, based on a vague recollection of a single time I did this over a year ago, is to play around with as many different conversions as you can until you can finally hit on one that works for all or most data. I think that I did a bunch of one-by-one transformations that ended up being a waste of time when I stumbled on some conversion that worked for all (most?) data. Here is what I did. Use at your own risk. (This was vbulletin, FWIW). def char_map(raw_original) raw = raw_original.dup debug = false # (raw.length > 50) # windows 1252 all = '' win_encoded = '' ### WIN1252 encoding win_encoded = '' begin win_encoded = raw.force_encoding('utf-8').encode("Windows-1252", invalid: :replace, undef: :replace, replace: "" ).force_encoding('utf-8').scrub rescue => e puts "\n#{'-'*50}\nWin1252 failed for \n\n#{raw}\n\n" win_encoded = '' end ### ISO 8859 encoding iso_encoded = '' if all.length == 0 && win_encoded.length > 0 && win_encoded != raw all = (debug ? "Win1252--" : '') + win_encoded else all = raw end all = old_char_map(all) all end

That code is used within the import script or on the server/database side?

In the import script. I don’t like to mess with the database. Somewhere you call this function on raw to fix raw (and maybe titles?).

Ok, thanks very much! This should give me a huge head-start in debugging this.

We got this to work by adding a simple command to encode the file as UTF-8 while reading it, using something like encode"UTF-8" on lines 76-80 of the vanilla.rb import script. I’m just waiting for confirmation on the exact syntax from the guy who did it via command line. I will update this when it I have it.

Here is what he used to fix this, starting on line 76 of vanilla.rb def read_file puts "reading file..." string = [File.read](http://file.read/)(@vanilla_file) .force_encoding('UTF-8').encode("UTF-8").gsub("\\N", "") .force_encoding('UTF-8').encode("UTF-8").gsub(/\\$\n/m, "\\n") .force_encoding('UTF-8').encode("UTF-8").gsub("\\,", ",") .force_encoding('UTF-8').encode("UTF-8").gsub(/(?<!\\)\\"/, '""') .force_encoding('UTF-8').encode("UTF-8").gsub(/\\\\\\"/, '\\""') [StringIO.new](http://stringio.new/)(string) end

Error importing from vanilla: invalid byte sequence in UTF-8

Product Support Migration

pfaffman (Jay Pfaffman) December 15, 2018, 1:09pm 2

You either need to keep trying to get it to really be UTF-8 or modify the import script to do it. It is a frustrating problem.

Topic		Replies	Views
Migrate a Vanilla forum to Discourse Sysadmins how-to	44	17034	January 30, 2023
[Paid] Need a Vanilla 2 Import tool Marketplace	66	11711	March 3, 2015
Error when importing from Vanilla Migration	4	2075	September 28, 2015
Migrate/Convert WP Posts to Discourse Topics Development	3	831	November 4, 2021
Migrating vBulletin 5 database - Import script errors Migration vbulletin5	46	3468	March 8, 2023

Error importing from vanilla: invalid byte sequence in UTF-8

Related topics