Error when importing from Vanilla

Product Support Migration

AI-generated summary

huulbaek is attempting to migrate an ExpressionEngine 1.6.9 forum to Discourse using a modified Vanilla Porter, generating a 1.3GB CSV file. During import, a CSV::MalformedCSVError occurs at line 157118, prompting huulbaek to ask how to inspect that line. codinghorror notes CSV files are prone to formatting issues. huulbaek shares a workaround: copy the MySQL DB, convert to UTF-8, export via Vanilla Porter, import into Vanilla, then use vanilla_mysql.rb to migrate to Discourse — with a minor fix for a Ruby error where users was undefined. riking confirms this was a pasting error in their patch. Later, huulbaek reports success importing categories but encounters a Segmentation Fault during topic import, traced to rack-mini-profiler interfering with MySQL2’s each method in Ruby 2.0.0. The stack trace shows the fault occurs during batch processing of posts, halting the migration.

1

So, I am trying my very best to import a ExpressionEngine forum to Discourse. I rewrote some of the Vanilla Porter to make it work with our version of ExpressionEngine (1.6.9) and got the 1.3gb CSV file. But, when finally trying to import to Discourse this happens:

$ RAILS_ENV=production ruby script/import_scripts/vanilla.rb /shared/export.txt
loading existing groups...
loading existing users...
loading existing categories...
loading existing posts...
loading existing topics...
parsing file...
reading file...
parsing conversations...
parsing user_conversations...
parsing conversation_messages...
parsing permissions...
parsing users...
parsing roles...
parsing user_roles...
parsing categories...
parsing discussions...
parsing comments...
/usr/local/lib/ruby/2.0.0/csv.rb:1823:in `block (2 levels) in shift': Missing or stray quote in line 157118 (CSV::MalformedCSVError)
	from /usr/local/lib/ruby/2.0.0/csv.rb:1816:in `each'
	from /usr/local/lib/ruby/2.0.0/csv.rb:1816:in `block in shift'
	from /usr/local/lib/ruby/2.0.0/csv.rb:1776:in `loop'
	from /usr/local/lib/ruby/2.0.0/csv.rb:1776:in `shift'
	from /usr/local/lib/ruby/2.0.0/csv.rb:1718:in `each'
	from /usr/local/lib/ruby/2.0.0/csv.rb:1732:in `to_a'
	from /usr/local/lib/ruby/2.0.0/csv.rb:1732:in `read'
	from /usr/local/lib/ruby/2.0.0/csv.rb:1291:in `parse'
	from script/import_scripts/vanilla.rb:63:in `parse_file'
	from script/import_scripts/vanilla.rb:17:in `execute'
	from /var/www/discourse/script/import_scripts/base.rb:45:in `perform'
	from script/import_scripts/vanilla.rb:254:in `<main>'

I guess my question is: How would I find out what line 157118 contains? I guess this is a line after the importer has done something to the original CSV? Because the original line looks fine.

2

CSV files are notorious for having formatting issues.

3

I accept that answer :smile:

This is what I ended up doing should someone else get themselves in the same situation (I doubt that!):

  1. Make copy of MySQL database.
  2. Convert database copy and tables to UTF-8 if necessary.
  3. Export to CSV via Vanilla Porter - requires some small changes in the source to support older ExpressionEngine versions.
  4. Install Vanilla and import CSV into new database.
  5. Use vanilla_mysql.rb to import from this database - either remotely or locally.

@riking: Regarding that last step I had to make minor correction because users is not defined in

   next if all_records_exist? :users, users.map {|u| u['UserID'].to_i}

Not making any PR’s because I know nothing about Ruby.

4

Oops, that must have been a pasting error in my patch to skip finished chunks in the import!

5

I guess I was a little early in claiming success. Importing users and categories goes well but fails with Segmentation fault when starting to import topics.

importing categories...
       27 / 27 (100.0%)  
importing topics...
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.7/lib/patches/db/mysql2.rb:6: [BUG] Segmentation fault
ruby 2.0.0p647 (2015-08-18 revision 51631) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0012 p:---- s:0049 e:000048 CFUNC  :each
c:0011 p:0018 s:0046 e:000045 METHOD /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.7/lib/patches/db/mysql2.rb:6
c:0010 p:0054 s:0038 E:001798 METHOD /var/www/discourse/script/import_scripts/base.rb:408
c:0009 p:0088 s:0029 E:0026b8 BLOCK  script/import_scripts/vanilla_mysql.rb:99
c:0008 p:0008 s:0025 e:000024 BLOCK  /var/www/discourse/script/import_scripts/base.rb:670 [FINISH]
c:0007 p:---- s:0023 e:000022 CFUNC  :loop
c:0006 p:0012 s:0020 e:000019 METHOD /var/www/discourse/script/import_scripts/base.rb:669
c:0005 p:0051 s:0015 E:000990 METHOD script/import_scripts/vanilla_mysql.rb:87
c:0004 p:0013 s:0011 e:000010 METHOD script/import_scripts/vanilla_mysql.rb:25
c:0003 p:0029 s:0008 e:000007 METHOD /var/www/discourse/script/import_scripts/base.rb:45
c:0002 p:0084 s:0004 E:000ca8 EVAL   script/import_scripts/vanilla_mysql.rb:232 [FINISH]
c:0001 p:0000 s:0002 E:000be8 TOP    [FINISH]

-- Ruby level backtrace information ----------------------------------------
script/import_scripts/vanilla_mysql.rb:232:in `<main>'
/var/www/discourse/script/import_scripts/base.rb:45:in `perform'
script/import_scripts/vanilla_mysql.rb:25:in `execute'
script/import_scripts/vanilla_mysql.rb:87:in `import_topics'
/var/www/discourse/script/import_scripts/base.rb:669:in `batches'
/var/www/discourse/script/import_scripts/base.rb:669:in `loop'
/var/www/discourse/script/import_scripts/base.rb:670:in `block in batches'
script/import_scripts/vanilla_mysql.rb:99:in `block in import_topics'
/var/www/discourse/script/import_scripts/base.rb:408:in `create_posts'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.7/lib/patches/db/mysql2.rb:6:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.7/lib/patches/db/mysql2.rb:6:in `each'