Error when importing from Vanilla

migration
1

So, I am trying my very best to import a ExpressionEngine forum to Discourse. I rewrote some of the Vanilla Porter to make it work with our version of ExpressionEngine (1.6.9) and got the 1.3gb CSV file. But, when finally trying to import to Discourse this happens:

$ RAILS_ENV=production ruby script/import_scripts/vanilla.rb /shared/export.txt
loading existing groups...
loading existing users...
loading existing categories...
loading existing posts...
loading existing topics...
parsing file...
reading file...
parsing conversations...
parsing user_conversations...
parsing conversation_messages...
parsing permissions...
parsing users...
parsing roles...
parsing user_roles...
parsing categories...
parsing discussions...
parsing comments...
/usr/local/lib/ruby/2.0.0/csv.rb:1823:in `block (2 levels) in shift': Missing or stray quote in line 157118 (CSV::MalformedCSVError)
	from /usr/local/lib/ruby/2.0.0/csv.rb:1816:in `each'
	from /usr/local/lib/ruby/2.0.0/csv.rb:1816:in `block in shift'
	from /usr/local/lib/ruby/2.0.0/csv.rb:1776:in `loop'
	from /usr/local/lib/ruby/2.0.0/csv.rb:1776:in `shift'
	from /usr/local/lib/ruby/2.0.0/csv.rb:1718:in `each'
	from /usr/local/lib/ruby/2.0.0/csv.rb:1732:in `to_a'
	from /usr/local/lib/ruby/2.0.0/csv.rb:1732:in `read'
	from /usr/local/lib/ruby/2.0.0/csv.rb:1291:in `parse'
	from script/import_scripts/vanilla.rb:63:in `parse_file'
	from script/import_scripts/vanilla.rb:17:in `execute'
	from /var/www/discourse/script/import_scripts/base.rb:45:in `perform'
	from script/import_scripts/vanilla.rb:254:in `<main>'

I guess my question is: How would I find out what line 157118 contains? I guess this is a line after the importer has done something to the original CSV? Because the original line looks fine.

2

CSV files are notorious for having formatting issues.

3

I accept that answer :smile:

This is what I ended up doing should someone else get themselves in the same situation (I doubt that!):

  1. Make copy of MySQL database.
  2. Convert database copy and tables to UTF-8 if necessary.
  3. Export to CSV via Vanilla Porter - requires some small changes in the source to support older ExpressionEngine versions.
  4. Install Vanilla and import CSV into new database.
  5. Use vanilla_mysql.rb to import from this database - either remotely or locally.

@riking: Regarding that last step I had to make minor correction because users is not defined in

   next if all_records_exist? :users, users.map {|u| u['UserID'].to_i}

Not making any PR’s because I know nothing about Ruby.

3 Likes
4

Oops, that must have been a pasting error in my patch to skip finished chunks in the import!

5

I guess I was a little early in claiming success. Importing users and categories goes well but fails with Segmentation fault when starting to import topics.

importing categories...
       27 / 27 (100.0%)  
importing topics...
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.7/lib/patches/db/mysql2.rb:6: [BUG] Segmentation fault
ruby 2.0.0p647 (2015-08-18 revision 51631) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0012 p:---- s:0049 e:000048 CFUNC  :each
c:0011 p:0018 s:0046 e:000045 METHOD /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.7/lib/patches/db/mysql2.rb:6
c:0010 p:0054 s:0038 E:001798 METHOD /var/www/discourse/script/import_scripts/base.rb:408
c:0009 p:0088 s:0029 E:0026b8 BLOCK  script/import_scripts/vanilla_mysql.rb:99
c:0008 p:0008 s:0025 e:000024 BLOCK  /var/www/discourse/script/import_scripts/base.rb:670 [FINISH]
c:0007 p:---- s:0023 e:000022 CFUNC  :loop
c:0006 p:0012 s:0020 e:000019 METHOD /var/www/discourse/script/import_scripts/base.rb:669
c:0005 p:0051 s:0015 E:000990 METHOD script/import_scripts/vanilla_mysql.rb:87
c:0004 p:0013 s:0011 e:000010 METHOD script/import_scripts/vanilla_mysql.rb:25
c:0003 p:0029 s:0008 e:000007 METHOD /var/www/discourse/script/import_scripts/base.rb:45
c:0002 p:0084 s:0004 E:000ca8 EVAL   script/import_scripts/vanilla_mysql.rb:232 [FINISH]
c:0001 p:0000 s:0002 E:000be8 TOP    [FINISH]

-- Ruby level backtrace information ----------------------------------------
script/import_scripts/vanilla_mysql.rb:232:in `<main>'
/var/www/discourse/script/import_scripts/base.rb:45:in `perform'
script/import_scripts/vanilla_mysql.rb:25:in `execute'
script/import_scripts/vanilla_mysql.rb:87:in `import_topics'
/var/www/discourse/script/import_scripts/base.rb:669:in `batches'
/var/www/discourse/script/import_scripts/base.rb:669:in `loop'
/var/www/discourse/script/import_scripts/base.rb:670:in `block in batches'
script/import_scripts/vanilla_mysql.rb:99:in `block in import_topics'
/var/www/discourse/script/import_scripts/base.rb:408:in `create_posts'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.7/lib/patches/db/mysql2.rb:6:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.7/lib/patches/db/mysql2.rb:6:in `each'