' 13: from /home/canapin/discourse/script/import_scripts/base.rb:47:in `perform' 12: from script/import_scripts/vbulletin5.rb:49:in `execute' 11: from script/import_scripts/vbulletin5.rb:300:in `import_posts' 10: from /home/canapin/discourse/script/import_scripts/base.rb:862:in `batches' 9: from /home/canapin/discourse/script/import_scripts/base.rb:862:in `loop' 8: from /home/canapin/discourse/script/import_scripts/base.rb:863:in `block in batches' 7: from script/import_scripts/vbulletin5.rb:320:in `block in import_posts' 6: from /home/canapin/discourse/script/import_scripts/base.rb:508:in `create_posts' 5: from /usr/local/rvm/gems/ruby-2.6.5/gems/rack-mini-profiler-2.0.4/lib/patches/db/mysql2.rb:8:in `each' 4: from /usr/local/rvm/gems/ruby-2.6.5/gems/rack-mini-profiler-2.0.4/lib/patches/db/mysql2.rb:8:in `each' 3: from /home/canapin/discourse/script/import_scripts/base.rb:509:in `block in create_posts' 2: from script/import_scripts/vbulletin5.rb:321:in `block (2 levels) in import_posts' 1: from script/import_scripts/vbulletin5.rb:450:in `preprocess_post_raw' script/import_scripts/vbulletin5.rb:450:in `gsub': invalid byte sequence in UTF-8 (ArgumentError) ``` How can I properly identify the post to see what the content looks like in the vbulletin database? ---

### Author: ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman) #### Post date: [18 augustus 2020 om 21:32 UTC](https://meta.discourse.org/t/migrate-a-vbulletin-4-forum-to-discourse/54881/98 "2020-08-18T21:32:28Z")

> [@Canapin](#): > > How can I properly identify the post to see what the content looks like in the vbulletin database? Someone suggested ways to use `rescue` to solve those, so you might go back and find that (I can’t remmber if it was in this topic or another one). You could put a `put` in the `rescue` to print out the `id` and/or the text that caused the problem. > [@Canapin](#): > > `invalid byte sequence in UTF-8` You have an encoding problem. I used this in a similar import (I think you’d put it in `preprocess_post_raw`) ```plaintext begin win_encoded = raw.force_encoding('utf-8').encode("Windows-1252", invalid: :replace, undef: :replace, replace: "" ).force_encoding('utf-8').scrub rescue => e puts "\n#{'-'*50}\nWin1252 failed for \n\n#{raw}\n\n" win_encoded = '' end ``` ---

### Author: ![Canapin](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/canapin/32/119591_2.png) [@Canapin](https://meta.discourse.org/u/Canapin) #### Post date: [19 augustus 2020 om 07:59 UTC](https://meta.discourse.org/t/migrate-a-vbulletin-4-forum-to-discourse/54881/100 "2020-08-19T07:59:26Z")

Hi, I modified the the importer and added your script as following: ```plaintext def preprocess_post_raw(raw) return "" if raw.blank? begin win_encoded = raw.force_encoding('utf-8').encode("Windows-1252", invalid: :replace, undef: :replace, replace: "" ).force_encoding('utf-8').scrub rescue => e puts "\n#{'-'*50}\nWin1252 failed for \n\n#{raw}\n\n" win_encoded = '' end # decode HTML entities raw = @htmlentities.decode(raw) # fix whitespaces raw = raw.gsub(/(\\r)?\\n/, "\n") .gsub("\\t", "\t") ``` The invalid byte sequence in UTF-8 happens on this part : `raw = raw.gsub(/(\\r)?\\n/, "\n") .gsub("\\t", "\t")`. Then I started the importer again. Though it skips already importing data, it took about 6 hours to get to the post that generates an error, and it didn’t add the expected information to see the post content. 😖Any idea why? edit: This is probably the post raw content that leads to the error: ```plaintext I wonder if Billy is enjoying the parade. Qwertyuiopasdfghjklzxcvbnm�� ``` I’ll try to modify the importer script to make it skip (for real) the previous 1.4M posts. WIsh me luck. 🤞 ---

### Author: ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman) #### Post date: [19 augustus 2020 om 12:58 UTC](https://meta.discourse.org/t/migrate-a-vbulletin-4-forum-to-discourse/54881/101 "2020-08-19T12:58:48Z")

I modified many other importers to include an import\_after setting to allow importing only recent data. You can look at some others to see how I did that. ---

### Author: ![Canapin](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/canapin/32/119591_2.png) [@Canapin](https://meta.discourse.org/u/Canapin) #### Post date: [19 augustus 2020 om 22:48 UTC](https://meta.discourse.org/t/migrate-a-vbulletin-4-forum-to-discourse/54881/102 "2020-08-19T22:48:48Z")

Hi, I’ve been able to import almost all my posts! I fixed a few dozens by hand and restarted the import each time it came across a new utf-8 error… 😅 Now, I need to import the attachments (which are stored in the VBulletin database), but it doesn’t work: When it starts the process, my ram consumption increases a lot in about 10 or 20 seconds and this error happens: ```plaintext importing attachments... Failed to create upload: Cannot allocate memory - grep Fail ``` My ram: ![image](https://global.discourse-cdn.com/meta/original/3X/a/9/a9e432ab2b77cedeecfcf7a1bdd2d3ddc127855f.png) I use a Discourse development version on a Ubuntu 18 subsystem on Windows 10 and I have 16 GB of RAM. The attachments take 7 GB from the 13 GB vBulletin database. Note that I use the **vbulletin5** importer. The issue comes from this query: ```SQL SELECT n.parentid nodeid, a.filename, fd.userid, LENGTH(fd.filedata) AS dbsize, filedata, fd.filedataid FROM #{DB_PREFIX}attach a LEFT JOIN #{DB_PREFIX}filedata fd ON fd.filedataid = a.filedataid LEFT JOIN #{DB_PREFIX}node n on n.nodeid = a.nodeid ``` If I do this query in mysql, my remaining ram is filled within seconds. * * * (editing my post to remove useless info and questions since I’m figuring out things and providing a workaround) ## Workaround: I added a limit and a offset to the importer SQL query. I imported the attachments by selecting 20000 of them each time: ```ruby uploads = mysql_query <<-SQL SELECT n.parentid nodeid, a.filename, fd.userid, LENGTH(fd.filedata) AS dbsize, filedata, fd.filedataid FROM #{DB_PREFIX}attach a LEFT JOIN #{DB_PREFIX}filedata fd ON fd.filedataid = a.filedataid LEFT JOIN #{DB_PREFIX}node n on n.nodeid = a.nodeid LIMIT 20000 OFFSET 0 SQL ``` I also added an `exit` at the end of the `uploads.each do |upload|` loop to prevent the import script to continue doing things after importing my 20000 uploads. When my 10000 uploads are imported, I edit the script (thanks `nano +353 ./scripts/import_scripts/vbulletin5.rb` to open the file at the right line) to increase the SQL query `OFFSET` by 10000, and start the importer again… And doing so for my 65000 attachments. During the attachment imports, I faced several errors and warnings including: - `W, [2020-08-20T12:05:37.402860 #31042] WARN -- : Bad date/time value "0000:00:00 00:00:00": mon out of range` - `Post for 490451 not found` (dangling old attachments I guess?) - some EXIF data error it seems - `Fail` This one puzzled me and stopped the import script. I checked the first “Fail” I got and the bulletin attachment was sort of broken (no filename), so I commented the `exit` instruction to let the importer continue his importing job when he “fails”, hoping that wouldn’t break anything. ```ruby puts "Fail" #exit ``` I also had a more annoying error that interrupted the import: ```plaintext 1: from /usr/local/rvm/gems/ruby-2.6.5/gems/activerecord-6.0.3.2/lib/active_record/validations.rb:53:in `save!' /usr/local/rvm/gems/ruby-2.6.5/gems/activerecord-6.0.3.2/lib/active_record/validations.rb:80:in `raise_validation_error': Validation failed: Body is limited to 32000 characters; you entered 32323. (ActiveRecord::RecordInvalid) ``` Fortunately, it was a rare error, and I just skipped this attachment until I came across the next identical error. It happened maybe a dozen times on a total of 65000 attachments. I just restarted the import script with a different SQL query offset. ---

### Author: ![Canapin](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/canapin/32/119591_2.png) [@Canapin](https://meta.discourse.org/u/Canapin) #### Post date: [22 augustus 2020 om 09:38 UTC](https://meta.discourse.org/t/migrate-a-vbulletin-4-forum-to-discourse/54881/103 "2020-08-22T09:38:04Z")

Hi, I noticed that the custom field **import\_pass** was absent for about 400 users of my remaining 27000 users (I cleaned up 154000 inactive users). Any idea why? The forum was migrated from phpBB to vBulletin in May. Could it have something to do with that? I won’t try to “fix” this thing and import passwords for these 400 users (unless there’s an easy way to do it…?) and that’s not a big issue, so I’m just being curious more than anything else. ---

### Author: ![Canapin](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/canapin/32/119591_2.png) [@Canapin](https://meta.discourse.org/u/Canapin) #### Post date: [22 augustus 2020 om 18:10 UTC](https://meta.discourse.org/t/migrate-a-vbulletin-4-forum-to-discourse/54881/104 "2020-08-22T18:10:14Z")

Hey guys, Imported images have the wrong width/height ratio unless I rebake the posts. I’d like to find a way to have the correct ratio (during the import for example) without rebaking. ### More verbose description of the issue: From what I understand, imported posts aren’t “baked” when Discourse creates the corresponding post (though the **cooked** field is generated somehow), so that’s why importing posts is way faster than baking existing Discourse posts. My issue is that my imported images have the wrong width/height ratio. Example of the raw Discourse text related to an imported image: `![SH-MUniFrame.JPG|600x800](upload://6Li1nnjbA8zDz6YJ3FqeYHV5zXK.jpeg)` The content of the “cooked” field: `

` How the image appears in: the post ![image](https://global.discourse-cdn.com/meta/original/3X/b/d/bdd2ab141ab2ca549d8e46ffa60c62573d17ed14.jpeg) Here is the original image: [https://global.discourse-cdn.com/meta/original/3X/f/7/f73a0ae8594219dd5a1620e59b3c17f9b02b1583.jpeg](https://global.discourse-cdn.com/meta/original/3X/f/7/f73a0ae8594219dd5a1620e59b3c17f9b02b1583.jpeg) The original image size from the vBulletin database is: ```plaintext select width, height from filedata where filedataid = 76237 +-------+--------+ | width | height | +-------+--------+ | 600 | 800 | +-------+--------+ ``` My understanding is that the height attribute is constrained by Discourse’s setting which sets a max height of 500px, hence the same value in the `` height attribute. The `` width is somewhat modified from 600 to 517 though I can’t figure it how and why. The issue is the same for older images that have 0 in both width and height vBulletin attachment fields. They also have the wrong height/width issue. I don’t know if these values are really used during the import. The issue is resolved by rebaking (rebuild HTML) the post. the image will then be properly resized and the image viewer is added. But I have 1.6M posts and I’d prefer to avoid rebake all of them. A quick fix would be to use this CSS on my Discourse: ```css .cooked img:not(.emoji) { height: auto; width: auto; } ``` But it implies that no one will be able to choose an arbitrary size when uploading an image, and there may be collateral effects I’m not aware of. Why idea how I could have proper images width/height ratio on imported attachments? ---

### Author: ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman) #### Post date: [22 augustus 2020 om 19:37 UTC](https://meta.discourse.org/t/migrate-a-vbulletin-4-forum-to-discourse/54881/105 "2020-08-22T19:37:48Z")

I suspect that’s because you didn’t let them cook after the import. I cannot imagine a way to solve the problem without rebaking the posts. Perhaps you want to just rebake the posts that are broken rather than all of them? ---

### Author: ![Canapin](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/canapin/32/119591_2.png) [@Canapin](https://meta.discourse.org/u/Canapin) #### Post date: [22 augustus 2020 om 19:47 UTC](https://meta.discourse.org/t/migrate-a-vbulletin-4-forum-to-discourse/54881/106 "2020-08-22T19:47:02Z")

Are they supposed to be cooked automatically over time after the import? Starting from the last or the first created post? That’s not a big issue though, and if they are not automatically cooked, I’d probably start a rebake of all posts and be patient, although I admit that I read this post a few days ago and it scared me a little bit: [My journey into a massive posts rebake job](https://meta.discourse.org/t/my-journey-into-a-massive-posts-rebake-job/84816). I also have questions about that, but I’ll ask them in the proper topic. 😊 ---

### Author: ![michaeld](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/michaeld/32/1594_2.png) [@michaeld](https://meta.discourse.org/u/michaeld) #### Post date: [22 augustus 2020 om 20:10 UTC](https://meta.discourse.org/t/migrate-a-vbulletin-4-forum-to-discourse/54881/107 "2020-08-22T20:10:32Z")

> [@Canapin](#): > > If I do this query in mysql, my remaining ram is filled within seconds. Hmm yes that looks like it’s my code. Sorry for that. 😕 This should be following this pattern: ``` batches(BATCH_SIZE) do |offset| (Sql code) LIMIT #{BATCH_SIZE} OFFSET #{offset} (Other code) end ``` ---

### Author: ![michaeld](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/michaeld/32/1594_2.png) [@michaeld](https://meta.discourse.org/u/michaeld) #### Post date: [22 augustus 2020 om 20:13 UTC](https://meta.discourse.org/t/migrate-a-vbulletin-4-forum-to-discourse/54881/108 "2020-08-22T20:13:37Z")

> [@Canapin](#): > > `Body is limited to 32000 characters` Just raise the `max post length` site setting prior to the import. [Vorige pagina](https://meta.discourse.org/t/migrate-a-vbulletin-4-forum-to-discourse/54881.md?page=4) [Volgende pagina](https://meta.discourse.org/t/migrate-a-vbulletin-4-forum-to-discourse/54881.md?page=6)