# Importers for large forums **URL:** \ **Category:** Announcements\ **Created:** [2017年四月26日 23:42 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681 "2017-04-26T23:42:37Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![mtawil](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mtawil/32/120812_2.png) [@mtawil](https://meta.discourse.org/u/mtawil)\ **Post date:** [2017年七月24日 11:16 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/21 "2017-07-24T11:16:19Z")
Yeah, I edited it, and it works. But now I face a new issue when importing users: ```plaintext Importing users... script/bulk_import/vbulletin.rb:363:in `strptime': invalid date (ArgumentError) from script/bulk_import/vbulletin.rb:363:in `parse_birthday' from script/bulk_import/vbulletin.rb:78:in `block in import_users' from /var/www/discourse/script/bulk_import/base.rb:438:in `block (2 levels) in create_records' from /usr/local/lib/ruby/gems/2.4.0/gems/rack-mini-profiler-0.10.5/lib/patches/db/mysql2.rb:6:in `each' from /usr/local/lib/ruby/gems/2.4.0/gems/rack-mini-profiler-0.10.5/lib/patches/db/mysql2.rb:6:in `each' from /var/www/discourse/script/bulk_import/base.rb:437:in `block in create_records' from /usr/local/lib/ruby/gems/2.4.0/gems/pg-0.20.0/lib/pg/connection.rb:160:in `copy_data' from /var/www/discourse/script/bulk_import/base.rb:436:in `create_records' from /var/www/discourse/script/bulk_import/base.rb:178:in `create_users' from script/bulk_import/vbulletin.rb:72:in `import_users' from script/bulk_import/vbulletin.rb:25:in `execute' from /var/www/discourse/script/bulk_import/base.rb:33:in `run' from script/bulk_import/vbulletin.rb:377:in `
' ``` ---
**Author:** ![zogstrip](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/zogstrip/32/512781_2.png) [@zogstrip](https://meta.discourse.org/u/zogstrip)\ **Post date:** [2017年七月24日 11:18 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/22 "2017-07-24T11:18:07Z")
Can you share some examples of birthday dates? Maybe the format isn’t the same. ---
**Author:** ![mtawil](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mtawil/32/120812_2.png) [@mtawil](https://meta.discourse.org/u/mtawil)\ **Post date:** [2017年七月24日 11:23 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/23 "2017-07-24T11:23:12Z")
The incorrect date is (02-29-1989) ---
**Author:** ![mtawil](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mtawil/32/120812_2.png) [@mtawil](https://meta.discourse.org/u/mtawil)\ **Post date:** [2017年七月24日 11:33 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/24 "2017-07-24T11:33:55Z")
I know it’s the wrong date, but that what I get from users, I can’t check all dates manually and edit it. The easiest way is that making `parse_birthday` function to check if the date is correct or not. I wish I can develop on ruby; I try to learn it. ---
**Author:** ![zogstrip](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/zogstrip/32/512781_2.png) [@zogstrip](https://meta.discourse.org/u/zogstrip)\ **Post date:** [2017年七月24日 11:36 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/25 "2017-07-24T11:36:48Z")
What if you replace the method by this? ```ruby def parse_birthday(birthday) return if birthday.blank? date_of_birth = Date.strptime(birthday, "%m-%d-%Y") rescue nil date_of_birth ||= Date.strptime(birthday, "(%m-%d-%Y)") rescue nil return if date_of_birth.nil? date_of_birth.year < 1904 ? Date.new(1904, date_of_birth.month, date_of_birth.day) : date_of_birth end ``` ---
**Author:** ![mtawil](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mtawil/32/120812_2.png) [@mtawil](https://meta.discourse.org/u/mtawil)\ **Post date:** [2017年七月24日 11:59 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/26 "2017-07-24T11:59:18Z")
OK, it works and skips the wrong date now. New issue 😩 : After importing more than 200K users (I stop it by CTRL+C) I went to check out the imported users and sadly there are no inserted data in ‘discourse\_development.users’ table. Should it complete the importer to show up the users data or what? ---
**Author:** ![zogstrip](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/zogstrip/32/512781_2.png) [@zogstrip](https://meta.discourse.org/u/zogstrip)\ **Post date:** [2017年七月24日 12:26 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/27 "2017-07-24T12:26:20Z")
I pushed the [fixes](https://github.com/discourse/discourse/commit/57d6a5dc9c6aaa3f9bfa99adf3b6e18f2258d78b) so that I don’t forget. > [@mtawil](#): > > After importing more than 200K users (I stop it by CTRL+C) I went to check out the imported users and sadly there are no inserted data in ‘discourse\_development.users’ table. > > Should it complete the importer to show up the users data or what? Yup. The bulk importer works in “batches”. And by “batch” I mean “table by table”. For ⚡ speed, everything is kept in memory before it’s sent to the database. ---
**Author:** ![zogstrip](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/zogstrip/32/512781_2.png) [@zogstrip](https://meta.discourse.org/u/zogstrip)\ **Post date:** [2017年七月24日 12:57 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/28 "2017-07-24T12:57:58Z")
Also, I just [merged a PR](https://github.com/discourse/discourse/pull/4987) fixing the bulk importer regarding to our email extraction into another table. @mtawil you should totally update before re-running the import. ---
**Author:** ![quangbuule](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/quangbuule/32/120671_2.png) [@quangbuule](https://meta.discourse.org/u/quangbuule)\ **Post date:** [2017年七月24日 13:05 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/29 "2017-07-24T13:05:03Z")
Thank @zogstrip! There are some problems with charset, some forums started from very early of vBulletin era, at that time, its charset was latin1. So the texts after importing are really messy. Will have a PR on that. ---
**Author:** ![mtawil](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mtawil/32/120812_2.png) [@mtawil](https://meta.discourse.org/u/mtawil)\ **Post date:** [2017年七月24日 16:41 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/30 "2017-07-24T16:41:55Z")
Thank you! I will update it of course. BTW, The speed rate continues to slow down by time. For more clarification, please check this video out: [![](https://global.discourse-cdn.com/meta/original/4X/5/c/5/5c5d4d3126f26b833e3e2e02a3075d74640f5196.jpeg "Speed rate issue - Discourse") ](https://www.youtube.com/watch?v=VJ8dghCbBXo) That’s just ~1.5M users, imagine more than 80M posts, the speed rate maybe will going to be 1/sec 🙂 ---
**Author:** ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)\ **Post date:** [2017年七月24日 17:02 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/31 "2017-07-24T17:02:41Z")
How much ram do you have? You’ll need to have enough ram to hold the whole table. My guess is that you’re starting to swap and that’s slowing your down. Is swap on a hard drive or ssd? ---
**Author:** ![mtawil](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mtawil/32/120812_2.png) [@mtawil](https://meta.discourse.org/u/mtawil)\ **Post date:** [2017年七月24日 17:56 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/32 "2017-07-24T17:56:15Z")
Well, Does `Intel 16 Core Xeon CPU @ 2.30GHz w/ 64GB of RAM` is not enough? ---
**Author:** ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)\ **Post date:** [2017年七月24日 17:58 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/33 "2017-07-24T17:58:24Z")
Oh. Darn. So much for that explanation. 🤯 ---
**Author:** ![mtawil](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mtawil/32/120812_2.png) [@mtawil](https://meta.discourse.org/u/mtawil)\ **Post date:** [2017年七月25日 11:04 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/34 "2017-07-25T11:04:46Z")
Can it be “row by row”? Instead of “table by table”? And starts from where it ended (after last imported ID)? This will be very helpful for large forums. ---
**Author:** ![zogstrip](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/zogstrip/32/512781_2.png) [@zogstrip](https://meta.discourse.org/u/zogstrip)\ **Post date:** [2017年七月25日 11:09 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/35 "2017-07-25T11:09:03Z")
Then it won’t be a **bulk** importer, would it? 😉 ---
**Author:** ![mtawil](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mtawil/32/120812_2.png) [@mtawil](https://meta.discourse.org/u/mtawil)\ **Post date:** [2017年七月25日 11:14 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/36 "2017-07-25T11:14:25Z")
Well, can we call it a **chunk** importer? 🙄 ---
**Author:** ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)\ **Post date:** [2017年七月25日 11:52 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/37 "2017-07-25T11:52:14Z")
That’s what the regular importer does. Perhaps that’s what you want. ---
**Author:** ![mtawil](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mtawil/32/120812_2.png) [@mtawil](https://meta.discourse.org/u/mtawil)\ **Post date:** [2017年七月25日 11:58 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/38 "2017-07-25T11:58:23Z")
What I mean is when I want to stop the bulk importer and rerun it (for speed rate issue), it should start at the last row, not from the first row. ---
**Author:** ![quangbuule](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/quangbuule/32/120671_2.png) [@quangbuule](https://meta.discourse.org/u/quangbuule)\ **Post date:** [2017年七月25日 14:59 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/39 "2017-07-25T14:59:26Z")
Hi guys, is there any need of mapping old forums (categories) into new configurable categories and tags? We can merge or split old categories into a new structure. My idea is having .yml file that contains new categories and their old categories’ ids, including tags somehow. ---
**Author:** ![mtawil](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mtawil/32/120812_2.png) [@mtawil](https://meta.discourse.org/u/mtawil)\ **Post date:** [2017年七月26日 12:44 UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/40 "2017-07-26T12:44:31Z")
_ **New issue:** _ ```plaintext discourse@ip-10-0-1-178-app:/var/www/discourse$ IMPORT=1 RAILS_ENV=production ruby script/bulk_import/vbulletin.rb Loading application... Starting... Preloading I18n... Fixing highest post numbers... Loading imported group ids... Loading imported user ids... Loading imported category ids... Loading imported topic ids... Loading imported post ids... Loading groups indexes... Loading users indexes... Loading categories indexes... Loading topics indexes... Loading posts indexes... Importing groups... Importing users... 1270000 - 119/sec /var/www/discourse/script/bulk_import/base.rb:521:in `blank?': invalid byte sequence in UTF-8 (ArgumentError) from /var/www/discourse/script/bulk_import/base.rb:521:in `fix_name' from /var/www/discourse/script/bulk_import/base.rb:234:in `process_user' from /var/www/discourse/script/bulk_import/base.rb:486:in `block (2 levels) in create_records' from /usr/local/lib/ruby/gems/2.4.0/gems/rack-mini-profiler-0.10.5/lib/patches/db/mysql2.rb:6:in `each' from /usr/local/lib/ruby/gems/2.4.0/gems/rack-mini-profiler-0.10.5/lib/patches/db/mysql2.rb:6:in `each' from /var/www/discourse/script/bulk_import/base.rb:483:in `block in create_records' from /usr/local/lib/ruby/gems/2.4.0/gems/pg-0.20.0/lib/pg/connection.rb:160:in `copy_data' from /var/www/discourse/script/bulk_import/base.rb:482:in `create_records' from /var/www/discourse/script/bulk_import/base.rb:191:in `create_users' from script/bulk_import/vbulletin.rb:131:in `import_users' from script/bulk_import/vbulletin.rb:81:in `execute' from /var/www/discourse/script/bulk_import/base.rb:33:in `run' from script/bulk_import/vbulletin.rb:494:in `
' ``` * * * My forum encoding is “ **_UTF8mb4_** ” [上一頁](https://meta.discourse.org/t/importers-for-large-forums/61681.md?page=1) [下一頁](https://meta.discourse.org/t/importers-for-large-forums/61681.md?page=3)