# Importers for large forums **URL:** \ **Category:** Announcements\ **Created:** [26 באפריל,‏ 2017,‏ 11:42pm UTC](https://meta.discourse.org/t/importers-for-large-forums/61681 "2017-04-26T23:42:37Z")\ **Posts on this page:** 11\ **Page:** 3
**Author:** ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)\ **Post date:** [26 ביולי,‏ 2017,‏ 2:19pm UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/41 "2017-07-26T14:19:28Z")
Can you convert it to UTF-8? ---
**Author:** ![mtawil](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mtawil/32/120812_2.png) [@mtawil](https://meta.discourse.org/u/mtawil)\ **Post date:** [26 ביולי,‏ 2017,‏ 3:35pm UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/42 "2017-07-26T15:35:47Z")
Why? can we just support UTF8mb4? ---
**Author:** ![mtawil](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mtawil/32/120812_2.png) [@mtawil](https://meta.discourse.org/u/mtawil)\ **Post date:** [26 ביולי,‏ 2017,‏ 11:14pm UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/43 "2017-07-26T23:14:41Z")
I submitted a new pull request that fixed an encoding issue. [https://github.com/discourse/discourse/pull/5003](https://github.com/discourse/discourse/pull/5003) ---
**Author:** ![quangbuule](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/quangbuule/32/120671_2.png) [@quangbuule](https://meta.discourse.org/u/quangbuule)\ **Post date:** [27 ביולי,‏ 2017,‏ 3:42pm UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/44 "2017-07-27T15:42:08Z")
Hi guys, I found that the BBCode converting is not working correctly using RegExp. For example: `"[COLOR=red]foo [COLOR=blue]bar[/COLOR] baz[/COLOR]"` will become `foo [COLOR=blue]bar baz[/COLOR]` Does it worth writing a small AST parser? ---
**Author:** ![mtawil](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/mtawil/32/120812_2.png) [@mtawil](https://meta.discourse.org/u/mtawil)\ **Post date:** [8 באוגוסט,‏ 2017,‏ 11:45am UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/46 "2017-08-08T11:45:56Z")
New issue: ```plaintext discourse@ip-10-0-1-178-app:/var/www/discourse$ IMPORT=1 RAILS_ENV=production ruby script/bulk_import/vbulletin.rb Loading application... Starting... Preloading I18n... Fixing highest post numbers... Loading imported group ids... Loading imported user ids... Loading imported category ids... Loading imported topic ids... Loading imported post ids... Loading groups indexes... Loading users indexes... Loading categories indexes... Loading topics indexes... Loading posts indexes... Importing groups... Importing users... 1417464 - 106/sec� /usr/local/lib/ruby/gems/2.4.0/gems/pg-0.20.0/lib/pg/connection.rb:168:in `get_last_result': ERROR: invalid byte sequence for encoding "UTF8": 0xed 0xa0 0xbd (PG::CharacterNotInRepertoire) CONTEXT: COPY user_custom_fields, line 960626 from /usr/local/lib/ruby/gems/2.4.0/gems/pg-0.20.0/lib/pg/connection.rb:168:in `copy_data' from /var/www/discourse/script/bulk_import/base.rb:511:in `create_custom_fields' from /var/www/discourse/script/bulk_import/base.rb:193:in `create_users' from script/bulk_import/vbulletin.rb:131:in `import_users' from script/bulk_import/vbulletin.rb:81:in `execute' from /var/www/discourse/script/bulk_import/base.rb:33:in `run' from script/bulk_import/vbulletin.rb:494:in `
' ``` I am tired of facing problems. ---
**Author:** ![quangbuule](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/quangbuule/32/120671_2.png) [@quangbuule](https://meta.discourse.org/u/quangbuule)\ **Post date:** [11 באוגוסט,‏ 2017,‏ 4:52pm UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/47 "2017-08-11T16:52:51Z")
importing users was \>7000/sec for my case, and it increased gradually during the process. I am using MacBook Pro, the SDD speed is big advantage. Your forum is big and may used buggy MySQL versions in some occasions before, it may be also crashed, rebooted … several times. In addition, this bulk importation is so beta and need improvements. So, I think you should try modifying some code and running on local too. P/s: I had to write some more code like custom BBCode parser, attachments, user avatars importing… to make my import perfect. ---
**Author:** ![fefrei](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/fefrei/32/119538_2.png) [@fefrei](https://meta.discourse.org/u/fefrei)\ **Post date:** [12 באוקטובר,‏ 2017,‏ 11:23am UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/48 "2017-10-12T11:23:02Z")
I’d like to report that this can be used to correct problems where the post counts of users are off (e.g. there are negative post counts). It’s pretty fast, too! 😃 (I could imagine a Sidekiq task that runs this once a week being a good idea, actually…) ---
**Author:** ![fefrei](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/fefrei/32/119538_2.png) [@fefrei](https://meta.discourse.org/u/fefrei)\ **Post date:** [24 באוקטובר,‏ 2017,‏ 1:34pm UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/49 "2017-10-24T13:34:50Z")
> [@zogstrip](#): > > It should be safe. But I haven’t testing it thoroughly. Highly recommended that you take a backup first 😉 Looking though the code… > …it looks like this will nuke the read stats ☹ Am I overlooking something? ---
**Author:** ![RoldanLT](https://avatars.discourse-cdn.com/v4/letter/r/da6949/32.png) [@RoldanLT](https://meta.discourse.org/u/RoldanLT)\ **Post date:** [5 במאי,‏ 2018,‏ 8:58am UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/50 "2018-05-05T08:58:01Z")
Waiting for XF1 support. I got 9M post forum. ---
**Author:** ![zogstrip](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/zogstrip/32/512781_2.png) [@zogstrip](https://meta.discourse.org/u/zogstrip)\ **Post date:** [5 במאי,‏ 2018,‏ 9:01am UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/51 "2018-05-05T09:01:42Z")
Want it sooner rather than later? You can sponsor the development of the XenForo 1 bulk importer by subscribing for [1-year of hosting](https://discourse.org/pricing) (_starting from the Business plan_) 😉 ---
**Author:** ![JammyDodger](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/jammydodger/32/254611_2.png) [@JammyDodger](https://meta.discourse.org/u/JammyDodger)\ **Post date:** [1 בדצמבר,‏ 2023,‏ 12:17pm UTC](https://meta.discourse.org/t/importers-for-large-forums/61681/53 "2023-12-01T12:17:13Z")
[Previous page](https://meta.discourse.org/t/importers-for-large-forums/61681.md?page=2)