### Author:  [@leonardo](https://meta.discourse.org/u/leonardo)
#### Post date: [December 22, 2022, 1:46pm UTC](https://meta.discourse.org/t/migrating-vbulletin-5-database-import-script-errors/249495/28 "2022-12-22T13:46:35Z")
> [@Crius](#):
>
> seems like migration is not something Discourse has invested much time to support.
While I feel your pain (migrations is a bear of a subject), as a Migration Specialist with Discourse let me set the record straight.
We have a mature migration framework with [over 60 scripts](https://github.com/discourse/discourse/tree/main/script/import_scripts) for different platforms, a separate bulk framework with [5 scripts](https://github.com/discourse/discourse/tree/main/script/bulk_import), and a newer framework in the works which massively improves on every aspect - performance, code organization, testability, verifiability, documentation, and so on.
We have a separate Migrations team with extensive core developer support, and we contribute generic improvements back into the code with each migration we complete. We’re constantly doing migrations for customers ranging from trivial to unbelievably complex.
Our end goal is to make migrations as streamlined as possible both for hosted customers and for the community, but the amount of code that’s in scope during a migration is just too massive, and system level software configuration, third party software changes and input data variability only compounds the problem.
Again, I wish all this stuff was more painless, but making it so takes untold worker hours to create and maintain and there are only so many to go around.
Don’t give up! 🙂
---
### Author:  [@Crius](https://meta.discourse.org/u/Crius)
#### Post date: [December 22, 2022, 2:01pm UTC](https://meta.discourse.org/t/migrating-vbulletin-5-database-import-script-errors/249495/29 "2022-12-22T14:01:57Z")
I appreciate and understand that the scope is immense. It’s just frustrating to keep stumbling in exception over exception and the fact that the project is written in ruby doesn’t help in finding help beside coming here basically, which, cannot possibly accommodate all the requests for help as, as you say, some have very niche cases that are simply impossible to help with unless having hands on the actual data.
I also, for a large amount, put the blame on the absolute clusterfuck that is the structure of vbulletin.
I just checked this morning and this is a summary of table sizes.
To give context, the “text” table is where the actual content is.
the `node` table is holding the heriarchy and `closure`… let me quote here because I can’t even:
> The Closure table builds the parent-child relationships between all the nodes. That majority of your database is made up of attached files which shouldn’t be stored in the database anyway
So overall, for a forum with ~8GB of content, there is an overhead of 28GB. Great stuff, congrats vbulletin.
---
### Author:  [@Crius](https://meta.discourse.org/u/Crius)
#### Post date: [December 22, 2022, 4:57pm UTC](https://meta.discourse.org/t/migrating-vbulletin-5-database-import-script-errors/249495/30 "2022-12-22T16:57:48Z")
This is what I mean when I say that it’s frustrating.
Again, same set of actions (following a runbook written by myself with all the trial-and-error), running on a new discourse installation.
Result:
>
Where are you `import_user_account_id`? 😑
But most importantly? How did you manage to not cause an error the previous run where it failed on the topic import? 😖
Commenting out that function invocation (which seems like it was important anyway) and launching again:
Those duplicated keys errors… shouldn’t the script knows that it has already done those ids and move on?
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [December 22, 2022, 6:15pm UTC](https://meta.discourse.org/t/migrating-vbulletin-5-database-import-script-errors/249495/31 "2022-12-22T18:15:31Z")
> [@Crius](#):
>
> After the nth try (stopped counting after the 7th) I think I’m going to give up because seems like migration is not something Discourse has invested much time to support.
Every import is different. You’d think that a script that works for one instance of `your-previously-favorite-forum` would Just Work, but it doesn’t. And for a huge forum, it’s really hard. It’s simply not something that is easy to support. And the bulk importers access the database directly rather than counting on rails to be able to automagically check stuff as it goes.
> [@Crius](#):
>
> Using utf8 (default) simply generate lots of errors that are being reported but it’s not clear what is happening as the script goes ahead anyway.
That’s a not-infrequent problem, and not the fault of the script. You’ll need to figure out how to get your old database moved to utf8.
> [@Crius](#):
>
> . “Adoption” would need to have a strong migration support and it doesn’t seem like it’s the current state for discourse.
There is strong migration support. There just isn’t free migration support. I’ve done on the order of 100 migrations and written several import scripts for unsupported or bespoke systems. I’d likely charge $3000-5000 to import your database. That’s not an offer, that’s just to give you an idea of how much work it is for someone who’s done it a bunch of times. I suspect that if you were to pay for a year of Business hosting, CDCK would do it for free, which may be less than I would charge to do it. (Oh, but you might not be eligible for business hosting with a database that size).
---
### Author:  [@Crius](https://meta.discourse.org/u/Crius)
#### Post date: [December 22, 2022, 7:12pm UTC](https://meta.discourse.org/t/migrating-vbulletin-5-database-import-script-errors/249495/32 "2022-12-22T19:12:18Z")
Continuing my exploration here.
- The script reference a function that does not exists: `import_user_account_id`. You (discourse devs) might want to fix that.
- The logic that check for title’s of topics is somehow getting mad over some topics that for some reasons have empty string as title. As much as that shouldn’t happen, the check that evaluate that [should catch it and return `nil`](https://github.com/discourse/discourse/blob/main/script/bulk_import/base.rb#L854) but apparently that breaks the follow up logic written in the import (see [here](https://meta.discourse.org/t/migrating-vbulletin-5-database-import-script-errors/249495/27)).
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [December 22, 2022, 7:29pm UTC](https://meta.discourse.org/t/migrating-vbulletin-5-database-import-script-errors/249495/33 "2022-12-22T19:29:13Z")
> [@Crius](#):
>
> [should catch it and return `nil`](https://github.com/discourse/discourse/blob/main/script/bulk_import/base.rb#L854)
I had that issue with some import I did in recent memory. Better would be for it to return something like “topic XXX is missing a title” or pull the first line of text from the post, but that’s hard to do in this context. I think what I’d be tempted to do is fix it by munging your database and use something else to generate titles where they are missing.
---
### Author:  [@Crius](https://meta.discourse.org/u/Crius)
#### Post date: [December 22, 2022, 7:31pm UTC](https://meta.discourse.org/t/migrating-vbulletin-5-database-import-script-errors/249495/34 "2022-12-22T19:31:03Z")
Yep, I’m basically “sanitizing” the db when I found these issues but it’s just hard as I have to debug the script and then guess what is causing the issue each time 😅
Still no clue about the missing function `import_user_account_id` 😑
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [December 22, 2022, 7:33pm UTC](https://meta.discourse.org/t/migrating-vbulletin-5-database-import-script-errors/249495/35 "2022-12-22T19:33:24Z")
> [@Crius](#):
>
> Still no clue about the missing function `import_user_account_id` 😑
Especially given that the holidays are coming very soon, it’s unlikely that anyone will be fixing that unless they are using the script themselves. (Usually when I say that Richard will come in and save the day.)
---
### Author:  [@RGJ](https://meta.discourse.org/u/RGJ)
#### Post date: [December 22, 2022, 8:12pm UTC](https://meta.discourse.org/t/migrating-vbulletin-5-database-import-script-errors/249495/36 "2022-12-22T20:12:19Z")
> [@pfaffman](#):
>
> Usually when I say that Richard will come in and save the day
LOL, well, I guess I’m going to disappoint you today. I tried though, but I suspect that the commit of this importer was incomplete and did not include some changes to `base.rb`. @justin worked on this, maybe he knows. I do suspect that this could have been a customer specific thing that can be commented out without further consequences.
I have never used the bulk importers myself either.
Yes, import scripts can be complex and depend on the database specifics, but some scripts are simply not in a working state. That goes for this one as well, and there are some more scripts with for instance `# frozen_string_literal: true` that are just not working out of the box.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [December 22, 2022, 8:18pm UTC](https://meta.discourse.org/t/migrating-vbulletin-5-database-import-script-errors/249495/37 "2022-12-22T20:18:06Z")
> [@RGJ](#):
>
> LOL, well, I guess I’m going to disappoint you today.
Ha!
> [@RGJ](#):
>
> I suspect that the commit of this importer was incomplete and did not include some changes to `base.rb`.
That’s (at least)
> [@RGJ](#):
>
> and there are some more scripts with for instance `# frozen_string_literal: true` that are just not working out of the box.
part of why I find it so difficult to submit PRs for the changes that I make. By the time I’m done there’s so much case-specific stuff in there I’m afraid that whatever I submit will be broken somehow.
> [@RGJ](#):
>
> and there are some more scripts with for instance `# frozen_string_literal: true` that are just not working out of the box.
Yeah. I think that something went through and added `frozen_string_literal` to every file. Most files got fixed because they had tests, but there are no tests for the import scripts.
---
### Author:  [@Crius](https://meta.discourse.org/u/Crius)
#### Post date: [December 22, 2022, 8:28pm UTC](https://meta.discourse.org/t/migrating-vbulletin-5-database-import-script-errors/249495/38 "2022-12-22T20:28:23Z")
Hey, just to clarify, I’m not expecting anyone to fix this now (Karen style). I’m just pointing out some things that clearly have some issues on the codebase itself and probably are just “uops, I forgot to add this change to the commit! 😅”
I’ve already accepted that this migration won’t happen before January, at the least, at this point.
Everyone should just enjoy the holidays 🙂
I’ll bring this up or open a new thread after the holidays even if I’ll definitely have less time to dedicated to this migration ☹
---
### Author:  [@RGJ](https://meta.discourse.org/u/RGJ)
#### Post date: [December 22, 2022, 8:38pm UTC](https://meta.discourse.org/t/migrating-vbulletin-5-database-import-script-errors/249495/39 "2022-12-22T20:38:17Z")
> [@pfaffman](#):
>
> part of why I find it so difficult to submit PRs for the changes that I make
Yeah, totally true - for imports I usually don’t submit a PR before I did _two_ imports from different clients.
---
### Author:  [@Crius](https://meta.discourse.org/u/Crius)
#### Post date: [December 23, 2022, 11:47am UTC](https://meta.discourse.org/t/migrating-vbulletin-5-database-import-script-errors/249495/40 "2022-12-23T11:47:39Z")
Just keeping this updated.
I’ve made some changes to the `base.rb` after discussing it with some other engineers we have in our community. Lots of errors were caused by `gsub!` erroring because apparently we had some topics with `''` as title.
We added a function mimicking the `normalize_text` function that simply return the `imported_id` of the thread if there is not content, casted as string.
```plaintext
def normalize_text_thread(text, imported_id)
return imported_id.to_s unless text.present?
@html_entities.decode(normalize_charset(text.presence || "").scrub)
end
```
Then in `vbulletin5.rb` changed the line in the `create_topic` into:
```plaintext
create_topics(topics) do |row|
created_at = Time.zone.at(row[5])
title = normalize_text_thread(row[1], row[0])
```
That got rid of the issue. Basically `gsub!` doesn’t cope well with getting a `nil` in input.
However, this made the script go on but when it reached the `import_private_topics` it hanged there. There are 253.427 private topics (pm) in our DB which are several order of magnitude less than replies. After 9 hours I stopped the script to see what was going on actually.
Firing up the interface I noticed a couple of things.
1. My account was not imported because the admin user created was using the same email I suppose. Obvious but something that should be written somewhere maybe?
2. Only some of the categories (vbulletin subforums) where imported
3. Only topics and their first reply were imported (not sure if all of them really) and they were all imported without being in the correct categories, even the ones that had category that would have been created. Everything is imported “without category”.
4. The “replies counter” shows `-1`, probably because the replies were actually not imported at all.
I’ll add the overall, LOTS of issues with this bulk imported would go away if it was implementing a pagination approach. I think replies have gone missing because the script tried to go through them all at once and with 7GB of data it was impossible. It baffles me that a bulk importer doesn’t approach the import with a pagination approach to be honest. Even simply taking 1000 record at the time, writing them and storing the last record id written and looping would solve any issue with big databases.
---
### Author:  [@JammyDodger](https://meta.discourse.org/u/JammyDodger)
#### Post date: [December 23, 2022, 11:51am UTC](https://meta.discourse.org/t/migrating-vbulletin-5-database-import-script-errors/249495/41 "2022-12-23T11:51:00Z")
> [@Crius](#):
>
> Just keeping this updated.
FWIW, I’m following this with interest, and am very much appreciating the updates. 🙏 I don’t know much about migrations so far, but I am finding this very informative.
[Previous page](https://meta.discourse.org/t/migrating-vbulletin-5-database-import-script-errors/249495.md?page=1)
[Next page](https://meta.discourse.org/t/migrating-vbulletin-5-database-import-script-errors/249495.md?page=3)