Just a quick update. Thanks to the work of an incredibly dedicated small group we are nearly done.
From next Monday we will start a few test runs on a staging machine but the results are promising.
This is the total numbers we have:
Size-wise, the DB in vbulletin3 format is around 8GB.
And the tests run from local machines connecting to the source db remotely is of around 6 hours to complete.
The script migrate all forum/subforums, translating them in categories and subcategories. Requires 3rd level subcategories because we have quite an old-style forum and there are some “clan” forums hosted there that are really REALLY nested in.
Anything beside the 3rd level is automatically converted in a tag, maintaining the hierarchical structure it had in terms of parent/child subforum relationship (using tag groups).
Every subforum that was either set with custom permission (read only for example), or mods/admin only or again, just hidden with password access, is getting migrated as “only accessible to staff” instead. They will be around a dozen in the end and we can have the staff re-enable them for the correct usergroups manually.
Users, usergroups and private messages are migrated as well. Private messages are migrated the “discourse way” meaning that instead of N topic with 1 message as you would see in a simple 1:1 migration of the database (silly database structure really), it will have the thread-organized way that discourse uses.
The script also does already the cooking, to speed up the process, for all posts.
Topic and posts migrations is done with several parallel connections and will always try to use as much connections as the source DB allow for.
We’ll see how much time it takes on average on a small 2vcore/4gb RAM but it’s already several order of magnitude faster than the current (unfinished) bulk migration script that is available.
Several parts could be optimized better and LOTS of it is really custom designed for our forum (we have even mapping in json to reorganize lots of the forum structure so it’s less chaotic) so I doubt it could be raised and used by anyone else without some tinkering but we’ll discuss internally if we want to open the source repo to the public after our migration completes.