# Large Drupal forum migration, importer errors and limitations

**URL:** https://meta.discourse.org/t/large-drupal-forum-migration-importer-errors-and-limitations/246939
**Category:** Migration
**Tags:** drupal
**Created:** [November 28, 2022, 2:26pm UTC](https://meta.discourse.org/t/large-drupal-forum-migration-importer-errors-and-limitations/246939 "2022-11-28T14:26:50Z")
**Posts on this page:** 1
**Showing post:** 19

<div class="post-metadata">

### Author: ![rahim123](https://avatars.discourse-cdn.com/v4/letter/r/df705f/32.png) [@rahim123](https://meta.discourse.org/u/rahim123)
#### Post date: [January 3, 2023, 1:11am UTC](https://meta.discourse.org/t/large-drupal-forum-migration-importer-errors-and-limitations/246939/19 "2023-01-03T01:11:52Z")

</div>

Thanks Jay! Appreciate the encouragement.

> [@pfaffman](#):
>
> Do you have an estimate of how much time you spent doing your customizations?

Ugh, I’d prefer to not think about that. 😜 It was probably upwards of 15 or 20 hours after you put me on the right path with the SQL query.

I’d like to pick your brain on this if you have any thoughts:

> [@Ruby multi-CPU threading](https://meta.discourse.org/t/ruby-multi-cpu-threading/248433/15):
>
> I followed a combination of these two[[1]](https://meta.discourse.org/t/set-up-an-environment-to-migrate-another-forum-to-discourse/88252) guides[[2]](https://meta.discourse.org/t/use-an-import-script-that-requires-mysql/63349) for importing with access to another Docker container running a copy of the source forum database in MySQL. But it dawned on me that instead of creating a separate import container I can just use a single app container and add the mysql-dep.tempate to it: templates: - "templates/postgres.template.yml" - "templates/redis.template.yml" - "templates/web.template.yml" - "templates/web.ratelimited.template.yml" - "templates/web.ssl.templat…

It took around 70 hours to do a complete trial run with production data on a very powerful VPS. I’d like to get my users interacting again ASAP even if the posts and PMs import is still incomplete. Or another alternative idea I thought about would be to disable the `preprocess_posts` function, which I also heavily modified with additional `gsub` regexp replacements and also to pass all the posts and PMs though Pandoc with one of two different commands depending on whether the original post was Textile markup or pure HTML. If I disable the entire `preprocess_posts` routine it would probably cut the import time almost in half, and then I could add all that formatting stuff into the `postprocess_posts` section once all the raw data is imported. But the downside is that after the fact I wouldn’t be able to easily access the original database column that shows the source format (Textile or HTML) for each post, which is a conditional for my Pandoc manipulation. Or could I add a custom field to each post labeling it as `textile` or `html` and then retrieve that later during post-processing? Dunno, just thinking out loud here.

---

_[View the full topic](https://meta.discourse.org/t/large-drupal-forum-migration-importer-errors-and-limitations/246939)._