I’ve been trying to import my vBulletin forum into Discourse for a while now. I’ve been following this guide: Importing from vBulletin 4.
I’m running the product on my Macbook using the Vagrant image as described on https://github.com/discourse/discourse/blob/master/docs/VAGRANT.md.
I’ve managed to finish the main database import after about 56 hours (~750K posts, ~10K users), but I’m running into trouble in the sidekiq task. It’s queued up 2 million tasks, and while it starts off right (cpu usage of the host going to 400%), it seems to just lock up after a while; host CPU usage is at exactly 100%, and the Vagrant image becomes unresponsive - can’t exit the main discourse process, can’t exit Sidekiq, and htop (which I had also running in screen) was just as unresponsible.
I’ve run it again, this time not starting the main Discourse process and having htop in the foreground. It’s done it again, this is the last what I can see in htop:
Nothing major, just a lot of processes running at low speeds, not even high CPU usage, but it’s still frozen.
Is this an issue in the software somewhere, or is this a known problem when running in Vagrant? Should I just get the whole thing set up locally? Or are there tricks to making this work?
Some technical details: Virtualbox gets access to 8 CPUs (autodetected by vagrant; I have 4 physical cpu cores, 8 with hyperthreading) and 4 GB of memory, a late 2013 model Macbook Pro. I’ve tweaked the database and sidekiq settings so that the DB has a pool of 25 and sidekiq a parallelism of 10 (-c argument).
It looks like it’s in a deadlock of sorts.
I’m running the HEAD commit of the main discourse repo, commit b963307 from about a week ago.
Any configuration tweaks I can do to help fix this?