Problems after import from semi-large forum

Product Support

AI-generated summary

huulbaek reported issues after importing a semi-large forum from Vanilla, including a persistently high number of queued jobs (over 1.5 million), no emails being sent, and many topics missing categories. sam asked for import stats (28k users, ~575k posts) and suggested the email issue and missing categories may stem from the unresolved job queue. riking advised increasing Sidekiq workers in app.yml to clear the backlog. huulbaek confirmed the fix worked after scaling workers, though the process took over 48 hours on a 16GB DigitalOcean droplet.

1

Continuing the discussion from Importing from Vanilla:

It took about 30 hours to import from Vanilla MySQL database. However, I have encountered a couple of issues that I hope someone could help me with.

  • System reports a lot of queued jobs which stays constant at the same number - what does it even mean?

    The number of queued jobs is 1573616, which is high. This could indicate a problem with the Sidekiq process(es), or you may need to add more Sidekiq workers.

  • Emails are not being sent - there are no connection attempts to the smtp server. I have another instance of Discourse running with exact same SMTP setup where it is working. SMTP testet ok on host via telnet. Can this be related to queued jobs?

  • A lot of topics are without category - may be related to first problem?

3

How many topics / posts / categories / users are you importing?

Topics that are missing categories are going to stay with missing categories, sounds like you have missing mapping somewhere.

I would not expect any emails to come through before the job queue clears, may take quite a few hours.

4

About 28.000 users - that was fast import, categories likewise. .I think it was 575.000 posts. Everything seems right in the MySQL database.

The number is now 1.579.987 - so it seems to be going up if anything? :scream:

5

This is what you want to do - change the app.yml to launch more Sidekiq workers to process all the work from the import, until that’s all done.

6

It eventually worked, increasing the workers. However, it still took more than 48 hours on a DigitalOcean 16GB droplet. But thanks! :smile: