UNICORN_WORKERS vs UNICORN_SIDEKIQS, DISCOURSE_DB_POOL?

Installation
1

Like a lot of people I am wanting to migrate my phpbb3 forum across to discourse. It’s a reasonable sized forum of 38,000 users and 98,000 threads. I have done one trial run and impressed by the import process so far.

The problem is the queued jobs processing time, and reading advice about creating more sidekiqs etc. Just across the posts people seem to be approaching this with different methods (setting workers, sidekiqs etc). I have search around but can’t find any description as to what is best.

Within your app.yml you can set:

UNICORN_WORKERS

But what does that set/do compared to setting

UNICORN_SIDEKIQS

In one post about increasing SIDEKIQS, it was recommended to increase DISCOURSE_DB_POOL as well, do you have to do this if you increase the workers? Or does the workers setting wrap sidekiqs? So what is best practice? Just setting UNICORN_WORKERS and it works it all out?

I am running my site more old school. (None of this cloud business). So it is a dedicated box:

Xeon E5-1620v2 4 Physical Cores, 8 Logical Cores, 3.7ghz
64gb Ram
2x2TB Spinny Disks Soft Raid.

The only other thing the box is currently running is an apache2.4 server, covering my wordpress install, and the old phpBB forums.

My current app.yml sets db_shared_buffers to 16GB, db_work_mem to 40MB.

Thanks for any help, Cheers.

4 Likes
2

This setting will increase the count of processes running the job queue. Each job queue out of the box runs 5 threads (5 concurrent jobs)

Increasing this to say 8, will mean you can run 40 concurrent background jobs. Which may be useful when you are struggling to clear up a job queue post import.

Keep in mind that at some point too much concurrency hurts you and stuff starts getting slower, so you want to play with this a bit to find the right number.

After you clear your job queue it is very rare to need more than 2 unicorn sidekiqs.

8 Likes
3

Thanks, so what does UNICORN_WORKERS do? Is that processes running the forum itself?

4

yes, that is the number of web workers, aka concurrent web requests the Rails stack can run.

7 Likes
5

If so, why the default is 5?

6

I’d like to ask how can I change number of threads (job queues) per sidekiq?

7

You would use the environment var: DISCOURSE_SIDEKIQ_WORKERS to increase that.

2 Likes