We have an #Announcements category in which all our members are automatically subscribed to watch the first post.
This allows us to schedule the publishing of posts in to that category on set times/dates.
In turn, the announcement is then emailed to around 25k members.
The problem we face is that the emails take over an hour to get sent, not ideal for time-critical announcements.
If I watch Sidekiq I can see the “Scheduled” counter racking up all the individual emails one by one, once it gets to around 20,000 it then moves them over to the “Enqueued” tab, and then they eventually start sending out.
Am I able to speed this process up at all?
For time critical emails it’d be nice to send them out around 100x faster than they currently go
This discussion may help you, as it explains how to set DISCOURSE_MAX_DIGESTS_ENQUEUED_PER_30_MINS_PER_SITE, which defines the global limit for digests.
For the first, I’m not 100% confident on this but I think lowering email_time_window_mins means the notifications get queued sooner.
Once the email jobs are scheduled, your sidekiq workers are working through them one at a time. Bumping up the sidekiq workers (set DISCOURSE_SIDEKIQ_WORKERS up from 5 to 10, 15, or 20 depending on server capacity) means more jobs get processed at the same time, so the queue gets emptied 2x/3x/4x faster.
I don’t know how the backend works in the finest detail, but the email_time_window_mins is just a delay setting before the first email goes out. During this time, any user that is set to receive the email may “forfeit” the email if they happen to be active within the window of period (official terminology: email skipped; skip reasoning: User was seen recently).
The default delay is 10 minutes, meaning the post must be live for 10 minutes, and only then will the emails be sent.
Richie’s issue is the time difference between the first email and the last email… a delay of one hour. This is probably due to the sheer amount of emails that need to be sent, though I cannot say for certain either.
Changing the setting above would only expedite the sending of the first email, but not address the completion duration of the entire batch of 22,000 emails.
What would be the recommended setting, obviously dependent on infrastructure capability?
Could a high setting result in server issues in terms of load or other?
It’s 100% dependent on the OP’s server capacity - too many and it’ll slow things down, too few and it’ll take longer to process.
Given the CPU graph hitting 40% (is that of a single CPU or total capacity?) I’d probably start with bumping it either 2× (conservative) or 3× (aggressive) and see what happens, depending on risk tolerance for slowdowns.
env:
LANG: en_US.UTF-8
# DISCOURSE_DEFAULT_LOCALE: en
## How many concurrent web requests are supported? Depends on memory and CPU cores.
## will be set automatically by bootstrap based on detected CPUs, or you can override
UNICORN_WORKERS: 8
## Added this line on 04/10/25
## REF: https://meta.discourse.org/t/is-there-a-way-i-can-send-email-notifications-faster/383103/12
DISCOURSE_SIDEKIQ_WORKERS: 7
I’ll rebuild later in the week and time it sending emails
Just so I know my change has taken effect, am I right in thinking this Threads value should increase from 5 to 7?
It’s not limiting itself per se but each sidekiq worker will process one job at a time, so if you have 22000 emails waiting to go out seven of them will be processed at once.
On the ridiculous side of things, the server probably won’t be able to keep up if you set the number to 1000 parallel workers. So it’s about finding a number that suits all of your needs:
process as many jobs as possible at the same time to get through the 22000 emails faster
but don’t take up ALL of the server resources which would leave none left for users of the site
