Is there a way I can send email notifications faster?

[Richie (Richie Rich)]

We have an #Announcements category in which all our members are automatically subscribed to watch the first post.

This allows us to schedule the publishing of posts in to that category on set times/dates.

In turn, the announcement is then emailed to around 25k members.

The problem we face is that the emails take over an hour to get sent, not ideal for time-critical announcements.

If I watch Sidekiq I can see the “Scheduled” counter racking up all the individual emails one by one, once it gets to around 20,000 it then moves them over to the “Enqueued” tab, and then they eventually start sending out.

Am I able to speed this process up at all?

For time critical emails it’d be nice to send them out around 100x faster than they currently go

[jahan_gagan]

How to set Email sending speed limit?

I have a forum on discourse but with slow sending speed limit (20/sec) Am I able to set sending speed limit on discourse? I’m asking this because I found that I got nearly 500,000 records everyday on unsubscribe_keys table, but only 9000 in mail_log and 1000 in skipped_mail_log, our forum have nearly 500,000 members, too. I’m not sure whether it’s caused by sending email too fast and make some weird problem happened, so I’m trying to throttle the sending speed but with no luck. Unsubscribe_k…

This discussion may help you, as it explains how to set DISCOURSE_MAX_DIGESTS_ENQUEUED_PER_30_MINS_PER_SITE, which defines the global limit for digests.

[Richie (Richie Rich)]

@Richie

If I watch Sidekiq I can see the “Scheduled” counter racking up all the individual emails one by one, once it gets to around 20,000 it then moves them over to the “Enqueued” tab, and then they eventually start sending out.

I timed it this evening.

I scheduled a post to publish in to the #Announcements category at 18:30.

---

At 18:40 there were 15,000 emails in the Scheduled queue.

---

At 18:45, so 15 minutes after the post, there were 22,000 emails in the Scheduled queue.

---

At 18:48 those emails then gradually started moving over to the Enqueued queue:

---

At 18:51 they were still moving over:

---

At 19:03 the emails were on their way out:

---

By 19:10 there were just 10,000 emails remaining:

---

At 19:27 there were just 569 emails remaining:

---

And at 19:29 all the emails had gone out:

---

So there we have it, one full hour to send 22,000 email notifications.

Can anyone help me identify the bottleneck here?

I’d very much like to be able to send these emails out faster than their current 22000-per-hour rate.

[TempAccount]

Shooting from the hip,
Is it possible it’s an infrastructure load issue?

[Richie (Richie Rich)]

Perhaps?

I genuinely don’t know

The CPU spiked to around 44% on the server:

And I use AWS SES for the SMTP.

[supermathie (Michael Brown)]

There’s two things going on here:

• delay while email jobs are enqueued
• processing time for sending the actual email

For the first, I’m not 100% confident on this but I think lowering email_time_window_mins means the notifications get queued sooner.

Once the email jobs are scheduled, your sidekiq workers are working through them one at a time. Bumping up the sidekiq workers (set DISCOURSE_SIDEKIQ_WORKERS up from 5 to 10, 15, or 20 depending on server capacity) means more jobs get processed at the same time, so the queue gets emptied 2x/3x/4x faster.

[TempAccount]

@supermathie

For the first, I’m not 100% confident on this but I think lowering email_time_window_mins means the notifications get queued sooner.

I don’t know how the backend works in the finest detail, but the email_time_window_mins is just a delay setting before the first email goes out. During this time, any user that is set to receive the email may “forfeit” the email if they happen to be active within the window of period (official terminology: email skipped; skip reasoning: User was seen recently).

The default delay is 10 minutes, meaning the post must be live for 10 minutes, and only then will the emails be sent.

Richie’s issue is the time difference between the first email and the last email… a delay of one hour. This is probably due to the sheer amount of emails that need to be sent, though I cannot say for certain either.

Changing the setting above would only expedite the sending of the first email, but not address the completion duration of the entire batch of 22,000 emails.

---

@supermathie

DISCOURSE_SIDEKIQ_WORKERS

What would be the recommended setting, obviously dependent on infrastructure capability?

Could a high setting result in server issues in terms of load or other?

[supermathie (Michael Brown)]

@TempAccount

What would be the recommended setting, obviously dependent on infrastructure capability?

Could a high setting result in server issues in terms of load or other?

“As many as the server can handle”.

It’s 100% dependent on the OP’s server capacity - too many and it’ll slow things down, too few and it’ll take longer to process.

Given the CPU graph hitting 40% (is that of a single CPU or total capacity?) I’d probably start with bumping it either 2× (conservative) or 3× (aggressive) and see what happens, depending on risk tolerance for slowdowns.

[Richie (Richie Rich)]

@supermathie

Once the email jobs are scheduled, your sidekiq workers are working through them one at a time. Bumping up the sidekiq workers (set DISCOURSE_SIDEKIQ_WORKERS up from 5 to 10, 15, or 20 depending on server capacity) means more jobs get processed at the same time, so the queue gets emptied 2x/3x/4x faster.

Great insight, thank you

Does that DISCOURSE_SIDEKIQ_WORKERS have to be a multiple of 5? Could I set it to 7 for example?

I don’t have that parameter setting in my app.yml so I’ll assume it’s on a default of 5 somewhere.

Can I just create that setting under the existing unicorn workers setting, then rebuild?

Eg:

expose:
  - “443:443”

env:
    UNICORN_WORKERS: 8
    DISCOURSE_SIDEKIQ_WORKERS: 7

Is it as simple as that?

[Richie (Richie Rich)]

@Richie

Can I just create that setting under the existing unicorn workers setting, then rebuild?

Eg:

expose:
  - “443:443”

env:
    UNICORN_WORKERS: 8
    DISCOURSE_SIDEKIQ_WORKERS: 7

Is it as simple as that?

Is anyone able to confirm this is the correct change to make to my app.yml when the DISCOURSE_SIDEKIQ_WORKERS parameter does not currently exist?

[TempAccount]

I believe yes, given a brief inquiry to the AI bot.

Set Environmental Variables

This is a how-to guide for setting up environmental variables in a self-hosted Discourse installation. Required user level: Administrator Environmental variables play a crucial role in configuring your Discourse instance. They can store sensitive data like API keys and database passwords, making your installation more secure and flexible. In Discourse, environmental variables are set in the app.yml file within your Docker container setup…

[Richie (Richie Rich)]

Thanks @TempAccount

Interestingly, DISCOURSE_SIDEKIQ_WORKERS only appears eleven times across the whole of meta:

https://meta.discourse.org/search?q=%22DISCOURSE_SIDEKIQ_WORKERS%22%20order%3Alatest_topic

…and one of those is in this topic

[supermathie (Michael Brown)]

Yes, this is correct.

Any (most?) setting can be overridden in this manner. The default value comes from discourse_defaults.conf.

[Richie (Richie Rich)]

Thanks @supermathie

My app.yml now reads:

env:
  LANG: en_US.UTF-8
  # DISCOURSE_DEFAULT_LOCALE: en

  ## How many concurrent web requests are supported? Depends on memory and CPU cores.
  ## will be set automatically by bootstrap based on detected CPUs, or you can override
  UNICORN_WORKERS: 8

  ## Added this line on 04/10/25
  ## REF: https://meta.discourse.org/t/is-there-a-way-i-can-send-email-notifications-faster/383103/12
  DISCOURSE_SIDEKIQ_WORKERS: 7

I’ll rebuild later in the week and time it sending emails

Just so I know my change has taken effect, am I right in thinking this Threads value should increase from 5 to 7?