Endless loop in digest emailing

AI-generated summary

A Discourse server entered an endless loop of sending digest emails, consuming 100% CPU and blocking all other email delivery. doom369 reported the issue began around the digest cron time and persisted for 3 days despite app rebuilds and disabling summary emails. pfaffman suggested queued emails were being delivered repeatedly, but doom369 noted the problem was ongoing and referenced a similar memory issue. The workaround was temporarily removing the SMTP auth token, which caused errors and reduced load, after which restoring it resolved the issue. doom369 later identified the root cause: their email provider had removed verified host records needed for the from field, causing SMTP errors that triggered the loop. codinghorror and codinghorror agreed Discourse should handle such failures more gracefully. sam confirmed the issue was resolved after updating mail credentials doom369.

Hello all.

Our discourse server went into endless loop sending emails non stop. Here is shared/standalone/log/rails/production.log output:

Delivered mail 7db8eecd-c36a-419c-ae34-0561b08bae25@community.xx.cc (1322.2ms)
  Rendered user_notifications/digest.text.erb (Duration: 50.1ms | Allocations: 12577)
  Rendering user_notifications/digest.html.erb within layouts/email_template
  Rendered user_notifications/digest.html.erb within layouts/email_template (Duration: 139.5ms | Allocations: 26910)
  Rendering user_notifications/digest.text.erb
  Rendered user_notifications/digest.text.erb (Duration: 71.7ms | Allocations: 13505)
Delivered mail 5bbf2e54-96a8-437d-a032-69f98a00f22a@community.xx.cc (1485.2ms)
  Rendering user_notifications/digest.html.erb within layouts/email_template
  Rendered user_notifications/digest.html.erb within layouts/email_template (Duration: 70.9ms | Allocations: 20613)
  Rendering user_notifications/digest.text.erb
  Rendered user_notifications/digest.text.erb (Duration: 38.2ms | Allocations: 11390)

This started happen suddenly (probably when digest cron started:

)

Now 1 core is fully busy with sending digset email.

  • App rebuild doesn’t help;

  • Updated to the latest version - no change;

  • Turn on “Disable summary emails for all users.” and rebuiled the app - no change;

    #### Installed
    
    ### 2.4.0.beta5
    

What else can I do? How ca I stop this?
Thanks in advance.

1 Like

I could be wrong, but I think that what happened is that the summary emails got queued and now it’s delivering them. It’s going to keep going until it delivers them.

If I’m right, You can disable emails then find the table that has them and delete them. But it’s just going to happen again the next time that summary emails go out.

1 Like

Yes, but this happens for 3 days already. All other mails (register, for example are not delivered anymore. Community works, but all emailing - not).
I found very similar issue - Extreme memory usage due to bad mail credentials. So look like this is some kind of tricky bug.

The only way I found in order to stop this - I removed the auth token in my SMTP provider. Discourse start showing hundreds of errors and only after that load decreased. I restored the token and now all seems ok.

1 Like

Did you have bad email credentials?

@codinghorror I found the root cause of the issue. Our mail provider removed (for their own stupid reasons) verified host records for our account. The host records were used to provide from field in the email with our host. So after they did that - discourse went into endless loop. I think discourse flow has a bug as well for that case. Because it should report the error and stop load 1 core for 100%.

So account is correct. But mail provider returns an error on mail send “Wrong sending host” or something like that and this flow causes the issue.

2 Likes

I agree, this is a bad state to be in and we should handle it better.

1 Like

Is everything back to normal now?

Yes. After I changed credentials for mail.

1 Like