Feature Requests: Separate Email Account for Digests

[neounix (Dark Matter)]

From a novice, today we had an experience of all our email basically shutdown and users could not register, recover passwords or login with email because the weekly digest kicked off after a legacy forum migration and there were over 300K emails in the sidekiq queue; so anyone who tried to login via email, register, recover their password, etc did not get any email and were hosed (as they say)…

The issue was based on the fact that we use GMAIL as our mail relay, and (free) GMAIL set limits on this sort of SMTP relaying and so GMAIL shut us down for the day.

I would like to ask for this feature in the future (unless there is another way to address this).

Proposal

Add another set of app.yml vars which permit admins to set up a different email relay for digests.

During the setup process, the dialog could add Do you want to set up a different SMTP server for digests? and the user could use the same SMTP relay if they wished.

Rationale

For large forums with a lot of digest activity, it would be good to have an option to relay these digest emails via a different SMTP relay than the one used for the key tasks, like password recovery, login and registration.

For now, we have turned off all digests. We did see a way to limit this the seen in last X number of days setting. The default, when I was checking it out today, was 365 days. For some reason, our migrated server queued up over 300K messages.

Discussion

It’s not a huge issue, but it would be good, I think, to separate digest vs. mission critical emails; because even if the queuing priority is higher for mission critical emails, if the SMTP relay blocks because of an excessive number of digests, the mission critical email will be blocked as well.

In addition, some forums may be experiencing a similar situation and not be aware of why their SMTP mail is not working; when in fact it was blocked for the reason mentioned.

Thank you for your thoughtful consideration.

TangentialDuck

[Stephen]

Please note that using the free Gmail to send any email from Discourse isn’t supported. It’s again Google’s terms of service. Using Gmail on this way borders on an unsupported-install.

It’s hard to see how this feature request makes sense to implement, had you used a supported email mechanism the incident couldn’t have occurred.

[neounix (Dark Matter)]

@Stephen

Please note that using the free Gmail isn’t supported. It’s again Google’s terms of service.

It’s hard to see how this feature request makes sense to implement, had you used a supported email mechanism the incident couldn’t have occurred.

That is not true.

We have a Google domain and free email has been supported for many years and is supported now.

Try to cut me some slack after running a forum to 20 years… We did not just hatch out of an egg yesterday, @Stephen

[Stephen]

It absolutely is, communities who ignore our email guide run into this all the time.

Google doesn’t allow any transactional email to be sent from Gmail accounts, and we have seen communities lose accounts permanently when trying to do so. Paid GSuite accounts also have hard limits on transactional email per day. You aren’t the first person to be in this position and we can’t assist you with engineering your way around the terms implemented by Google.

[neounix (Dark Matter)]

I just logged into our GSuite account and checked the TOS and what you are saying is not true:

https://policies.google.com/terms

@Stephen, you are too quick to pull the trigger. I did not ask for a feature to “get around Google’s TOS” and so please refrain from putting words in my posts.

You are too quick to pull the trigger and falsely accuse people @Steven so maybe let others reply to my requests who are not so trigger happy?

[Stephen]

There’s some inconsistency in what you’re saying here:

@neounix

The issue was based on the fact that we use GMAIL as our mail relay, and (free) GMAIL set limits on this sort of SMTP relaying and so GMAIL shut us down for the day.

Gmail isn’t G-Suite. Paid G-Suite accounts have the following limits:

Limit type	Limit
Messages per day
Daily sending limit*	2,000 (500 for trial accounts)

Free Gmail accounts always have the 500 email limit.

@neounix

https://policies.google.com/terms

That’s the Google TOS, it has nothing to do with the above. There’s definitely a difference between ‘it worked until now’ and being allowed to do so. We’ve seen users attempt this over the past seven years and it has never worked out:

Some more reading on the topic

New setup - errors when trying to send emails through gmail - #2 by pfaffman
Office 365 SMTP settings - #2 by codinghorror
https://meta.discourse.org/t/setup-smtp-in-discourse/79173/2
Anyone using gmail for SMTP? - #8 by sam
Gmail SMTP Relay Setup not working - #2 by justin
Can I use gmail smtp? - #2 by fefrei
POP3 polling error - #4 by pfaffman
Sidekiq queue too large - Google email provider problems - #2 by codinghorror

There’s an email setup guide with a list of recommended providers. You don’t have to follow it, but we can’t help you use Gmail or GSuite in ways that weren’t intended.

[neounix (Dark Matter)]

@Stephen

Limit type	Limit
Messages per day
Daily sending limit*	2,000 (500 for trial accounts)

Free Gmail accounts always have the 500 email limit.

Yes, I knew all of that before I posted @Steven

You really should refrain from under-estimating the knowledge of someone who has been on line since before there was a mosaic browser, in my view, and be careful of how you reply. Don’t take the fun of Discourse and tech in general by making accusatory statements toward others.

First you told me “there is NO free GMAIL” (which I knew was wrong) and you accused me of some malicious activity, then you went and did your homework and found out that GSuite has a 2,500 a day limit, which I have known for a long time since I have managed this GSuite account for years.

You should apologize when you pull the trigger so quickly, start accusing people of bad things and are wrong on the details.

It’s not fun, really, to post a simple feature request and have to reply to this negative energy.

[pfaffman (Jay Pfaffman)]

@neounix

The issue was based on the fact that we use GMAIL as our mail relay, and (free) GMAIL set limits on this sort of SMTP relaying and so GMAIL shut us down for the day.

This suggests that you’re using Gmail and not gsuite, which is against the rules. Lots of people, topically people who don’t know what system administration is, try to use Gmail and it is a violation of the terms of service. Telling people that is a good idea, and not mean or rude.

But you’re using gsuite, which is totally fine. That was impossible to tell from your original post. So that’s why it seems like you’re being treated so unfairly.

Except Gsuite doesn’t actually work on a forum with much traffic, as you describe. You are asking for a new feature so that Discourse will somehow work with a mail service that limits you to 2500 messages per day.

It’s possible, but won’t be very easy, to create a plugin to do that. I suspect that it would be several days work for someone familiar with discourse development.

The solution is to use a mail service that will deliver the amount of mail that you need to send.

[Stephen]

Just to follow up, this topic is a duplicate. The request for multiple SMTP services has been requested previously:

Multiple transactional email providers with auto-switch between them

Mandrill gives 12k free emails per month. Sendgrid gives another 12k free emails per month. … and there are more of such providers. It would be cool to sign up with a few and enter the corresponding SMTP authorization details of all of them in a config file and set a quota, i.e: Mandrill: username, password, 11900 (first 11900 emails) Sendgrid: username, password, 11900 (next 11900 emails) ProviderX: username, password (all the other emails) For a growing community, would save at least $10…

The use case here is a little different, but the problem arose from a digest misconfiguration. Unix.com has 138062 users, a limit of 2000 emails per day even for mission critical stuff like password resets would only account for 1.8% of those users being able to interact on a daily basis.

edit: corrected from 2500 to 2000, to reflect the actual limit

[neounix (Dark Matter)]

@pfaffman

But you’re using gsuite, which is totally fine. That was impossible to tell from your original post. So that’s why it seems like you’re being treated so unfairly.

That is why @pfaffman that it is best for all on line to not be fast on the trigger to make accusatory statements towards others regarding tech and/or their motives. Normally, we should ask questions before we pull trigger and start firing our gun, LOL.

Yes, I said GMAIL instead of GSUITE, but when I created this domain decades ago, there was no GSUITE and it was just called GMAIL. Moreover, the reason I did not use any precision, because my feature request has ZERO do to with GMAIL. That is a completely tangential discussion.

If I wanted (or you wanted), we could go to any server, as many here can do, and type apt install postfix or apt install sendmail and in 15 minutes or less we could run our own SMTP relay up and running.

Please let’s not change the subject of moving the heavy traffic from a digest process to a GMail v. GSuite v. Blah Blah discussion. apt install postfix it’s trivial.

The issue is one of reliability. It is basic 101 stuff.

Running mission critical email on the same SMTP relay as digests is not how I would like to run things, so I asked for this feature request.

Let’s keep this, please, on target regarding what I am requesting, since I do happen to know what I am talking about when it comes to building mission critical apps.

Here it is again:

I do not want my mission critical email on the same SMTP relay as digest email and I think this is a good idea and makes the system more robust.

Let’s move off the very tangential discussion about GMAIL or GSuite or MonkeyMail… blah blah. I am sorry I even mentioned it because the email server is not germane to my feature request.

Slow down. Be Kind to Others… If anyone posts something, and you do not understand something or feel confused, then ask; don’t shoot at people @Steven

I can build an SMTP relay in 10 minutes. We all can. apt install postfix done… If I want to use GSuite or postfix or donkey kong mail, that is up to me, not others.As I said…

The Main Point … (again)

It is more reliable to have mission critical email on a different server than on a digest channel. This is just basic stuff for a very busy server.

apt install postfix creates an SMTP relay. I am requesting a feature that permits mission critical email (password resets, registration email, email login) be on a different channel than digests for reliability and performance reasons.

[neounix (Dark Matter)]

@Stephen

The use case here is a little different, but the problem arose from a digest misconfiguration. Unix.com has 138062 users, a limit of 2500 emails per day even for mission critical stuff like password resets would only account for 1.8% of those users being able to interact on a daily basis.

@Steven

That is not the point. That is tangential

Side Note: Actually, we used to have over 500K users, but I pruned them

I am talking about having mission critical email on a different channel (SMTP relay) than digest traffic.

Surely, this simple concept and it is very easy understand the concept of why two channels are better?

This discussion is spinning way off the original idea behind my feature request.

Sorry I even posted and asked…

This discussion subsequent to my original post has been very far off target, IMHO.

[Justin_Vega]

@neounix

You really should refrain from under-estimating the knowledge of someone who has been on line since before there was a mosaic browser

You shouldn’t underestimate any of the people who are trying to help you. Nobody’s against you.

[neounix (Dark Matter)]

This is my last post on this topic.

On the subject of GSuite (which is not the point at all of my feature request)

• The maximum number of messages a user can send in a 24-hour period is 10,000. However, this can vary, depending on the number of user licenses in your G Suite account.
• A registered G Suite user can’t relay messages to more than 10,000 unique recipients in a 24-hour period.

Ref: Route outgoing SMTP relay messages through Google - Google Workspace Admin Help

However, this is totally off topic, for me; because even if GSuite permitted 1000M+ emails a day via SMTP relay, I would still like a different SMTP relay channel for mission critical email versus digest email.

This is the point.

Thank you. Please consider this in a future upgrade. I think it’s relatively easy to add this.

This feature request is not about personality and nor it is about the merits or details of email providers

My feature request is about reliability and keeping mission critical email out of the digest channel.

Over and out…

[pfaffman (Jay Pfaffman)]

@neounix

I do not want my mission critical email on the same SMTP relay as digest email and I think this is a good idea and makes the system more robust.

The people who develop discourse have mail systems that work. They aren’t going to add a feature for people who want to use multiple unreliable mail systems. You can develop a plugin if you want. It will be difficult to develop and will be troublesome to maintain.

It may be easy to install postfix, but it’s much harder to run a functioning mail server now than it was when I ported Sendmail and uucp to Linux. If running a mail server were easy you would have a working mail server and not want two.

@neounix

I think it’s relatively easy to add this.

That’s not what I think, but you might know a lot more about discourse plugin development than I do.

[itsbhanusharma (Bhanu Sharma)]

One simple workaround is to just “disable” digest emails globally if that causes so many issues. The best suggestions may already have been made, I can only suggest use an email provider that doesn’t impose any limits. e.g. mailgun.
This way, everyone is going to be happy (unless you have constrains that limit you to using a particular email provider for some reason)

[Stephen]

@pfaffman

This suggests that you’re using Gmail and not gsuite, which is against the rules. Lots of people, topically people who don’t know what system administration is, try to use Gmail and it is a violation of the terms of service. Telling people that is a good idea, and not mean or rude.

I agree, there’s a lot of inconsistency in what @neounix initially posted, and lots of details change in the subsequent responses.

@neounix

The issue was based on the fact that we use GMAIL as our mail relay, and (free) GMAIL set limits on this sort of SMTP relaying and so GMAIL shut us down for the day.

Emphasis mine in the above, the free account restrictions have already been documented elsewhere in this topic. Grandfathered ‘free’ gsuite accounts have these limits too. If this is a paid for GSuite account, the comment above is misleading and explains my response.

@neounix

We have a Google domain and free email has been supported for many years and is supported now.

Again you aren’t clear as to which service you’re using here, if it is the legacy Google Apps on the free tier then you’re limited to 500 recipients per day per user, the same as the free Gmail product. If you’re using a paid GSuite account then you want to be using the Google SMTP relay service, the limits are 10k/day per user, better but still not great, especially with over 130,000 users who need to request a password reset. It’s great to hear that you culled some users in the migration, although I’m not sure how that’s really material.

I can understand that you’re frustrated here, your testing didn’t catch the digests that would be queued. It caused a period of effective unavailability for anyone trying to reset passwords and get back into their accounts.

A couple of times in the above you’ve suggested that I put words in your mouth, but from what I can see the responses are in-line with the information you provided. Apologies if you don’t agree, but re-reading over the posts it’s still unclear exactly which product you’re using.

Incidentally I’m @Stephen, not @Steven - you’re tagging and notifying a totally different user.

[codinghorror (Jeff Atwood)]

@neounix

I think it’s relatively easy to add this.

If so, then feel free to fund it by posting in marketplace with a budget. There are also some really good suggestions above for reducing email volume, I suggest taking advantage of them.

[neounix (Dark Matter)]

For now, we ended up just turning off digests completely; and here is the updated back story:

We also have a staging server, and because Discourse enables weekly digests (with a 365 day window) as a default, out-of-the-box when installed, that staging server also kicked off a digest last weekend (which we were not expecting), LOL

So, before knowing this would happen (or might happen)… I tried to do a backup on the staging server and the system refused to do it… giving an error. I forgot the exact error message in the admin console, but I recall there was a clue pointing to the mail queue.

With that clue, now looking at sidekiq - and sure enough there were over 300K digest messages in the queue from the default digest configuration of ‘on’ and ‘last 365 days’

So, I cleared the mail queue from the command line, went to the backup admin panel again, and the backup worked fine.

Because the Discourse email system is built on sidekiq, this must be the reason why setting up a different channels (different email servers) for mission critical and digest might be tricky. I see it is not so simple like I originally thought (just set up two email servers in environments).

Here is the funny part… LOL

At first I thought it would good idea to disable digests by default OOTB instead of having them enabled with a 365 day login window; but then I recall the error was all mine; and that was not a good suggestion to post to meta, LOL.

When Discourse was installed, it set all the migrated users (from our old vB3 forum), by default, to last_seen_at around 50 years ago.

I went into the database and manually changed all users to last seen 10 days ago! LOL. I had no idea at the time about the digest configuration; and thought it was a migration error to have all the users last seen 50 year ago after migration. Haha… I was wrong. There is a good reason for this.

Discourse kicked off a massive weekly digest process that overwhelmed sidekiq on both our production and staging servers; because I did the “last seen 10 days ago” DB change hack on the staging server, and exported to production. This was the mistake which caused this issue.

Since most people do not go into postgres and hack around like me and do things like:

UPDATE users SET last_seen_at "today minus 10 days"

… on a legacy migration with many users…

they will not have this kind of massive digest issue.

LOL

I apologize for creating all this tension because of my UPDATE hack to the users table.

Still, I think it is reasonable to have the mission critical email and the digest email on two different email servers, but looking at sidekiq it’s not yet clear to me if there is a simple way to do it, as I have no experience (yet) with sidekiq

However, I can advise migrators, if you are migrating a forum to discourse (which is a great idea), leave the default last_seen_at in users to the default

[Stephen]

I would usually recommend putting Mailhog between a staging instance and the outside world. It’s excellent in this type of scenario.

[pfaffman (Jay Pfaffman)]

I’m going to add DISCOURSE_DISABLE _DIGEST _EMAILS to my staging instances. But with disable emails set to non staff this isn’t a big issue.