Self hosted Reply by Email stopped working after latest update

[Paul_King]

My forum no longer responds to replies sent by email.

Reply by email has previously been working well, but it looks like this functionality stopped working around September 29th

I have no definitive evidence for this timing, as the forum is not very active, but certainly it no longer works now, and the forum logs show no messages received after 29th September.

The rejected emails log also has most recent entry on 29th September. All messages rejected have throwaway addresses and content that look like spam - so this seems to be working as it should.

The bounced emails log is empty / shows ‘no logs found’

Messages are still being sent out by the forum generated by logged in user activity (I am getting them at least) though activity levels are even lower than normal due to above. Nearly all active users prefer email rather than browser based interaction.

Test emailed replies to forum post email notifications sent from either my own Microsoft hosted email address or my Gmail address are not getting bounce warnings. They are simply disappearing without a trace. Nothing appears in the forum email log.

The forum error log shows a couple of warnings (yellow exclamation mark icon) for September 29th ‘Email can not be processed: Email::Receiver::BadDestinationAddress Received…’ which seem innocuous / no different to previous similar logged events. I presume just rejected spam.

On October 1st there is an actual error logged as:

Message

ActionDispatch::Http::MimeNegotiation::InvalidType ("%{#context[‘com.opensymphony.xwork2.dispatcher.httpservletresponse’].addheader(‘cbu0psig’" is not a valid MIME type)
lib/middleware/omniauth_bypass_middleware.rb:71:in call' lib/content_security_policy/middleware.rb:12:in call’
lib/middleware/anonymous_cache.rb:353:in call' config/initializers/100-quiet_logger.rb:23:in call’
config/initializers/100-silence_logger.rb:31:in call' lib/middleware/enforce_hostname.rb:23:in call’
lib/middleware/request_tracker.rb:187:in `call’

Backtrace

actionpack (6.1.4.1) lib/action_dispatch/http/mime_negotiation.rb:31:in rescue in block in content_mime_type' actionpack (6.1.4.1) lib/action_dispatch/http/mime_negotiation.rb:24:in block in content_mime_type’
rack (2.2.3) lib/rack/request.rb:69:in fetch' rack (2.2.3) lib/rack/request.rb:69:in fetch_header’
actionpack (6.1.4.1) lib/action_dispatch/http/mime_negotiation.rb:23:in content_mime_type' actionpack (6.1.4.1) lib/action_dispatch/http/request.rb:269:in media_type’
actionpack (6.1.4.1) lib/action_dispatch/http/request.rb:355:in form_data?' rack (2.2.3) lib/rack/request.rb:445:in POST’
actionpack (6.1.4.1) lib/action_dispatch/http/request.rb:400:in block (2 levels) in POST' actionpack (6.1.4.1) lib/action_dispatch/http/parameters.rb:88:in parse_formatted_parameters’

Env

HTTP HOSTS: nzarchitecture.net.nz

I have no idea if this is relevant though, and no further errors or fatal errors (denoted with light or dark red cross icons) have appeared in the log since then.

Neither of my email addresses appear to be spammy or otherwise blacklisted when tested on www.mail-tester.com, and no issues with communicating with humans from these addresses have been encountered.

The forum uses Mailgun, though I presume this is just for sending bulk emails, and that any issues with the Mailgun account or API key should not affect messages incoming? As it happens there are no obvious issues or error messages with the Mailgun found when I log in to my Mailgun account.

I assume the Mailgun API key must be ok if the forum is still sending mail ok.

No forum settings have been changed since reply by email was working, and I see the ‘reply by email’ setting checkbox is still ticked.

The forum is hosted on Digital Ocean. No DNS settings for the domain have been changed in Digital ocean settings, and forum MX records seem ok/unchanged.

The forum has been updated to 2.8.0 beta 7 since the issue started (presumably rebuilding in the process), but no improvement.

What am I missing here?
What likely went wrong?
How do I get reply by email to start working again?

[RGJ (Richard - Communiteq)]

That error seems unrelated.

In general, it’s easier to test “email in” than it is to test reply-by-mail. Check the manual polling enabled and email in settings, add an email address to the staff category and see if you can send email to that using the same email address you’re using for your forum admin user account.

Then check Admin - Email - Bounced / Received / Rejected to see what is going on.

Is your reply by email address configured correctly?

[Paul_King]

Hi thanks Richard

I can confirm that the manual polling enabled and email in settings are still enabled.

I added my gmail address as a custom address to the staff category, but I can’t see a way to send messages to staff via the forum? All ‘contact us’ links are set up in forum settings text as mailto: links that just go to my personal everyday email address - clicking on one of those just opens my email application, prepopulated with my personal direct email address, meaning the forum would never receive the message.

[RGJ (Richard - Communiteq)]

@Paul_King

I added my gmail address as a custom address to the staff category

No, you should set up something like staff@nzarchitecture.net.nz in the category settings, then use your Gmail to send a message to that address.

[Paul_King]

Ah, ok.

@RGJ

set up something like staff@nzarchitecture.net.nz in the category settings, then use your Gmail to send a message to that address.

I tried exactly this, but nothing has appeared in any of the Bounced / Received / Rejected logs

[IAmGav (Gavin Perch)]

your mail server is set to Digital Ocean.

Do you have a mail server with them ?

[RGJ (Richard - Communiteq)]

That’s the IP of the droplet, which is running the mail-receiver.

nzarchitecture.net.nz. 2727 IN A 159.65.140.176

[IAmGav (Gavin Perch)]

its been changed in the last 5 months

[RGJ (Richard - Communiteq)]

@Paul_King

Reply by email has previously been working well, but it looks like this functionality stopped working around September 29th

I know what is going on. It’s that @#$($($* LetsEncrypt thing again that broke half the internet many things on Sept 30.

Self-hosted mail-receiver update following Let's Encrypt root certificate change

This announcement only affects self-hosters who are running Direct-delivery incoming email for self-hosted sites. Managed hosting customers, and self-hosters using POP3 for incoming mail are not affected. As you may be aware, Let’s Encrypt recently changed their root certificate. The old root certificate expired today, and has caused a number of issues for out-of-date clients across the web. All of our systems at CDCK are up-to-date, and were not affected by today’s expiration. Unfortunately we…

Just rebuild the mail receiver Docker.

[IAmGav (Gavin Perch)]

@RGJ

It’s that @#$($($* LetsEncrypt thing again that broke

hahahahaha, yes. i forgot about that. LOL

[IAmGav (Gavin Perch)]

You got to follow the topic that @RGJ posted.

That should fix your issue.

Self-hosted mail-receiver update following Let's Encrypt root certificate change

This announcement only affects self-hosters who are running Direct-delivery incoming email for self-hosted sites. Managed hosting customers, and self-hosters using POP3 for incoming mail are not affected. As you may be aware, Let’s Encrypt recently changed their root certificate. The old root certificate expired today, and has caused a number of issues for out-of-date clients across the web. All of our systems at CDCK are up-to-date, and were not affected by today’s expiration. Unfortunately we…

[Paul_King]

@RGJ

Just rebuild the mail receiver Docker.

Ah, ok. That sounds promising!
How do I rebuild ‘mail receiver docker’? is this different to the rebuilding of ‘docker manager’ that happens when updating the forum via dashboard?

Do I just follow this? How do I manually update Discourse and Docker image to latest? - howto / admins - Discourse Meta

[IAmGav (Gavin Perch)]

you need to login to the command line side of your site.

It’s not via the forum admin dashboard.

[Paul_King]

Hi, I was able to rebuild mail receiver docker using instructions at that link Direct-delivery incoming email for self-hosted sites - howto / sysadmin - Discourse Meta

Had to upgrade / resize my Digital Ocean droplet to do it, as even after deleting all backups etc stored on host, there was not enough disk space to do a rebuild

After rebuild, I was able to send that message to staff@nzarchitecture.net.nz - and the forum log this time acknowledged it.
However when I try to reply by email to an existing topic of the forum that I have been notified about, while the incoming messages are now acknowledged, none appear on the forum, and all get Mail Delivery Failure warnings in email log.

Incoming messages are not appearing in Bounced Email log, but do all appear in Rejected Email log with warning [Email::Receiver::BadDestinationAddress] - including my own administrator email account which I would hope does not suddenly have a bad destination address

[pfaffman (Jay Pfaffman)]

Have you rebuilt your mail receiver lately?

[Paul_King]

Yes - did that about half an hour ago, which resulted in the post above.
I have just now done another full rebuild, and (touch wood) things seem to be working again

[pfaffman (Jay Pfaffman)]

@Paul_King

have just now done another full rebuild, and (touch wood) things seem to be working again

Could be that force https was not set and the rebuild fixed it.

[Paul_King]

Actually I had just spotted a warning about exactly that in dashboard, and so clicked the handy provided link to appropriate setting and ticked the box.

I had not realised that force https was mandatory for receiving incoming email

Just possible that lack of enforced https has also been causing issues with using facebook login - I have recently been notified by Facebook that my site was in breach of their terms of service and has been suspended. There were no action items in my Facebook apps developer control panel, so I appealed, and the response was that they could not verify the site due to an unspecified error generated by my forum url,

[Paul_King]

@Paul_King

Just possible that lack of enforced https has also been causing issues with using facebook login

It seems ticking the ‘force https’ box has not helped at all with Facebook login. Facebook tech support are still saying the site landing page generates a ‘your connection is not private
NET::ERR_CERT_COMMON_NAME_INVALID’ security warning for them.

The issuer of the certificate, per the error page, is shown as ‘R3’ - which a google search suggests is related to Let’s Encrypt - the same people who’s certificate expiry triggered the need to rebuild Discourse installation.

Is this a coincidence? Does this suggest the latest Discourse build (2.8.0 beta 7) still has a certificate problem? or is this an unrelated issue to do with hosting/Digital Ocean?

[Paul_King]

My own blundering google research has let me to test my url using https://www.whynopadlock.com/, which results led me to this post by a Let’s Encrypt user

Let’s Encrypt recently updated its intermediate certificate from “Let’s Encrypt Authority X3” to “R3”.

If you use a well-behaved ACME client, it would have automatically started using the new intermediate at your last renewal. You shouldn’t have noticed any difference.

In your case, perhaps you have been hardcoding the intermediate certificate. If that’s the case, you’ll need to use the new intermediate, which you can find on Chain of Trust - Let's Encrypt : https://letsencrypt.org/certs/lets-encrypt-r3-cross-signed.pem

Is current version of Discourse perhaps wrongly ‘hardcoding the intermediate certificate’?

Are ‘intermediate certificates’ something that Discourse administrators are now required to manage? And if so, how?

Please let me know if I am now off topic - I am not sure if part of the same issue or not.