Reply-by-Email was working, now broken

support
1

Our Reply-by-Email worked for two years, then stopped working 17 days ago for no apparent reason. I’ve searched the staff action logs for that time period and there were no relevant configuration changes. no upgrades, and no new plugins. What could have broken?

No received emails after Sep 30:

image
image1174×495 64.9 KB

They’re not being rejected either:

image
image1397×489 63.7 KB

Nothing obviously related in the error logs, but there is this:

aws-sdk-core-3.112.1/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'

aws-sdk-s3-1.96.1/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'

aws-sdk-s3-1.96.1/lib/aws-sdk-s3/plugins/dualstack.rb:36:in `call'

aws-sdk-s3-1.96.1/lib/aws-sdk-s3/plugins/accelerate.rb:50:in `call'

aws-sdk-core-3.112.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'

aws-sdk-core-3.112.1/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'

aws-sdk-core-3.112.1/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'

aws-sdk-core-3.112.1/lib/seahorse/client/plugins/request_callback.rb:71:in `call'

aws-sdk-core-3.112.1/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'

aws-sdk-core-3.112.1/lib/seahorse/client/plugins/response_target.rb:24:in `call'

aws-sdk-core-3.112.1/lib/seahorse/client/request.rb:72:in `send_request'

aws-sdk-s3-1.96.1/lib/aws-sdk-s3/client.rb:1274:in `copy_object'

/var/www/discourse/lib/backup_restore/s3_backup_store.rb:61:in `block in vacate_legacy_prefix'

/var/www/discourse/lib/backup_restore/s3_backup_store.rb:60:in `each'

/var/www/discourse/lib/backup_restore/s3_backup_store.rb:60:in `vacate_legacy_prefix'

/var/www/discourse/app/jobs/onceoff/vacate_legacy_prefix_backups.rb:7:in `execute_onceoff'

/var/www/discourse/app/jobs/onceoff/onceoff.rb:25:in `execute'

/var/www/discourse/app/jobs/base.rb:232:in `block (2 levels) in perform'

rails_multisite-3.1.0/lib/rails_multisite/connection_management.rb:80:in `with_connection'

/var/www/discourse/app/jobs/base.rb:221:in `block in perform'

/var/www/discourse/app/jobs/base.rb:217:in `each'

/var/www/discourse/app/jobs/base.rb:217:in `perform'

sidekiq-6.2.2/lib/sidekiq/processor.rb:196:in `execute_job'

sidekiq-6.2.2/lib/sidekiq/processor.rb:164:in `block (2 levels) in process'

sidekiq-6.2.2/lib/sidekiq/middleware/chain.rb:138:in `block in invoke'

/var/www/discourse/lib/sidekiq/pausable.rb:138:in `call'

sidekiq-6.2.2/lib/sidekiq/middleware/chain.rb:140:in `block in invoke'

sidekiq-6.2.2/lib/sidekiq/middleware/chain.rb:143:in `invoke'

sidekiq-6.2.2/lib/sidekiq/processor.rb:163:in `block in process'

sidekiq-6.2.2/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'

sidekiq-6.2.2/lib/sidekiq/job_retry.rb:112:in `local'

sidekiq-6.2.2/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'

sidekiq-6.2.2/lib/sidekiq.rb:38:in `block in <module:Sidekiq>'

sidekiq-6.2.2/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'

sidekiq-6.2.2/lib/sidekiq/processor.rb:257:in `stats'

sidekiq-6.2.2/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'

sidekiq-6.2.2/lib/sidekiq/job_logger.rb:13:in `call'

sidekiq-6.2.2/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'

sidekiq-6.2.2/lib/sidekiq/job_retry.rb:79:in `global'

sidekiq-6.2.2/lib/sidekiq/processor.rb:124:in `block in dispatch'

sidekiq-6.2.2/lib/sidekiq/logger.rb:11:in `with'

sidekiq-6.2.2/lib/sidekiq/job_logger.rb:33:in `prepare'

sidekiq-6.2.2/lib/sidekiq/processor.rb:123:in `dispatch'

sidekiq-6.2.2/lib/sidekiq/processor.rb:162:in `process'

sidekiq-6.2.2/lib/sidekiq/processor.rb:78:in `process_one'

sidekiq-6.2.2/lib/sidekiq/processor.rb:68:in `run'

sidekiq-6.2.2/lib/sidekiq/util.rb:43:in `watchdog'

sidekiq-6.2.2/lib/sidekiq/util.rb:52:in `block in safe_thread'

Using manual polling, not pop3 polling:

image

Senders trying to reply by email do not get any rejection notice or error message.

Sending emails from the forum works fine. Nothing bad at AWS SES (although I don’t think that has anything to do with reply-by-email?).

Nothing bad in discourse-doctor (except it does note non-official plugins, but those did not change in this time period).

1 Like
2

This could merely be a coincidence (and part of me thinks it is), but…

2 Likes
3

It’s quite an affluent group- doubt there are many old devices being used.

1 Like
4

I was thinking that it’s more of a server-side issue, not a client-side issue.

2 Likes
5

I think that you need to rebuild your mail NFLcontainer.

1 Like
6

I’m afraid I’ll need more explanation- there’s no other mention in all of meta about NFL (except for football teams). NFS? No such containers in /var/discourse/containers.

1 Like
7

Sorry. Auto correct.

Rebuild your mail container.

cd /var/discourse 
./launcher rebuild mail-receiver
3 Likes
8

Thanks @pfaffman, that fixed it. Sounds like you’ve seen this before- is there something that should have been done better? In our case, there were repercussions beyond just not receiving user emails:

VERP relies on incoming emails. So we weren’t handling bounces and complaints for a few days, our reputation with SES plummeted, and they suspended our outgoing emails.

9

It’s a result of DST Root CA X3 Expiration (September 2021) - Let's Encrypt. It caught a few people off guard and the issues that it caused for Discourse self-hosters are subtle enough that it got overlooked. And there are still issues that need to be resolved.

I suspect that they stopped being handled October 1.

I never figured out how to get SES working. An emergency stopgap would be to set up outgoing mail with mailgun.

10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.