Reply-by-Email was working, now broken

Our Reply-by-Email worked for two years, then stopped working 17 days ago for no apparent reason. I’ve searched the staff action logs for that time period and there were no relevant configuration changes. no upgrades, and no new plugins. What could have broken?


No received emails after Sep 30:


They’re not being rejected either:


Nothing obviously related in the error logs, but there is this:

aws-sdk-core-3.112.1/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'

aws-sdk-s3-1.96.1/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'

aws-sdk-s3-1.96.1/lib/aws-sdk-s3/plugins/dualstack.rb:36:in `call'

aws-sdk-s3-1.96.1/lib/aws-sdk-s3/plugins/accelerate.rb:50:in `call'

aws-sdk-core-3.112.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'

aws-sdk-core-3.112.1/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'

aws-sdk-core-3.112.1/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'

aws-sdk-core-3.112.1/lib/seahorse/client/plugins/request_callback.rb:71:in `call'

aws-sdk-core-3.112.1/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'

aws-sdk-core-3.112.1/lib/seahorse/client/plugins/response_target.rb:24:in `call'

aws-sdk-core-3.112.1/lib/seahorse/client/request.rb:72:in `send_request'

aws-sdk-s3-1.96.1/lib/aws-sdk-s3/client.rb:1274:in `copy_object'

/var/www/discourse/lib/backup_restore/s3_backup_store.rb:61:in `block in vacate_legacy_prefix'

/var/www/discourse/lib/backup_restore/s3_backup_store.rb:60:in `each'

/var/www/discourse/lib/backup_restore/s3_backup_store.rb:60:in `vacate_legacy_prefix'

/var/www/discourse/app/jobs/onceoff/vacate_legacy_prefix_backups.rb:7:in `execute_onceoff'

/var/www/discourse/app/jobs/onceoff/onceoff.rb:25:in `execute'

/var/www/discourse/app/jobs/base.rb:232:in `block (2 levels) in perform'

rails_multisite-3.1.0/lib/rails_multisite/connection_management.rb:80:in `with_connection'

/var/www/discourse/app/jobs/base.rb:221:in `block in perform'

/var/www/discourse/app/jobs/base.rb:217:in `each'

/var/www/discourse/app/jobs/base.rb:217:in `perform'

sidekiq-6.2.2/lib/sidekiq/processor.rb:196:in `execute_job'

sidekiq-6.2.2/lib/sidekiq/processor.rb:164:in `block (2 levels) in process'

sidekiq-6.2.2/lib/sidekiq/middleware/chain.rb:138:in `block in invoke'

/var/www/discourse/lib/sidekiq/pausable.rb:138:in `call'

sidekiq-6.2.2/lib/sidekiq/middleware/chain.rb:140:in `block in invoke'

sidekiq-6.2.2/lib/sidekiq/middleware/chain.rb:143:in `invoke'

sidekiq-6.2.2/lib/sidekiq/processor.rb:163:in `block in process'

sidekiq-6.2.2/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'

sidekiq-6.2.2/lib/sidekiq/job_retry.rb:112:in `local'

sidekiq-6.2.2/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'

sidekiq-6.2.2/lib/sidekiq.rb:38:in `block in <module:Sidekiq>'

sidekiq-6.2.2/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'

sidekiq-6.2.2/lib/sidekiq/processor.rb:257:in `stats'

sidekiq-6.2.2/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'

sidekiq-6.2.2/lib/sidekiq/job_logger.rb:13:in `call'

sidekiq-6.2.2/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'

sidekiq-6.2.2/lib/sidekiq/job_retry.rb:79:in `global'

sidekiq-6.2.2/lib/sidekiq/processor.rb:124:in `block in dispatch'

sidekiq-6.2.2/lib/sidekiq/logger.rb:11:in `with'

sidekiq-6.2.2/lib/sidekiq/job_logger.rb:33:in `prepare'

sidekiq-6.2.2/lib/sidekiq/processor.rb:123:in `dispatch'

sidekiq-6.2.2/lib/sidekiq/processor.rb:162:in `process'

sidekiq-6.2.2/lib/sidekiq/processor.rb:78:in `process_one'

sidekiq-6.2.2/lib/sidekiq/processor.rb:68:in `run'

sidekiq-6.2.2/lib/sidekiq/util.rb:43:in `watchdog'

sidekiq-6.2.2/lib/sidekiq/util.rb:52:in `block in safe_thread'

Using manual polling, not pop3 polling:

image


Senders trying to reply by email do not get any rejection notice or error message.

Sending emails from the forum works fine. Nothing bad at AWS SES (although I don’t think that has anything to do with reply-by-email?).

Nothing bad in discourse-doctor (except it does note non-official plugins, but those did not change in this time period).

1 Like

This could merely be a coincidence (and part of me thinks it is), but…

2 Likes

It’s quite an affluent group- doubt there are many old devices being used.

1 Like

I was thinking that it’s more of a server-side issue, not a client-side issue.

2 Likes

I think that you need to rebuild your mail NFLcontainer.

1 Like

I’m afraid I’ll need more explanation- there’s no other mention in all of meta about NFL (except for football teams). NFS? No such containers in /var/discourse/containers.

1 Like

Sorry. Auto correct.

Rebuild your mail container.

cd /var/discourse 
./launcher rebuild mail-receiver 

3 Likes

Thanks @pfaffman, that fixed it. Sounds like you’ve seen this before- is there something that should have been done better? In our case, there were repercussions beyond just not receiving user emails:

VERP relies on incoming emails. So we weren’t handling bounces and complaints for a few days, our reputation with SES plummeted, and they suspended our outgoing emails.

It’s a result of DST Root CA X3 Expiration (September 2021) - Let's Encrypt. It caught a few people off guard and the issues that it caused for Discourse self-hosters are subtle enough that it got overlooked. And there are still issues that need to be resolved.

I suspect that they stopped being handled October 1.

I never figured out how to get SES working. An emergency stopgap would be to set up outgoing mail with mailgun.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.