It’s not a transformation of the header per se - when reply-by-email is enabled, DIscourse sends out email with the envelope-from set to the reply_by_email_address address pattern and also sets reply-to: if the email is replyable.
For example, here on meta it’s:
notification_email: notifications@meta.discoursemail.com
reply_by_email_address: incoming+%{reply_key}@meta.discoursemail.com
which turns into:
Return-Path: <incoming+verp-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@meta.discoursemail.com>
From: Person <notifications@meta.discoursemail.com>
Reply-To: Person <incoming+xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@meta.discoursemail.com>
The notification email goes into the From header, and reply_by_email_address goes into envelope-from. We’ve configured our incoming mail such that anything addressed to …@meta.discoursemail.com gets delivered to meta, which makes this work.
But if your email server does not permit you to use this envelope-from, you’ll get the error you’re reporting.
This error is most definitely coming from the mail server. Discourse might be reporting it to you, but the mail server is generating it.