I was referring to the mail.domain.tld A record being set to DNS Mode Only rather than the forum.domain.tld A record, however I realised that I have been misinterpreting how SMTP clients authenticate TLS certificates.
The behaviour I was seeing was an artefact of the default opportunistic method which does not validate the hostname, so my assertion that it validates the hostname of the MX record rather than the target of the MX record was incorrect. It would work in most cases but not if DANE or MTA-STS are used for enforcing TLS identity authentication.