That sounds like it could be a bug, though it seems improbable that there could be a bug in the mail-sending code.
Have a look at Troubleshooting email on a new Discourse install for other stuff to try. (like how to do a test without running discourse-doctor).
Ah. Here. “YES” is not a valid value for smtp_validation:
Must be plain, login, or cram_md5.
It’s in app/services/email_settings_validator.rb if someone wants to find it on github.