Found out that you also have to enable log mail processing failures in settings.
log mail processing failures