I’ve just merged a change to help with this… some more details here Upcoming change for the review queue: "Yes" now always means there's a problem
It’s being released as a beta upcoming change called Consistent review queue answers, which provides a little adjustment time to opt out if necessary.