I was thinking something like
DELETE FROM topic_users
USING topic_users tu
LEFT JOIN users u ON u.id = tu.user_id
WHERE u.id IS NULL
AND topic_users.id = tu.id
EDIT: it’s pushed now 
https://github.com/discourse/discourse/commit/aebb15337b2ecfa8611b7dd7616eb3528dc75e3c
If I’m reading the query right that looks like it would be the safest in terms of not doing any wrong deletions.
And better than my MAX which assumes NULL and may in fact miss some NULL
I don’t imagine leaving any bogus rows would have much effect on performance.
The only possible problem that I can think of might be if a member has set notification level for something, the corresponding topic would be reset to Normal
And I don’t know for certain if that could happen, though as the bogus gets added last I think it might be possible.
Fantastic work @Mittineague and @fefrei tracking this down. Gold 
for both of you!