Bug? Can't undo a vote in a non-voting category

By looking at the users’ profiles and going to the Votes section of their activity tab. EG, I just moved a topic from Feature Requests to General ('Pronunciation' feature for taxonomic names - Feature Requests - iNaturalist Community Forum) but the two users who originally voted for it still have it on their profile, eg Profile - agnes_clamfanger - iNaturalist Community Forum