On your invites page (https://HOSTNAME/my/invited/pending) next to the invite, click the trash can:
https://HOSTNAME/my/invited/pending
So if you want to do it from the API, you’ll see the reverse engineering link above.