That’s more tricky. Because the button has no unique class, there’s no way to hide it specifically – but the following code hides the fourth button, which just happens to be the invite button in the current Discourse version 
#topic-footer-main-buttons button:nth-child(4) {
display: none;
}