Found the problem, which was me.. 
Going through everything once more, of course I noticed that in “Chat allowed groups” not all the groups were added. On our site, we’re pretty strict on who can chat or not and I forgot I set up only specific “Chat allowed groups” 
Problem solved and thanks for the feedback!