Found the problem, which was me.. 
Going through everything once more, of course I noticed that in āChat allowed groupsā not all the groups were added. On our site, weāre pretty strict on who can chat or not and I forgot I set up only specific āChat allowed groupsā 
Problem solved and thanks for the feedback!