I think you could create another WITH query to get the users who belong to a group, and then do an inner join to the in_group table in your final query. This should exclude any records that don’t have a user_id in the in_group table
-- returns user_ids for a named group
WITH in_group AS (
SELECT
gu.user_id
FROM group_users gu
JOIN
groups g
ON g.id = gu.group_id
WHERE g.name = :group_name
)
If you want another query that excludes the members of a group, you could do something like this
not_in_group AS (
SELECT
u.id
FROM users u
WHERE u.id NOT IN (SELECT * FROM in_group)
)