I think a JOIN with the group_users table ON user_id WHERE group_id should do it.
group_users
id serial primary key
group_id integer fkey groups
user_id integer fkey users
created_at timestamp
updated_at timestamp
owner boolean default false
notification_level integer default 2