Thanks Blake - distinct does the trick:
User.joins(:group_users).where.not(group_users: {group_id: 41}).distinct.count
=> 2572
But something about my .not is off as it’s counting all users, not just the non-members of the group. The number should be 1823 instead of 2572.
users = User.joins(:group_users).where(group_users: {group_id: 41}).distinct.count
=> 749