Users are not receiving the Nice Reply badge when category is not public

No, they need to have “everyone” can “See”. (and not the “Do not grant badges for posts in this category” box but I assume that isn’t your problem)