Thanks for the help but I’m not comfortable delving into the hosting settings (and someone else hosts it for us).
I spoke to the community and decided to just remove the current Nice Reply badge and make our own that will be granted on the first instance only, plus some extra badges for getting lots of them.
I much prefer single-grant badges over multi-grant ones.