Wow, that was tedious but I figured out what happened!
The good news is blocking is indeed working as expected. I checked the server logs for the users who created accounts and they just used the normal flow. I reviewed the code and tested it myself and the blocking is working.
So how did they get in?
Well that rule you found is actually an automatic roll up. Our server runs a background job that identifies many banned ips from the same subnet. If it reaches a threshold
min_ban_entries_for_roll_up, it will automatically ban the subnet.
I found a log that this exact operation happened on May 25th. So Discourse was smart enough to ban them from coming back, but it wasn’t clear that the rule was a rolled up rule that was added after they signed up.