What's the maximum number of categories we can have?

UI considerations aside, I’m curious what happens when you get up into the 1000+ categories range. Would responses just take a little bit longer on average, or are there catastrophic consequences? Is this something that can be solved by throwing better hardware at the problem?

Like the OP, I’m working on a project that could easily have 1000s of categories. It uses group based permissions heavily and an individual user may only be able to see a handful (this is why I can’t use tags). So I’m not worried as much about the UI being overwhelming, but would like to know what the bottleneck is at that higher range.