Editing tags on tags page extremely slow

Probably better that you figured this out now, than once your site was live.

Can you not use tag groups and enable particular groups against categories?