There isn’t an option to order categories by the number of posts or number of posts per month. If you only have a few categories you could do that manually by selecting the fixed category positions setting an then reordering your categories periodically based on posts per month. That’s probably not an ideal solution for you though.