I don’t know if theme components can prevent this, but hiding categories with CSS wouldn’t prevent them from loading, potentially leading anyway to the slowness you experience when all your categories are loaded in the sidebar. 
2 Likes