Blake, thanks for your reply. Indeed, this is the route I went - locally cached categories.
For people doing similar stuff to what I am doing: since in my case I needed all the categories and subcategories, I also ended up using the site.json API point instead of categories.json (which seems to return only top level categories).