Correct, yes, for the crawler view I have opted to outputting just the top menu. Can鈥檛 really know from the Rails app what a theme will output in this route.
I am thinking about the suggestions here @nolo, will make some time shortly to test out some changes. Please 
with me.