I am not seeing the problem in the link you provided?
What webcrawler agent does this software use? Are we correctly identifying it as a crawler? What happens when you set your user-agent to the same user-agent that crawler uses, and browse your site?
I would start there.