Troubleshooting twitter cards while search engines disallowed

(Tobias Eigen) #1

Hi everyone - I am struggling a bit to troubleshoot twitter cards on links to my forum and understand what to do to fix it. There are some interrelated features that I am no clear on how they play together. Any guidance much appreciated.

The card validator at indicates “ERROR: Fetching the page failed because it’s denied by robots.txt.” for topic URLs on my forum. But links to topics on meta work fine. I have twitter logins enabled, though I use SSO. I also have search engine indexing disabled.

(Sam Saffron) #2

This is by design, you have banned all crawlers and bots from visiting your site per:

(Tobias Eigen) #3

yes, that’s what I assumed. I’d be very appreciative for a pointer on how to change robots.txt to allow twitter cards and facebook opengraph? Thanks!

(Sam Saffron) #4

We plan to allow more flexibility around robots.txt, but for the time being all you can do is write a custom plugin.

(Jeff Atwood) #5

May I ask why you disabled all indexing from web spiders?

(Tobias Eigen) #6

It was a compromise with shy colleagues in my organization back at the beginning when we first started our community. Maybe it’s time to revisit the decision now that we’re seeking to start promoting discussions on twitter and facebook. I’ll check.