The default robots.txt file in Discourse contains a rule Disallow: /u, which prevents Twitter from fetching images for Twitter cards because the image path /uploads starts with a u. HaPe discovered the issue and found a temporary fix by adding a slash at the end of the rule, making it Disallow: /u/.
The problem was previously encountered, and a fix was removed in a commit. HaPe created a pull request to fix the default robots.txt file, which eviltrout has merged.
devporto and others have encountered the same issue, and the quick fix provided by HaPe has been found to work.
By default, the robots.txt file contains the following entry:
User-agent: *
# ...
Disallow: /u
This configuration blocks the image for Twitter cards. Checking the card of various forums on Twitter’s validator does not display the logo and shows the following warning:
The image URL forum.[…].org/uploads/[…].png specified by the ‘twitter:image’ metatag may be restricted by the site’s robots.txt file, which will prevent Twitter from fetching it.
The problem is that the path /uploads also starts with a u and is therefore matched by the robots.txt rule.
I know that the robots.txt file can be overwritten but I think it would be good to change the default. Unfortunately, I don’t understand the code well enough to create a pull request.
When overwriting, keep in mind that Twitter checks the robots.txt file less often than the actual html of the page, so it takes a few hours to get picked up.
Yes, at least it makes the Twitter cards work. I hope that it does not have unintended side effects.
You can open the setting “allow index in robots txt” and click “override robots.txt”. Then look up the line Disallow: /u and place a slash in the end, so it becomes Disallow: /u/. It will take Twitter a few hours to pick up the change.
Looks like this was a problem (see this thread) in the past and then got re-introduced.
In this commit, the robots rules with slash got removed because it was thought they are no longer needed. Later, in this commit they were re-introduced but without the slash in the end.
