Make Discourse play nice with the Wayback Machine

[codinghorror (Jeff Atwood)]

This looks OK to me now, or am I missing something cc @falco @ibnesayeed @anarcat

http://web.archive.org/web/20190202031942/https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579

We’re getting the crawler version which seems to work fine and is a good outcome?