# Make Discourse play nice with the Wayback Machine
**URL:** https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579
**Category:** Feature
**Created:** [2015 年 10 月 16 日午前 2:39 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579 "2015-10-16T02:39:46Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@dandv](https://meta.discourse.org/u/dandv)
#### Post date: [2015 年 10 月 16 日午前 2:39 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/1 "2015-10-16T02:39:46Z")
I’ve noticed that when archiving Discourse topics using [archive.org](http://archive.org), the CSS is mangled. Here is an [example](https://web.archive.org/web/20151014020610/https://meta.discourse.org/t/core-dump-when-running-the-mybb-import-script/34332/9):
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [2015 年 10 月 16 日午前 3:23 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/2 "2015-10-16T03:23:11Z")
Cannot be resolved, as wayback does not understand and cannot archive pure JavaScript sites at all. Either you serve it plain crawler HTML, or it will show nothing.
You may want to raise this with them, if they can make their crawler archive pure JavaScript sites, but it requires an extremely advanced crawler that runs headless browsers. Pure text `curl` retrieval is no longer sufficient, they have to retrieve the JS _and execute it_.
---
### Author:  [@riking](https://meta.discourse.org/u/riking)
#### Post date: [2015 年 10 月 16 日午前 3:50 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/3 "2015-10-16T03:50:45Z")
> [@codinghorror](#):
>
> You may want to raise this with them, if they can make their crawler archive pure JavaScript sites, but it requires an extremely advanced crawler that runs headless browsers. Pure text curl retrieval is no longer sufficient, they have to retrieve the JS and execute it.
orrrrr we could build a topic view that looks like this 😄 (from my [AMPproject.org](http://AMPproject.org) experiment branch)
This layout is both extremely easy to crawl, and works with both wide and narrow screens.
However, I seem to have forgotten the topic’s category.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [2015 年 10 月 16 日午前 4:35 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/4 "2015-10-16T04:35:28Z")
Very strongly opposed to adding _another_ renderer, one that has to be kept in sync with the primary JavaScript renderer, plus the crawler 1996 HTML renderer we already have.
---
### Author:  [@riking](https://meta.discourse.org/u/riking)
#### Post date: [2015 年 10 月 16 日午前 6:27 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/5 "2015-10-16T06:27:20Z")
> [@codinghorror](#):
>
> plus the crawler 1996 HTML renderer we already have
The way I have it implemented, this replaces the crawling view outright on topics.
---
### Author:  [@FSanches](https://meta.discourse.org/u/FSanches)
#### Post date: [2016 年 11 月 25 日午後 6:58 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/11 "2016-11-25T18:58:48Z")
May I suggest we focus back on the original issue here?
There’s people willing to use the Internet Archive Wayback Machine (myself included) in order to keep copies of Discourse threads for posterity and right now that seems to be impossible to achieve with the current implementation (of both Discourse and Wayback).
What are the things that can be done to the Discourse codebase that can improve this? And what are the things that can be done on the Archive’s crawling system that could help as well?
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [2016 年 11 月 25 日午後 7:00 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/12 "2016-11-25T19:00:50Z")
I believe the CSS should look a lot better with @falco’s latest changes in that area. Can you quickly check using the [archive.org](http://archive.org) tools @falco?
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [2016 年 11 月 25 日午後 7:05 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/13 "2016-11-25T19:05:07Z")
I think all we need is detecting [archive.org](http://archive.org) UA as a crawler, let me take a look.
---
### Author:  [@FSanches](https://meta.discourse.org/u/FSanches)
#### Post date: [2016 年 11 月 25 日午後 7:06 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/14 "2016-11-25T19:06:29Z")
I actually experienced a server error while attempting to save a Discourse page on the wayback machine today:
[https://web.archive.org/save/\_embed//t/recuperando-dados-de-disquetes-antigos/35](https://web.archive.org/save/_embed//t/recuperando-dados-de-disquetes-antigos/35)
This was the original URL:
> **[Engenharia Reversa do Mega Drive Portátil da TecToy](https://forum.fiozera.com.br/t/engenharia-reversa-do-mega-drive-portatil-da-tectoy/38)**
>
> Recentemente, por conta das notícias sobre o novo Mega Drive que a TecToy vai lançar no Brasil, acabei descobrindo a existência de um produto anterior, lançado na década passada: o MDPLAY. Trata-se de uma versão portátil do Mega Drive que, além de...
And I was trying to save it this way:
[https://web.archive.org/save/https://forum.fiozera.com.br/t/engenharia-reversa-do-mega-drive-portatil-da-tectoy/38](https://web.archive.org/save/https://forum.fiozera.com.br/t/engenharia-reversa-do-mega-drive-portatil-da-tectoy/38)
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [2016 年 11 月 25 日午後 7:58 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/15 "2016-11-25T19:58:45Z")
OK, this is kinda strange.
They are using the User Agent of the user who asked to save the page to download the HTML, and after that they try to show the page with a bunch of injected JS.
This topic when I click on the save button using Google Bot User Agent: [Make Discourse play nice with the Wayback Machine - feature - Discourse Meta](https://web.archive.org/web/20161125200006/https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579)
Sent an email to [archive.org](http://archive.org), let’s see.
---
### Author:  [@ibnesayeed](https://meta.discourse.org/u/ibnesayeed)
#### Post date: [2016 年 12 月 2 日午後 10:32 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/16 "2016-12-02T22:32:57Z")
> [@Falco](#):
>
> I think all we need is detecting [archive.org](http://archive.org) UA as a crawler, let me take a look.
That’s not enough actually, because there are [many other web archives](http://mementoweb.org/depot/) in town and new web archives come to life every now and then.
---
### Author:  [@ibnesayeed](https://meta.discourse.org/u/ibnesayeed)
#### Post date: [2016 年 12 月 2 日午後 10:56 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/17 "2016-12-02T22:56:36Z")
> [@codinghorror](#):
>
> Cannot be resolved, as wayback does not understand and cannot archive pure JavaScript sites at all. Either you serve it plain crawler HTML, or it will show nothing.
>
> You may want to raise this with them, if they can make their crawler archive pure JavaScript sites, but it requires an extremely advanced crawler that runs headless browsers. Pure text curl retrieval is no longer sufficient, they have to retrieve the JS and execute it.
There are many efforts of headless browser based archiving including [http://archive.is/](http://archive.is/) which is an on-demand single page archiving system. It renders the page using PhantomJS and then archives the rendered DOM plus necessary assets. However, doing it on a massive scale (not just for on-demand pages) takes a lot of time, because PhantomJS or any other renderer is orders of magnitude slower than traditional vanilla crawlers such as [Heritrix](https://github.com/internetarchive/heritrix3) that is used my Internet Archive and many other web archives.
Here is a relevant research work on the topic [Scripts in a Frame: A Two-Tiered Approach for Archiving Deferred Representations](https://www.slideshare.net/JustinBrunelle/scripts-in-a-frame-a-twotiered-approach-for-archiving-deferred-representations). Below is a blog post summarizing the research work and related resources.
> **[2016-04-15: How I learned not to work full-time and get a PhD](https://ws-dl.blogspot.com/2016/04/2016-04-15-how-i-learned-not-to-work.html)**
>
> The Web Science and Digital Libraries Research Group at Old Dominion University.
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [2016 年 12 月 2 日午後 11:38 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/18 "2016-12-02T23:38:38Z")
> [@ibnesayeed](#):
>
> > [@Falco](#):
> >
> > I think all we need is detecting [archive.org](http://archive.org) UA as a crawler, let me take a look.
>
> That’s not enough actually, because there are [many other web archives](http://mementoweb.org/depot/) in town and new web archives come to life every now and then.
Actually that didn’t work because they aren’t respecting user agent.
Discourse is already crawler friendly, you just need to ask the crawler version.
---
### Author:  [@grahamperrin](https://meta.discourse.org/u/grahamperrin)
#### Post date: [2017 年 5 月 14 日午前 11:50 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/19 "2017-05-14T11:50:07Z")
Attempts by me to save this topic result in:
- [Firefox 53.0](https://web.archive.org/web/20170514114324/https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579)
- [Chromium 57.0.2987.110](http://web.archive.org/web/20170514121205/https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579)
I can **not** describe the procedure, or the end result, as _playing nicely_.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [2017 年 5 月 14 日午後 2:28 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/20 "2017-05-14T14:28:47Z")
We should follow up and see if we can improve this somehow @falco, perhaps in the next few weeks or early after 1.9 begins beta.
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [2017 年 5 月 15 日午後 9:31 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/21 "2017-05-15T21:31:12Z")
> [@grahamperrin](#):
>
> I can not describe the procedure, or the end result, as playing nicely.
The problem, as described above, is that Wayback machine is not being nice, because they hijack their user-agent string, making it impossible for use to serve a non-js crawler version for then.
I tried their e-mail but got 0 responses.
---
### Author:  [@ibnesayeed](https://meta.discourse.org/u/ibnesayeed)
#### Post date: [2017 年 5 月 15 日午後 9:52 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/22 "2017-05-15T21:52:45Z")
@Falco, the Wayback machine is the archival replay system. The crawler used at Internet Archive is Heritrix. That said, would you mind telling me the exact user-agent string you are seeing from them? I might be able to approach some people at Internet Archive on personal channels and see what’s going on.
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [2017 年 5 月 15 日午後 9:56 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/23 "2017-05-15T21:56:05Z")
> [@ibnesayeed](#):
>
> I might be able to approach some people at Internet Archive on personal channels and see what’s going on.
Wow, this is amazing! 🎉
> [@ibnesayeed](#):
>
> @Falco, the Wayback machine is the archival replay system. The crawler used at Internet Archive is Heritrix. That said, would you mind telling me the exact user-agent string you are seeing from them?
It’s like I said here:
> [@Make Discourse play nice with the Wayback Machine](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/15):
>
> OK, this is kinda strange. They are using the User Agent of the user who asked to save the page to download the HTML, and after that they try to show the page with a bunch of injected JS. This topic when I click on the save button using Google Bot User Agent: [Make Discourse play nice with the Wayback Machine - feature - Discourse Meta](https://web.archive.org/web/20161125200006/https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579) Sent an email to [archive.org](http://archive.org), let’s see.
For example, if you are using Firefox 52 and click on the save, they will use your user-agent (Firefox 52 user agent) when asking the page.
If they add something that can distinguish this requests, we can serve the correct page.
---
### Author:  [@ibnesayeed](https://meta.discourse.org/u/ibnesayeed)
#### Post date: [2017 年 5 月 15 日午後 10:12 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/24 "2017-05-15T22:12:41Z")
I will try to talk to some friends at the Internet Archive later this week. Next month I will be meeting with many web archive folks (including Internet Archive) at an International Internet Preservation Consortium conference. I will raise this issue there.
---
### Author:  [@grahamperrin](https://meta.discourse.org/u/grahamperrin)
#### Post date: [2017 年 5 月 16 日午前 5:35 UTC](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579/25 "2017-05-16T05:35:24Z")
Thanks, and to clarify: I did _not_ suggest that Discourse did not play nicely. Please see earlier versions of my post, keyword:
- `/print`
– I meant to say that _the print workaround_ is not nice. I do not recommend it.
[Next page](https://meta.discourse.org/t/make-discourse-play-nice-with-the-wayback-machine/34579.md?page=2)