# A basic Discourse archival tool **URL:** https://meta.discourse.org/t/a-basic-discourse-archival-tool/62614 **Category:** Development **Created:** [May 12, 2017, 8:20pm UTC](https://meta.discourse.org/t/a-basic-discourse-archival-tool/62614 "2017-05-12T20:20:12Z") **Posts on this page:** 1 **Showing post:** 54
### Author: ![kyle315a](https://avatars.discourse-cdn.com/v4/letter/k/a587f6/32.png) [@kyle315a](https://meta.discourse.org/u/kyle315a) #### Post date: [July 31, 2022, 10:09pm UTC](https://meta.discourse.org/t/a-basic-discourse-archival-tool/62614/54 "2022-07-31T22:09:22Z")
A Discourse forum that I use is being taken offline in a couple weeks, so I set out to archive the site. I did a lot of research, trail and error, and I found a simple solution with HTTrack. Here’s everything I learned. **Archive a Discourse site with HTTrack** For Windows users, the best solution appears to HTTrack. This worked great and it archived the site to HTML files. All categories, threads, and posts were archived including all pages with relative navigation links. A basic tutorial on HTTrack is [here](https://fossbytes.com/download-website-offline-access-using-httrack-use-website-without-internet/). I left the settings on default with the following custom settings. - Web Addresses: - `https://forums.gearboxsoftware.com/c/homeworld/` - `https://forums.gearboxsoftware.com/c/homeworld-dok/` - Scan Rules: - `-gearboxsoftware.com/* -forums.gearboxsoftware.com/* +forums.gearboxsoftware.com/c/homeworld/* +forums.gearboxsoftware.com/c/homeworld-dok/* +forums.gearboxsoftware.com/t/* +forums.gearboxsoftware.com/user_avatar/* +sea2.discourse-cdn.com/*` - Browser ID (aka User Agent): - `Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)` Note: There’s a CSS issue preventing category links from working, however that can easily be fixed as described below. **CSS Issue** When viewing Category pages as googlebot, the thread links don’t work. An example is `[here](https://web.archive.org/web/20220731051419/https://forums.gearboxsoftware.com/c/homeworld/57)`. This makes navigation impossible on category pages in HTTrack, `archive.org` and google catch. This appears to be a Discourse issue in a CSS file. To fix the links, simply block/delete the following CSS file: - `stylesheets/desktop_theme_10_1965d1d398092f2d9f956b36e08b127e00f53b70.css?__ws=forums.gearboxsoftware.com` @codinghorror - Can you guys address this? **Challenges** I ran into the following challenges and eventually overcame them after much trial and error. - Discourse pages are dynamically generated with JavaScript. This makes for poor results with most archive/crawler tools. - Most threads only load with the first ~20? posts, the rest of the posts don’t appear until you scroll down. Pressing Ctrl+P loads a `/print` page with all posts visible. Users are limited to printing 5 pages an hour with print mode, but this limit can be increased by a Discourse site admin. - Adrelanos noted that multi-page threads weren’t being archived properly by HTTrack, however I suspect this issue was due to his HTTrack settings, as I did not have this issue. - Saving a page to PDF won’t include any collapsed details sections. - Pages can be loaded in basic HTML by adding `?_escaped_fragment_` to the end of a URL, but this trick only works for threads not categories. The above challenges aren’t a concern once you learn that all Discourse pages/content can be rendered properly as HTML for crawlers. To do this, you must change your crawler / browser’s user agent to googlebot to get the HTML version of pages. **`Archive.org`** If you use the “Save Page Now” feature on `web.archive.org`, it will archive the javascript version of Discourse with poor results. `Archive.org` uses the user agent of the person requesting the archive. So you must change your user agent to googlebot. You can get a Chrome extension called “User-Agent Switcher for Chrome”. In the options add: - Name: Googlebot - String: `Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)` - Group: Chrome - Indicator Flag: 1 **Alternative Archive Tools** Many tools are listed here: [Archive an old forum "in place" to start a new Discourse forum](https://meta.discourse.org/t/archive-an-old-forum-in-place-to-start-a-new-discourse-forum/13433) I also briefly tested GUI tools like Cyotek WebCopy, A1 Website Download, and WAIL. Command line tools include mcmcclur’s tool and wget. A tuturial on wget is `[here](https://letswp.justifiedgrid.com/download-discourse-forum-wget/)`. However for Windows users, the best solution appears to HTTrack. Note: Since I’m a new user, I’m limited to two links in a post. Hence I turned some links into preformatted text. --- _[View the full topic](https://meta.discourse.org/t/a-basic-discourse-archival-tool/62614)._