Any updates on the best way to create a HTML archive of a static site?

AI-generated summary

In a discussion titled “Any updates on the best way to create a HTML archive of a static site?”, user justin_gordon sought a method to create browseable HTML files for retiring their Discourse forum. While exploring options like the Discourse Archive and the ArchiveDiscourse script, they encountered issues with broken image links and internal URLs pointing to the decommissioned server justin_gordon. pfaffman suggested using wget with specific flags to mirror the site, though it required obtaining the session cookie _t pfaffman. Another user, 翔_贺, shared a Ruby snippet for serving static files, while supermathie noted that a custom exporter could be built from backups. Finally, manuel recommended discourse-static-archive as a promising solution that handles topics, categories, and search, with pfaffman providing a refined wget command using a cookie extraction script for those preferring that method.

Update!

This might be the answer:
https://github.com/jamesob/discourse-archive

I looked at:

Improving Discourse static HTML archive.

It’s old.

I’m going to retire https://forum.talksurf.com/.

Yes, I’m going to archive a backup.

But what if I just want some browseable HTML files?

Should I just run ArchiveDiscourse/archive-discourse.py at master · kitsandkats/ArchiveDiscourse · GitHub ?

Or is there something better?

Thanks in advance!

CC: @pfaffman

Aloha,

Justin

1 Like

Would something like Wayback Machine be similar?

This worked. I had to make a slight code update.

https://github.com/kitsandkats/ArchiveDiscourse/pull/4

3 Likes

But not much older than your Discourse version!

I have had some luck mirroring sites with wget. Something like

wget --mirror --page-requisites --convert-links --adjust-extension --compression=auto --reject-regex "/search" --no-if-modified-since --no-check-certificate --execute robots=off --random-wait --wait=1 --user-agent="Googlebot/2.1 (+http://www.google.com/bot.html)" --no-cookies --header "Cookie: _t=$COOKIE" https://forum.talksurf.com/

But you need to get the cookie named _t

Send me an email and I’ll see what I can do.

1 Like

I’ve been doing this recently, and this is how I did it。

 def serve
    file_path = File.expand_path(
      params[:path]+"."+params[:format],
      File.join(File.dirname(__FILE__), "../../public")
    )
    if File.file?(file_path)
      send_file file_path, type: "text/html", disposition: "inline"
    else
      render plain: "404 Not Found", status: 404
    end
  end

Just to let you know, this does not pull the images with new URLs. The photos will still point to your server (which is about to be decommissioned!).

Jay kindly sent me the dump, and I compared it to mine.

His technique works better in the sense that it saves the images.

However, his internal links don’t point to the articles, but rather to the decommissioned site. However, the articles can be found with images.

It would be a “nice to have” if Discourse supported a static export. :smile:.

2 Likes

The good thing is that you have all the data, so one could be written that exported the data directly from a backup if anyone had the inclination to do so.

But we’re not likely to write one :wink:

1 Like

It shouldn’t be too hard to fix the internal links, add it looks like they just need .html added.

I thought that the --convert-links would fix those links…

I’d still love to see a good archive generator for Discourse.

Just let Claude have a run at it, sharing here for reference and keeping this feature in the loop:

https://github.com/nolosb/discourse-static-archive

Its a basic setup so far: topics, categories, tags and users. Search also works. Uploaded an archive of a local demo instance to a github page:

3 Likes

That is probably better than this.

Paste this to assign the cookie to COOKIE and then paste the cookies copied to clipboard by the Cooke Chrome extension.


COOKIE=$(cat |jq -r '.[ ]|select(.name == "\_t")|.value')

enter the URL here:

then paste:


wget --mirror --page-requisites --convert-links --adjust-extension --compression=auto --reject-regex "/search" --no-if-modified-since --no-check-certificate --execute robots=off --random-wait --wait=1 --user-agent="Googlebot/2.1 (+http://www.google.com/bot.html)" --no-cookies --header "Cookie: \_t=$COOKIE" =FORUMURL=

4 Likes