Upgrading to New Math Plugin

I still have the old “Discourse Mathjax” plugin installed. Has anyone successfully converted their posts to display properly with the new plugin? Would you be kind to post a working command line?

The two plugins are not 100% compatible, as in some content will be broken if I just activate the new one.

For example, the new plugin requires double $$ before and after \begin{...}...\end{...} blocks, for example \begin{align}...\end{align} (without double $$) is rendered as math with the old plugin but not with the new one.

To convert those I would scrape all content and do a regexp search-replace to prepend/append the double $$ before/after \begin{..}/\end{..} (but nested such blocks would be a problem). There may be other cases where content will appear broken that I haven’t yet spotted (it’s a 5 years old forum with a lot of maths).

1 Like

Well, I spent more than than I care to admit to convert and fix broken content after upgrading to the new math plugin, so I’m posting here the important commands I used (excuse the ugly hackery, got too lazy). Some manual fixing of content was still necessary, but not that much in our case.

NOTE: This is for inspirational purposes. Recall the “I take no responsibility …” poem. Don’t run this blindly if you don’t understand it.

# replace \\ into \ (which also translates \\\\ into \\)
Post.where('raw ~ ?', "\\\\").each{|p| p.raw=p.raw.gsub(/\\\\/, "\\"); p.save; print "."; $stdout.flush;}; puts "";

# replace {align} with {aligned} and remove all starred versions
Post.where('raw ~ ?', "\\\\begin\\{align").each{|p| p.raw=p.raw.gsub(/(begin|end)\{align.*?\}/, "\\1{aligned}"); p.save; print "."; $stdout.flush;}; puts "";

# replace {array} with {matrix} and remove all starred versions
Post.where('raw ~ ?', "\\\\begin\\{array").each{|p| p.raw=p.raw.gsub(/(begin|end)\{array\*?\}/, "\\1{matrix}"); p.save; print "."; $stdout.flush;}; puts "";

# replace `\begin{equation}` with `$$`
Post.where('raw ~ ?', "\\\\begin\\{equation").each{|p| p.raw=p.raw.gsub(/\s*\\(begin|end)\{equation\*?\}\s*/, "$$"); p.save; print "."; $stdout.flush;}; puts "";

# add new lines before and after `$$`
Post.where('raw ~ ?', "\\$\\$").each{|p| p.raw=p.raw.gsub(/\$\$/,"\n$$\n"); p.save; print "."; $stdout.flush;}; puts "";

# add `$$` before and after `{aligned}` blocks
Post.where('raw ~ ?', "\\\\begin\\{align").each{|p| p.raw=p.raw.gsub(/\s*(\\begin\{align)/m,"\n$$\n\\1").gsub(/(\\end\{align.*?\})\s*/m,"\\1\n$$\n"); p.save; print "."; $stdout.flush;}; puts "";

# ... fix double `$$` in `{aligned}`
Post.where('raw ~ ?', "\\\\begin\\{aligned").each{|p| p.raw=p.raw.gsub(/(\\end\{align.*?\}\n\$\$)\n\$\$/,"\\1").gsub(/\$\$\n(\$\$\n\\begin\{align)/,"\\1"); p.save; print "."; $stdout.flush;}; puts "";

## apply the above two commands to other environments you may have besides {aligned}

# MANUAL FIX RECOMMENDED FOR THE FOLLOWING:
# Posts where `$$` appear in quoted `> ` environment. Only the opening and
# closing `$$` must be prefixed, i.e. `> $$`, the content in between must not
# or else it's rendered as math greater-than signs.
# Run this to identify the posts:
Post.where('raw ~ ?', ">\\s*\\n\\s*\\$\\$").each{|p| puts "https://discourse.domain.com/p/#{p.id}";};

# ... OR, run this to try an automated fix, albeit risky
# (worked fine for us, a few posts needed further manual work):
Post.where('raw ~ ?', ">\\s*\\n\\s*\\$\\$").each{|p| p.raw=p.raw.gsub(/(>\s*)\$\$(.*?)\$\$\s*?\n/m,"\\1> $$\\2> $$\n>"); p.save; print "."; $stdout.flush;}; puts "";

# Cosmetic, optional: replace 3+ consecutive newlines with 2 newlines
Post.where('raw ~ ?', "\\n{3,}").each{|p| p.raw=p.raw.gsub(/\n{3,}/,"\n\n"); p.save; print "."; $stdout.flush;}; puts "";

# Optional: replace 2+ consecutive newlines before and after `$$`
# with single newline
# WARNING: fine in the current version of the plugin, but it may display
# differently in a future version (?). E.g. In actual Latex, an empty line 
# after $$ before text does makes a difference in vertical spacing.
Post.where('raw ~ ?', "\\n\\$\\$\\n{2,}|\\n{2,}\\$\\$").each{|p| p.raw=p.raw.gsub(/\n\$\$\n{2,}/,"\n$$\n").gsub(/\n{2,}\$\$/,"\n$$"); p.save; print "."; $stdout.flush;}; puts "";
2 Likes

@sam since this plugin is now official, could we please make the “Quote” feature also work properly? When selecting text with math and then using the “Quote” button, currently I get all the math stripped yet doubled. The javascript code for the “Quote” feature could recognize the math html tags and add the needed $ and/or $$ into the edit box, instead of stripping them and doubling the math string.

Also, I think it would be nice to bundle a more recent version of Katex in the plugin. The bundled one is 2+ years old 0.11 whereas Katex stable is at 0.15.2 which is richer and has a lot of bugs fixed. I’m aware of the readme but few folks will bother.

Mathjax 3.x would also be nice (much faster than 2.7)

EDIT: There is a much easier upgrade of the bundled Katex that doesn’t require building Katex from sources. Just fetch the tarball, copy files, replace font folder in the css.

wget -O- https://github.com/KaTeX/KaTeX/releases/latest/download/katex.tar.gz | tar -zx
cp katex/fonts/*.woff* /path/to/discourse/plugins/discourse-math/public/katex/fonts/
cp katex/katex.min.* katex/contrib/mhchem.min.js /path/to/discourse/plugins/discourse-math/public/katex/
sed -i "s~url(fonts/~url(/plugins/discourse-math/katex/fonts/~g" /path/to/discourse/plugins/discourse-math/public/katex/katex.min.css
5 Likes

All of this sounds great, any chance you can submit some PRs for this?

3 Likes

Created a PR for updating KaTeX and also updated it to latest release. Not using MathJax myself so I didn’t bother with that one yet. Ultimately Discourse may want to allow linking to a CDN for these (allow the user to specify the URL so they can fix a release version if they wish; default to current host) even if it goes a little against the Discourse ethos, as these libraries are updated very often with fixes and whatnot.

Sadly, no time for a PR for the “Quote” feature fix – too snowed under with work these days, which is why I dropped a message hoping someone else can address it.

4 Likes

Thanks heaps for the PR @nordize merged it in!

5 Likes

I recall this thread contained a lot of messages and discussion - what happened?

I came to add some notes to my latest contribution I made earlier this year, as well as an improvement to allow macros to persist between math expressions with Katex (see here). Am I in the wrong thread?

The posts in this topic are on an auto-timer to clear them down, but I can dig any out if you’d like to move them to a Tips&Tricks style topic?

Why on earth?!? This plugin is important and one of the most popular. The discussion in this topic had many posts of interest. I wrote at least one guide in a post and trusted it will be there to refer to it. I and my colleagues even bookmarked some of the posts too … and now it’s all gone?

This is a very strange decision. Wasn’t one of Discourse purposes to be able to deal with long topics (adaptive loading and all that, etc)? “clear them down” … surely we’re not short on space! In this day and age, information posterity is more important. I don’t really understand such decisions as it’s not 1990 any more, and it basically says that the onus is still on the individual to save information and bend over backwards to share it from local sources, because it may disappear at any time from where it was posted online, which encourages waste, hassle, unnecessary attention and organization, and decreases trust and reliability. Helping community users becomes inefficient and problematic too if information is wiped after 1 month. This defeats quite a few purposes. We were supposed to go in the opposite direction.

It’s not a life and death matter, obviously, I don’t want ot blow things out of proportion, but still … Could you bring it back in full and perhaps reconsider this 1month policy going forward?

Anyway, @sam - I made this PR: katex: enable persistent macros by qnxor · Pull Request #51 · discourse/discourse-math · GitHub which I encourage to merge because Mathjax has that by default, so people switching between Mathjax and Katex would have broken content. The same goes for people upgrading from the old plugin.

1 Like

I could may have put that more delicately. :slightly_smiling_face:

Hopefully (all being well :crossed_fingers:) any useful information/interesting discussion will have been either moved up into the OP or split off into a new topic. There’s a chance this doesn’t always happen, but I’m happy to have a look and check. :+1:

2 Likes

Sure, not enough coffee maybe … but it’s great to discover that stuff one invests time in (for the sake of a community) is flushed down the drain. I guess I should have saved the guide and scripts I wrote for upgrading from the old plugin, as well as the other posts I was interested in, then hope people ask me for help by email. Who knew that posting them on the community forum was going to be the most unsafe option. Yeah, not enough coffee …

None of those happened here as far as I can tell.

Bringing back the topic would be too …? Definitely not enough coffee! :slight_smile:

1 Like

I think I’ve got all the right ones? I’ve moved them out of the auto-timer area and into support, so they should be safe from any deletion. :+1: Any tweaks or changes you want me to make?