Discourse CommonMark migration plans 🎊 🎈

[sam]

I have commenced our migration to markdown-it , all rejoice

markdown-it is a compliant CommonMark JavaScript library that is super fast, well maintained and straight forward to extend.

I already have it working on a branch, but so many of our extensions are broken that I do not want to subject everyone on meta to it quite yet.

The plan

Phase 1 - get this running on meta (underway)

The first phase of the migration to the new engine is getting us ready to turn markdown-it on at meta.

To make this feasible and minimize branching I designed it so we can merge in my changes early. Turning on markdown-it requires a hidden site setting and the actual work I am doing is totally segregated from current support.

This means I should be able to merge in my changes next week into the latest branch.

I plan to enable markdown-it at meta once I ported all our extensions and plugins running at meta to the new engine. This is a slow process as a lot of code needs to be re-written to support the new engine.

Expected time frame: 4-8 weeks

Phase 2 (stabalization)

The second phase of the project will be me submitting pull requests to various plugins and stabalizing the new engine. I anticipate running on meta for at least 2-4 weeks before “flicking the switch” and making the new engine default on.

During this phase I will make sure to close all the #markdown-it-review bugs. Not touching these bugs till this is live on meta.

Expected time frame: 2-4 weeks

Phase 3 (gutting out old engine)

The third phase of this process will be removing the old engine from core and adding “deprecation” so plugins that we did not catch can be notified of the change.

Expected time frame: 1-2 weeks

Phase 4 (removing deprecations)

Finally, in the release after next we will remove all the deprecation so we don’t have to carry around dead code.

[sam]

Also …

AN ENORMOUS thank you goes out to @Vitaly and team for creating the markdown.it engine!!!

[Vitaly]

Sam, there are CM spec related things, worth to resolve:

1. quotes syntax (with refer to original)
2. tables syntax
3. may be… math formulas embedding syntax/language

[gjtorikian]

I came here by way of the official CommonMark forum, and am morbidly curious: isn’t Discourse a Ruby stack? Why not use the Ruby implementation (GitHub - gjtorikian/commonmarker: Ruby wrapper for libcmark (CommonMark parser)) instead of a new JS one? Will the rendering be client-side or server-side?

As well, we’ve implemented an extensions spec at GitHub.

[sam]

We have always shipped with a js pipeline, cause its a lot more friendly network wise. We have a live preview going.[quote=“gjtorikian, post:4, topic:64234”]
Will the rendering be client-side or server-side?
[/quote]We support both, and radical extensions that need

[sam]

I may pull this out to another topic, but I feel that I need to some stuff.

Inline multiline quote[quote]that
spans
lines

and blocks

[/quote]Is a terrible format should not be supported

Instead

Inline multiline quote

[quote]
that
spans
lines

and blocks

[/quote]

Is a terrible format should not be supported

At least the new format can not be as unreadable.

[sam]

A big milestone today, I merged all my work-in-progress into core.

We still ship the old markdown engine as default, but now have a special flag you can set to turn on commonmark.

I plan to turn this on early next week on meta.

[sam]

Update

2. Phase 2 has been initiated, we have commenced testing on meta.discourse: CommonMark testing started here!

[sam]

Short of more documentation and more samples I feel like we are mostly done now.

We even integrated the “core plugin” test suite into our build process to catch regression going forward.

Old engine is gone, many plugins have been migrated to new engine.

Documentation is out: Developer's guide to Markdown extensions

I think I am closing this one up! will continue applying more over the next few weeks.

[sam]