HTML to Markdown and rebake

AI-generated summary

Mike_Gehrhardt is seeking help to convert 4-5 million HTML-formatted posts, migrated from another system, into Markdown for rebaking in Discourse. The HTML contains elements like <div>, <br />, and [quote] tags but lacks <p> tags, causing crawling issues and soft404 errors in Google Search Console. Sam mentions that Discourse has a built-in HTML-to-Markdown converter, though it’s not perfect, and suggests posting a job offer on the Marketplace channel. pfaffman offers to help with the conversion and directs Mike to contact Jay@literatecomputing.com. louquillio praises the converter and asks for guidance on extracting it for use outside Discourse, noting its reliability compared to other parsers like Aaron Swartz’s html2text. Mike_Gehrhardt, sam, pfaffman, louquillio

I need your help on a problem. We have a lot of topics stored as HTML in the database (raw_data) but it’s HTML that is “migrated” from an other system. It was done before we brought the website and would’ve never ever done it this way. What we want to achieve is to convert the HTML-String containing <div>, <link>, <br />, <span>, <blockquote>,<small> but no <p> and something that is no HTML like [quote][/quote] into Markdown and than rebake the posts, to get them into discourse style HTML so that they where optimized by discourse (e. g. crawler view). At the moment the plain old HTML content is used (cooked_method=2) wich leads to many crawling issues and soft404 in Google Search Console.
We have to do this for about 4-5Mio posts, so it will be very expensive job.

Any ideas?

Best, Mike

We have a built in HTML → Markdown convertor library, it is not perfect but it does the job for the quote function.

You could pass all the posts through that I guess, but what you are describing here, to me, sounds like a large amount of custom work. I would recommend reaching out the the community on Marketplace and putting a $$$ value on the job.

That’s the kind of thing that I do. You can email Jay@literatecomputing.com.

Actually, I like your html2markdown parser very much and would like to put it to use outside of Discourse, for my daily work. Any tips on extracting it into a textarea dingus?

There’s no shortage of html2markdown parsers. Heck, Aaron Swartz wrote one.

The difference is that I trust yours to do what I want — no more, no less.

Thanks.

LQ