[Announce] Search & replace / batch process Discourse posts

api

(Dan Dascalescu) #1

==> Discourse-post-process <==

I’m happy to introduce this simple but powerful Node.js script that uses the Discourse API to mass-modify Discourse posts. It essentially performs a regexp-based search & replace across all posts in a forum.

After migrating a few thousand posts from MyBB to Discourse, there was various little tweaks that needed to be made (fix BBCode -> Markdown conversion, re-attach attachments, fix @username mentions when the usernames have changed etc.) I’m not a Ruby coder, and writing a simple Ruby script to post-process posts was non-trivial. Using the Discourse API was much easier.

Features

I’ve tested the script manually on ~500+ posts in the Quantified Self Forum / Zeo thread and am quite satisfied with it.


Batch processing to search and replace in all posts
Importing from phpBB3
Manually added files in uploads folder are giving 404
Converting keywords in posts on the fly
Is there any api to import discourse details using ruby
Scroll through full user list?
(Jeff Wong) #2

Exactly what I was looking for, thank you!

I’m looking to mass search/replace phpbb [youtube] tags with actual youtube links from the phpbb importer, and this is a great first step.

PR for adding support for [youtube] tags.

I’ve thrown a PR your way for dry-run option, too.


(Adrian D'atri Guiran) #3

This is awesome!

Can you tell me, how tailored is this to MYBB. I’m considering running it on PHPBB imported data. And i’m just wondering if there are any gotcha’s i should watch out for?


(Erlend Sogge Heggen) #4

Seems @Adrian_D_Atri_Guiran ran into some scaling issues, explained in a different topic:

@dandv maybe the description should mention some ballpark numbers on maximum recommended forum size to use this method on.


(Dan Dascalescu) #5

I’ve run my script in batches just to make sure it wouldn’t break too many posts at once, and without any rate limits added or disabled. I think the largest batch was 1000 posts or so, but I haven’t run into Discourse limits.


#6

Is this one still working with latest Discourse version?


(Jay Pfaffman) #7

I would imagine so.

See also Replace a string in all posts