# \[Announce\] Search & replace / batch process Discourse posts
**URL:** https://meta.discourse.org/t/announce-search-replace-batch-process-discourse-posts/34029
**Category:** Extras
**Tags:** rest-api
**Created:** [October 2, 2015, 7:14am UTC](https://meta.discourse.org/t/announce-search-replace-batch-process-discourse-posts/34029 "2015-10-02T07:14:36Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@dandv](https://meta.discourse.org/u/dandv)
#### Post date: [October 2, 2015, 7:14am UTC](https://meta.discourse.org/t/announce-search-replace-batch-process-discourse-posts/34029/1 "2015-10-02T07:14:36Z")
## ==\> [Discourse-post-process](https://github.com/dandv/discourse-post-process) \<==
I’m happy to introduce this simple but powerful Node.js script that uses the [Discourse API](https://meta.discourse.org/t/discourse-api-documentation/22706) to mass-modify Discourse posts. It essentially performs a regexp-based search & replace across all posts in a forum.
After migrating a few thousand posts from MyBB to Discourse, there was various little tweaks that needed to be made (fix BBCode → Markdown conversion, re-attach attachments, fix @username mentions when the usernames have changed etc.) I’m not a Ruby coder, and [writing a simple Ruby script to post-process posts was non-trivial](https://meta.discourse.org/t/batch-processing-to-search-and-replace-in-all-posts/33621/4). Using the Discourse API was much easier.
### Features
- Apply a set of regexp-based transformations to each post, such as replacing [BBCode](https://en.wikipedia.org/wiki/BBCode) tags with Markdown ones, e.g. `[hr]` with `---`
- Mention as the edit reason all transformations that have been applied
- Warn about patterns that may cause rendering problems, such as unconverted `[list]`s or [newlines in nested quotes](https://meta.discourse.org/t/weird-parsing-rule-for-nested-quote-preceded-by-newline/33679).
- Warn about potentially missing attachments
- Detect whether `[color=]` tags have been used (install the [bbcode-color plugin](https://github.com/discourse/discourse-bbcode-color/) to support colors)
- Fix [quotes that have been incorrectly imported from MyBB (or other forums)](https://meta.discourse.org/t/mybb-import-doesnt-process-on-quoted-posts-pid-and-date/27412)
I’ve tested the script manually on ~500+ posts in the [Quantified Self Forum / Zeo thread](http://discourse.quantifiedselfforum.com/zeo) and am quite satisfied with it.
---
### Author:  [@featheredtoast](https://meta.discourse.org/u/featheredtoast)
#### Post date: [November 14, 2015, 11:20pm UTC](https://meta.discourse.org/t/announce-search-replace-batch-process-discourse-posts/34029/2 "2015-11-14T23:20:53Z")
Exactly what I was looking for, thank you!
I’m looking to mass search/replace phpbb [youtube] tags with actual youtube links from the phpbb importer, and this is a great first step.
PR for adding support for [youtube] tags.
[https://github.com/dandv/discourse-post-process/pull/3](https://github.com/dandv/discourse-post-process/pull/3)
I’ve thrown a PR your way for dry-run option, too.
[https://github.com/dandv/discourse-post-process/pull/2](https://github.com/dandv/discourse-post-process/pull/2)
---
### Author:  [@anon36895752](https://meta.discourse.org/u/anon36895752)
#### Post date: [November 24, 2015, 10:37pm UTC](https://meta.discourse.org/t/announce-search-replace-batch-process-discourse-posts/34029/3 "2015-11-24T22:37:06Z")
This is awesome!
Can you tell me, how tailored is this to MYBB. I’m considering running it on PHPBB imported data. And i’m just wondering if there are any gotcha’s i should watch out for?
---
### Author:  [@erlend\_sh](https://meta.discourse.org/u/erlend_sh)
#### Post date: [November 25, 2015, 11:55am UTC](https://meta.discourse.org/t/announce-search-replace-batch-process-discourse-posts/34029/4 "2015-11-25T11:55:48Z")
Seems @anon36895752 ran into some scaling issues, explained in a different topic:
> [@Migrate a phpBB3 forum to Discourse](https://meta.discourse.org/t/migrate-a-phpbb3-forum-to-discourse/30810/26):
>
> Interesting approach, i gave it a shot. It took me a while to figure out that i was crushing my API with requests and hitting rate limits. After editing some nginx config to temporarily remove the rate limits its running again now. But i don’t think it will be done anytime in 2017 at this rate. I have waaaaay too many posts to be doing this abstracted through a web API.
@dandv maybe the description should mention some ballpark numbers on maximum recommended forum size to use this method on.
---
### Author:  [@dandv](https://meta.discourse.org/u/dandv)
#### Post date: [December 4, 2015, 3:38am UTC](https://meta.discourse.org/t/announce-search-replace-batch-process-discourse-posts/34029/5 "2015-12-04T03:38:15Z")
I’ve run my script in batches just to make sure it wouldn’t break too many posts at once, and without any rate limits added or disabled. I think the largest batch was 1000 posts or so, but I haven’t run into Discourse limits.
---
### Author:  [@anon36484860](https://meta.discourse.org/u/anon36484860)
#### Post date: [November 24, 2017, 4:37pm UTC](https://meta.discourse.org/t/announce-search-replace-batch-process-discourse-posts/34029/6 "2017-11-24T16:37:24Z")
Is this one still working with latest Discourse version?
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [November 24, 2017, 5:13pm UTC](https://meta.discourse.org/t/announce-search-replace-batch-process-discourse-posts/34029/7 "2017-11-24T17:13:40Z")
I would imagine so.
See also [Replace a string in all posts](https://meta.discourse.org/t/replace-a-string-in-all-posts/48729)