Script framework to rearrange topics and categories

In order to script rearranging a large Discourse with working with my site leadership, I wrote a simple framework for packaging up a bunch of ruby scriptlets to be driven from a YAML configuration file. This way, I can restore a backup on a staging site, run my script, get feedback, change a few lines of YAML, restore the backup, run the script with new configuration, and repeat until satisfied.

I have almost 600 lines of configuration for my site, and doing that by manual manipulation through the UI would just not happen. Not even once, let alone many time to get it right. I know this because last time I proposed making sweeping changes I quite literally gave up trying. By contrast, with this script I can now complete the whole cycle in just a few minutes per iteration, even though the site has about half a million posts and over 100 categories. This lets me get fast feedback from my site leadership, and I’ll be prepared to migrate my site quickly.

More detailed documentation is in the README file in the repository with the code:

:warning: This script does no error checking. It’s pretty much crazy to run it on your live site. It is intended to run on a staging site, validate the result, and then be taken live. As the author I still intend to run it this way. If you run it directly on a live site you get to keep all the pieces when it breaks. :warning:

From the documentation, a configuration file might look like this:

---
- describe:
    context: Old Name
    category: 7
    name: New Name
    description: New description of category
    slug: new-slug
- movePosts:
    context: move only faq posts from the Support category to the Documentation category
    source: 3 # Support category ID
    target: 6 # Documentation category ID
    withTag: faq
    hide: false # do not hide the Support category when done
- movePosts:
    context: consolidate How-To category into documentation with how-to tag
    source: 8 # How-To category ID
    target: 6 # Documentation category ID
    addTag: how-to
    hide: true # hide the old How-To category, visible only to Admin

The progress output while it’s running might then look like this.

==========
Move hidden categories out of the way so they don't clutter admin view
setHiddenCategory: {:category=>11}
==========
Rename Old Name to New Name
describe: {:category=>7, :name=>"New Name", :description=>"New description of category", :slug=>"new-slug"}
==========
move only faq posts from the Support category to the Documentation category
movePosts: {:source=>3, :withTag=>"faq", :target=>6}
==========

To use this, put your YAML file in /var/discourse/shared/app/tmp/rearrange.yaml and then:

cd /var/discourse
./launcher enter app
git clone https://github.com/johnsonm/discourse-site-rearranger.git script/discourse-site-rearranger
ruby script/discourse-site-rearranger/rearrange.rb /shared/tmp/rearrange.yaml

However, it is reasonably likely that this script won’t do quite everything you need. It’s really a framework that makes it very easy to drop in new actions that automate more aspects of a scripted site modification with just a few lines of code. All you need to do is define one method with a few lines of code, and you’ll be able to invoke it properly from the YAML file.

Have you been thinking about improving the organization of your site, but shrinking from using the UI, or wondering how to trust that you’ll be able to replicate your testing on a staging site for making changes live? Take a look and tell me what you think!

5 Likes

I’m using this script to create a test site, as a clone of my main site. High visual fidelity is part of the point but could make it easy to accidentally post on the wrong site while reviewing changes, and have those accidental posts blown away the next time I refresh the test site.

I first made my test site read-only when I asked for public feedback. But login is disabled while the site is in read-only mode, so they couldn’t log in when I asked them to.

I’ve added a new publicCategoriesReadonly action that makes anonymously-visible categories writable only by :admins and :readonly by :everyone in order to allow people to log in and poke around, but help avoid accidentally posting on the wrong site. Now the site as a whole is not in read-only mode, but the public categories are.

It is possible to reference categories as hashtags by slugs. This script allows moving content of categories together and changing slugs, which would make previous posts no longer link after a rebake. (Before a rebake, permalink redirects would make the links work.)

I have now updated the script to track tags before and after the migration and remap references to changed categories in posts.

I have added a migrateRetortToReactions action to the framework for those of us who have had Retort installed and want to move to a supported plugin.

2 Likes

Wow! That’s great news. It would be great if someone could make it a rake task and submit it to the reactions plugin.

I’ll have a look at it soon.

1 Like

Having it as a rake task in the plugin sounds smart to me.

I’ve signed the CDCK CLA so any copyright interest on my part in the resulting work won’t get in the way, and it can be put under the same license as the plugin itself.

Also I could imagine that I got something wrong; if you find anything wrong with it, I’d love to know about it because I plan to run that script on my own site sometime in the next few weeks. :smiley:

On a larger scale, if you find this framework useful for your own work supporting Discourse installations but would like more error handling, I’d take PRs. I merely want to make sure that any PR to it is from someone who has signed the CLA and agrees that any useful pieces of it can be incorporated into official Discourse, which I know is true for you…

2 Likes

I discovered in the Retort topic that @angus has a branch with UI-driven conversion, which I had completely missed.

My script dives into internals which could conceivably change someday, and once I’ve done my conversions with this script I won’t notice if those internals change. I first tried to do it “the right way” but discovered that the Reactions plugin doesn’t expose an interface that enables this. @angus takes the longer view:

@pfaffman if you make a rake task, you might want to put it in a PR that also make the changes to the interfaces so that, for example, created_by can be passed in, as well as silent, rather than using the “back door” that I did. At that point, the rake task would be there for command-line migrations, and it would simultaneously unlock @angus doing a UI-driven migration.