My phpBB Migration Journey (postgresql)

migration
21

soft deleted posts get imported as regular posts (and are visible). this is clearly a problem because i think users soft delete by default, and deleted posts can contain sensitive info.

relevant columns include; post_delete_user, post_delete_time, post_visibility, post_delete_reason

seems like post_visibility is the only relevant one—the other column values don’t get cleared if a post is soft deleted and then restored. normal = 1, invisible = 2

my fix is to just discard soft deleted posts because who cares.

run this on your source db before migration:

DELETE FROM phpbb_posts WHERE post_visibility = 2;
1 Like
22

i have a poor man’s ‘continuous migration’ pipeline. it’s a bunch of shell scripts to do tasks like:

  • pull phpbb db, attachments, etc from remote server
  • create/manage discourse instances
  • specific site_settings get applied when a discourse instance is created from a template instance
  • back up/restore them
  • run migration with a specific set of import scripts from a git repo
  • do post-migration tasks

basically this allows me to work on a pre-migration discourse dctemplate instance that i play around with. i continue to tweak discourse settings to my liking.

nightly, i pull from the live, existing phpbb and do a new migration on a new discourse instance (dcstaging) just created from the newest template.

doing this takes hours, of course, so i also have a local phpbb running with a very small db that i use to do tests with and improve the migration scripts. i can do relatively fasts migrations to test on another discourse instance (dcdev).

so daytime test workflow is like

did some work on the template, i like where i’m at:

./dc_template_create.sh dctemplate

oops, i made a mistake on my template. restore it to the previous state:

./dc_template_restore.sh dctemplate

do a quick migration to test some stuff out

# create a new instance dcdev from dctemplate (and apply specific site_settings to dcdev)
./dc_template_restore.sh dcdev dctemplate
./dc_migrate.sh dcdev # runs with migration script repo specific to that instance

do a long migration overnight from the real, live phpbb:

./dc_template_create.sh dctemplate
./dc_template_restore.sh dcstaging dctemplate

./phpbb_pull.sh
./dc_migrate.sh dcstaging
sleep 300
./dc_rake.sh dcstaging # extra post-migration tasks not handled by the importer for (create moderators, detailed tagging, moving some topics around, etc)

if anyone’s interested enough, i could clean it up a little and share it.

2 Likes
23

discourse has tags, so it may be wise to leverage those and get rid of subcategories where it makes sense.

my subcategories each had several pinned/stickied topics though, and when combined into a big category, that is way too many pinned topics. it’s disruptive.

my solution was to go from categories like:

Food
- American
- Chinese
- ...

to this:

Food
- Featured

american, chinese, etc become tags, and all of the pinned topics go in Featured and get unpinned.

# unpin all pinned topics in a category and move them to a subcategory
def move_pinned_topics_to_subcategory(category, subcategory_name)
  subcategory = Category.where(name: subcategory_name).find_by(parent_category_id: category.id)
  topics = Topic.where(category_id: category.id).where.not(pinned_at: nil)
  topics.each do |topic|
    topic.update(pinned_at: nil, pinned_globally: false, pinned_until: nil,
      category_id: subcategory.id) if not topic.title[/About the .+ category/]
  end
  Category.update_stats
end

food_cat = Category.find_by(name: 'Food')
move_pinned_topics_to_subcategory(food_cat, 'Featured')
4 Likes
24

my forum actually also has a previous, dead incarnation (also phpBB) that we have a wget siterip of, so i’m working on an importer for that. parse html files with nokogiri, ingest into intermediary db, and then work with the importer base class to get rows into discourse.

i saw this:

i’m guessing this is an uncommon problem, but if anyone else is facing this issue, maybe i’ll put a little extra polish into it and share it. lmk.

or if someone has already made this and wants to share with me, that’s even better. i’m not that far into the project yet.

2 Likes