Is forum migration really worth it?

(Jeff Atwood) #1

Continuing the discussion from New posts, read posts, unread posts - I like this:

[quote=“Lee_Ars, post:4, topic:7618”]
The pain of migrating away from phpBB right now outweighs the benefits, since one of the huge positives about the Ars forum is its 14 years’ worth of posts. That was why it took so long to jump off of Infopop to phpBB in the first place—finding a way to migrate the posting history.[/quote]

Humor me for a minute. What exactly would be the downside of 14 years of read-only forum archives at a slightly different URL, and the forum starting over?

I’m curious about this, because at ForumCon this year one of the things I learned is that “all data must be migrated” is the Vietnam of forum software, leading to massive retention of ancient forum versions across the web and a huge black eye for forum software in general. I heard an untold number of horror stories from companies that do nothing but specialize in forum migrations. I could see it in their eyes… I remember… I remember everything…

It’s technically very difficult to migrate everything over, and perhaps even worse, it sociologically leads people to “expect” everything to be the same, since if all their data is there, why doesn’t the forum behave in exactly the same way it did 5 years ago, too?

Best practices for migrating to hosted Discourse from other forum software
"It's too complicated" complain the users. How did you find the user migration experience?
(Lee_Ars) #2

I’ve been pondering an answer to the questions you’re asking, and though I could probably respond at length with pages and pages and pages of babbling, every response I can think of essentially comes down to “but this is the way we’ve always done it.”

phpBB, for all its ludicrous complexity, works well for the scale of the Ars forums—which are bigger than most, but nowhere near as big as some. It’s well-understood from a moderation and security standpoint. The users are used to it and it’s not particularly bad at anything. It’s appropriately performant under load. It integrates with Wordpress for supporting front page comments (and the extra hackery we’ve added on to make comment voting & community moderation work). We’re not particularly looking for a change.

But, coming back to that long tail of data—the users do expect it to be there because it’s always been there. The disadvantage is that from a user’s perspective, migrating to Discourse would introduce a number of changes in the experience that aren’t necessarily quantifiably better, but are definitely different. It’s one thing if the userbase was clamoring for change because the forum software actually sucked—which infopop definitely did, holy crap that shit was ancient—but the Ars masses seem pretty content for now with phpBB, so introducing even a feature-complete Discourse would be disruptive without necessarily being good. Forgoing migration and leaving a static “old forums” archive site would be asking them to give up what many see as a very attractive benefit.

Plus, having that tremendous history has proved valuable more than once. See, for example, this craziness.

(Jorge Castro) #3

Heya Lee, nice to see you here! I am also running into this issue.

I remember when Kurt was driving the forums migration to phpbb and I had some emails with him about the set up because at the time I feel it was kind of ballsy to go with phpbb.

I think Ars is kind of an outlier in this regard, your forums look clean and tight and are very content driven. Out of all the forums I keep an eye on, it’s probably the one which sucks the least so you have probably less to gain than some of the other forums with tons of bling and junk in their layout.

I am having this long tail problem in Ubuntu as well. The idea of not having existing content migrated over seems to be a dealbreaker/non starter for folks. Maybe the solution is something like turning an existing forum into read only and integrate at the search level instead of a full blown import; dunno, seems like a hard problem to solve.

(Lee_Ars) #4

It’s one thing to drive users to change their expectations & behavior when there are clear benefits—see, for example, the McNeel forums transitioning from USENET to web-based forum. The pushback there is expected, but not that many folks can be expected to use newsgroups to contact a company these days. Hell, I’m a huge USENET nerd and from about 1994 through 2000-ish, I used USENET more than the web…but, today, I don’t even have a newsreader installed, and haven’t for years and years. Pulling those guys forward makes lots of sense and there are tangible benefits to the McNeel guys—increased customer engagement is a huge one.

But “modernizing” a forum by bringing it from phpbb to something like Discourse would remove (or at least complicate) a feature that the community truly values, while providing in its place a lot of changes that might be received as “change for change’s sake” rather than beneficial.

(Erlend Sogge Heggen) #5

For, our forum is our lifeblood.

  • Hundreds, maybe thousands of snippets of code reside there. This is far from ideal, but we’ve never had a dedicated web developer to help us build a more suited solution.
  • Likewise, countless threads are excellent QA resources reminiscent of StackOverflow.
  • This is our history. We’ve gone through 4 different forum platforms (phpBB, SMF, BuddyPress, bbPress) yet we can still find our way back to ancient threads that had some significance, like the baby steps of a new codebase.
  • Our forum is a huge part of our search rankings, probably much more important than our blog.

(F. Randall Farmer) #6

@erlend_sh - So, wouldn’t migrating the database break all those inbound links? That’s bad, right?

Some operators are archiving the old threads (read-only), and starting fresh with new software - often the first and final post on an archived topic a link to the new version(s) of that topic - providing for referencing legacy content, formatting and link management without mangling the content itself (or trying to force the new platform to meet outdated requirements - as @Lee_Ars USENET migration post addresses.)

Note that we also had this discussion when Blogs began to emerge as a replacement for many forums - and the general thought then was: Abandon it all - thought that thinking was probably the result of concluding that forums were the wrong format altogether for the discussions at hand.

So, perhaps this question should fork?

At what point is the costs of migration (technical, format-loss, legacy-link loss, etc.) exceed the benefits when migrating to a significantly/functionally different platform?

I’m fascinated that many of the reasons I hear FOR migration can instead be argued to be the reason to NOT migrate the data, but to instead leave it intact and then integrated with the new platform (whatever that is.)

I’m watching the proboards version migration in progress now, and it is a huge mess - outbound image links and internally stored files are breaking everywhere - search is non-responsive and pages are timing out everywhere. They don’t have a choice but to migrate - but it seems that non-standard options, customization, and plugins may be the destroyer of their worlds. Without a doubt there are people considering leaving, except they feel “locked in” by their data. “Lock-in” is something the platform providers think of as a “business advantage.”

Portability should be a serious industry discussion - but forum platform business models have been haphazard at best, and scrambling for customers has caused migration planning to be shunted off to the pile of features that is “nice-to-have” instead of core the long-term-health of the platform developers. Forum platform sales says “Yes! We will customize your forum”, and engineering delivers yet another non-portable/upgradable solution - furthering lock-in, but hamstringing innovation.

Platforms should compete on presentations, features, and service - a good platform should have import/export from the start. Perhaps an industry standard is in order? Forums are woefully missing from (Is this another fork?)

I’m truly glad to hear that Ars is happy, apparently indefinitely, with their current audience and solution - some people still prefer command-line OS interfaces as well (no insult here, as I have CYGWYN on my desktop). But, I worked at Yahoo! for 5 years on social products (including forum and group redesigns) and can tell you from research and direct experience : Forum software (circa 2008) is seen as archaic and unusable by the vast majority of the English speaking world. Since 2000, the internet has grown more than 10x in users - have forums? From the numbers I’ve seen, the answer is not even close.

I HOPE that it isn’t the case that other people who jumped in early and built forum-based communities feel they must reluctantly stick with their aging platform waiting on a migration path that will never come (or perhaps never should.)

(badp) #7

Probably the main technical issue for you guys is that you’d have to distinguish between news posts written with ye olde forums (and thus need to fetch their comments from PHPBB) and news post written after the switch (and thus need to fetch their comments there), plus you’d have to present them consistently.

At any rate, Jeff, do keep in mind that Ars has been migrating 14 years old stuff from their original forms to their current design in the news portion of their site (from this to this). Sure, the article does transpire its age from the way the (teensy tiny) images are embedded (“click to enlarge”) to the generally smaller article pages… but it’s all there.

(Errata: okay, it seems like they didn’t migrate everything, I can’t find the new version of this review for example.)

(Luke Larris) #8

For me at least, forum migration means having existing content for users to reply to, as well as search engine benefits (sure everything has to be re-indexed, but it’s better than starting from nothing).

I understand the problems and the huge complications that come with building and maintaining a migration system for any forum software, and having to support a lot of the major softwares out there in the process, but it’s really worth it in my opinion.

(Brian Gillespie) #9

To take this a bit further, we’re slowing down on the migration now that we’ve moved most of the NNTP traffic and away from some of the more awful forums that served smaller communities. We’ve still got several NING communities going that we’ll frankly probably keep on NING. Not because discussion is great, but because there are advantages to NING that Discourse doesn’t bring. Plus we’ve got some great momentum going on them. An example is the Grasshopper community.

We’ve moved over one VBulletin site so far, and are working to move from an ancient CommunityServer installation. As you can see, we’ve tried a lot of “forum” and “community” services over the years, and they each suck in their own very special way.

The advantage we get with NING is simple:
We know who the users are that care about certain sub-categories of Grasshopper (their “Groups” feature). We can contact them directly via e-mail, and we can tell how active they are. This targeted out-bound communication to casual readers is appealing. I’m not suggesting that Discourse needs to move toward community building, but there is a benefit to the community features that keep us on NING even though their forum is truly hideous.

And as an aside, I have no problem having archived read-only content from old systems. I’d guess that most of the content goes stale anyway, and moving it just sounds like a mess.

(Dan Friedman) #10

Sorry for resurrecting an old thread, but I wanted to chime in. There are probably some forums where migration doesn’t matter, but for sites like "how-to"s, video game conversations, Q&As, etc., I think it really does. I go to certain forums because I know they have a lot of both old and new content. I look up old content all the time cause I play old (cheap) video games and because, sometimes, the answer simples doesn’t really change. We’ve been around for 20+ years, we have a ton of content and 30% of our traffic is to our forums.

As a developer, it is very easy for me to tell my users to search the old forum for old content and the new forum for new content, but my customers aren’t going to do it. They’ll just go somewhere else.

When I do the migration, I plan to host it at a separate subdomain, and then have redirects from each old page to the corresponding new page. That way, all the links will continue to work.

Will migration be a huge pain? Probably. But we’ve done it before and we’ll do it again. Because it’s what is best for our customers and it’s what is best for our business. It’s just not what is best for the developers, but that’s life.

(Jeff Atwood) #11

Wouldn’t they just type the name of the game into Google and click search? I fail to see how that usage is harmed in the slightest if the old topic is at versus People don’t care, they’ll just search the web.

And if the old forum has obvious signposts to the new forum, and it’s read only so no logins or new posts are possible, surely those who are actually interested can find their way over?

(Glenn Drake) #12

I don’t have any stats on this but I’d hazard a guess and say that over 90% of forum content is discovered through search engines rather than the native search tools. Most forum platforms I’ve used have dreadful search capabilities… With a particular phpBB forum I frequented, I would routinely go to Google and site: search as it would turn up more accurate results.

Before moving our community to Discourse I had a GetSatisfaction forum that needed a new home. Our community is there to support an accounting platform we are developing, many of the discussions actually become dated very quickly as the software evolves. Still there’s some real gems in there, so I knocked up a simple .NET web app to host all the static content. GS were kind enough to supply me with a CSV file of all the posts and replies. The app deploys the posts from the CSV to a SQL database, sanitizes the links and images and builds an xml sitemap. Google have already indexed most of the content on the archived forum and I signpost users to there from our Discourse instance when they ask repetitive questions. I just thought I’d share that, I’m happy to zip up the application if anyone is also moving over from GS.

Oh, I actually did all this while I was on holiday :smile:, my wife was less than impressed.

The long and short of this post is, I’m not actually bothered where the content is as long as it can be strung together in some way and is discoverable. I’ve also yet to receive any complaints about making users log into a new system, I think most people are largely indifferent.

(Dan Friedman) #13

I agree that most (all?) forum search is dreadful, but it does give me results within the community I’m interested in. Sometimes, I’m ok with a simple web search, other times, I’m looking for results from a community I trust. Yes, site search is probably better, but that is something that only a relatively few actually know about or even do.

I’m in the fitness industry, some things change, some things stay the same, some things are just interesting as a reference.

Also, I’ve seen and MSDN forums update their forums. No loss of content. It would hurt their users, right?

StackOverflow is a type of forum. It contains tons of useful content. Main areas are split into subdomains, right? Now imagine that they decided to use a new platform and decided that migration was too hard. They could move all their old content to a new domain “*”. Or they could create a new domain “*”. Either scenario seems ludicrous to me.

Your thoughts @sam?

You wouldn’t want to kill SEO, so the new domain seems like the better option. But now when people want to post a question, they either have to remember a new domain (that you worked so hard to come up with) or get told they are in the wrong place or otherwise get redirected in some fashion that let’s the user know things have changed. There’s no history to let users know that a similar question was already posted. Anyone who had subscribed to a topic will no longer receive notifications, so less traffic. And I think graveyard forums are bad for UX. “Here’s a forum. You can’t reply to anything. It’s just to look at. Want to do something? Go over here instead.” I’ve seen people on here who have migrated their forum multiple times. Could you imagine if a website had 3, 4 or 5 graveyard forums? I would think that the company was crazy.

Also, I was just reminded of this: “Your stuff belongs to you. We have built in robust import and export tools, both for the site owner and for individual users. Your content belongs to you; take it with you wherever you want to go.” That makes it seem like migration was already accounted for in building Discourse. What am I missing?

So it this is already built-in and we are just arguing over whether it’s worth it? Or is there feature work that needs to be done?

And one final thing, when I tell the CEO that we could choose a new forum platform that can handle migration (and he knows that it can be done because we’ve done it before) or I tell him we can use a new platform where we have to start over, he wants the first one. He doesn’t care about the extra work. He cares about the end result.

(F. Randall Farmer) #14

The choice isn’t between “start over” and “migrate” - that’s the point of this thread - could you go back and read it a bit more closely?

The idea is to keep the old stuff exactly where it is (or in an improved archive form) - with the new threads pointing to it as needed.

But the question on the table is “Does every single thread always have to be accessed in exactly the same form, complete with the option to revive (necro) it, forever?”

For most communities, I don’t think that is a real requirement (nor is it possible to accomplish if you really want to upgrade to a new platform.)

The question isn’t “Migrate/Don’t” - it’s more “What form should data migration/archiving take, and why?”

(Dan Friedman) #15

When I said “start over”, I was referring to the concept of “not migrating”. If you had read more than the last sentence (the only sentence where I used that phrase) that would have been clear. I never said anything about deleting content. Please read through my comments again. They might not agree with your thoughts, but I tried to be thorough.

This conversation will probably not ever result in a great epiphany from either side, but I do like the idea by @Heisenberg of creating static files and your concept of thinking of it as an archived format, but, again, there are some cons to starting a new forum and leaving the old behind (covered above).

Yes, most. But what of, MSDN Forums and StackOverflow. Should they not migrate? And when is something important enough that it should be migrated. I don’t think anyone can decide that except each individual property.

And again, what’s the point of Import/Export functionality if we can’t import?

Note: I am not totally opposed to not migrating either. I just am not convinced enough, that I should try to convince my CEO that he is wrong (he believes the forum is part of the lifeblood of the company) and feel confident that I am leading him in the right direction.

(F. Randall Farmer) #16

Of course, all technology changes are disruptive, every community should way the choices carefully. And, upgrading to Discourse is definitely disruptive (it was a design requirement!) :slight_smile:

My desire by starting this thread was to get folks to consider something other than an overly simplified and binary “full migration” vs. “start (completely) over” choice.

Sorry if I triggered on the keywords. My bad.

(Glenn Drake) #17

I think what titled me towards having a static container for the old forum was largely due to the fact that our forum isn’t the core part of our business. Of course this is entirely different for Stack Overflow which would certainly suffer from this degree of fragmentation. I do accept that what worked for me may not work for others, I guess it all depends on the nature of the community.

(F. Randall Farmer) #18

Personally, I think the Stack Overflow example is a red herring. Discourse isn’t designed to do anything like what SO does well - it isn’t an upgrade to Stack Overflow in any way… Upgrades to SO should be backward compatible with SO.

On the other hand, there were some Stack Exchange experiments that failed - a theoretical deployment of Discourse for some those communities might be a reasonable consideration - since the old platform didn’t have the desired effects.

In those theoretical cases, it seems likely that, instead of attempting to twist the Stack Exchange Q&A format into a discussion format, that archiving would be the better choice.

(Joop Kiefte) #19

That would be a lot better for example for the Esperanto communities…


I really dont see a problem here.
Migrate the database, set up the 301 redirects with some regex values and there you go.
It can be done on a new domain or the same domain.
What am i missing here?