# Merging two Discourse forums
**URL:** https://meta.discourse.org/t/merging-two-discourse-forums/72578
**Category:** Migration
**Created:** [October 23, 2017, 12:23am UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578 "2017-10-23T00:23:27Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@omarfilip](https://meta.discourse.org/u/omarfilip)
#### Post date: [October 23, 2017, 12:23am UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578/1 "2017-10-23T00:23:27Z")
> [@sam](#):
>
> These days we have a category importer and exporter that @neil built, you would use that to merge instances
I’m trying to use this and I am not having much success.
> [@Move topics from one Discourse instance to another](https://meta.discourse.org/t/topic-and-category-export-import/38930):
>
> Need to move some stuff from one instance of Discourse to another? There is a command line tool to export a topic, a set of topics, or an entire category, and another to import them in at the other end. Topic Export/Import one or more topics and their replies the users who posted in the topic Category Export/Import the category and all its subcategories its security settings custom groups mentioned in the security settingswarning if any of the groups have ‘membership requests’ enable…
Discourse A has ~100k posts and Discourse B ~30k posts. I’ve moved all categories under one main category and then exported it from Discourse B. When I import into Discourse A only 5 posts are added and two are attributed to an incorrect user.
For the incorrectly attributed example post I’ve noticed that in Discourse B it was made by a user with a user id 176, and in Discourse A it is attributed to a user with the same user id. But those are two distinct users.
No users in A and B have the same email address, but ~100 have the same user name.
Oddly, when I first tried a test to see if this could work, I ran the export-import and it seemed that most of the 30k posts were transferred, but post ownership was completely jumbled.
Perhaps a fix to the post ownership issue could be to change user ids in Discourse B before export so that they don’t have a duplicate in Discourse A. Can this be done in bulk? Do i need to change usernames in B, and can it be done in bulk by adding “\_2” to each username?
As for the 29,995 posts that didn’t make it over - I’m stumped. I see those topics scroll by during import, but not in Discourse A after import.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [October 23, 2017, 5:01am UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578/2 "2017-10-23T05:01:02Z")
Hmm. I think the solution might be to write an importer for Discourse.
Could be an interesting project.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [October 23, 2017, 11:42am UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578/3 "2017-10-23T11:42:03Z")
If you have a budget and would like me to write an importer, let me know or post in #Marketplace.
Users, posts, and categories should be easy enough, but you’d also want to replace @mentions and post/topic links within discourse, which would add a bit of complexity. Maintaining user and post stats could also be done with more time or money.
---
### Author:  [@neil](https://meta.discourse.org/u/neil)
#### Post date: [October 23, 2017, 2:16pm UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578/4 "2017-10-23T14:16:56Z")
> [@omarfilip](#):
>
> No users in A and B have the same email address, but ~100 have the same user name.
I think I see why that’s happening. The [topic importer only looks at email address](https://github.com/discourse/discourse/blob/master/lib/import_export/topic_importer.rb#L21-L29), but it uses a common importer method to create a new user when it doesn’t find an existing user with the email address. That method will [look for a matching username](https://github.com/discourse/discourse/blob/master/script/import_scripts/base.rb#L272), which is not what I want… This is a bug so I’ll have a look.
Scratch that. It does work by looking only at email addresses. @omarfilip Was your Discourse A site a migration from other forum software?
---
### Author:  [@omarfilip](https://meta.discourse.org/u/omarfilip)
#### Post date: [October 23, 2017, 4:16pm UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578/5 "2017-10-23T16:16:31Z")
> [@neil](#):
>
> Was your Discourse A site a migration from other forum software?
Yes - “A” is from phpBB using @gerhard’s awesome importer, and “B” came from an ancient forum that @pfaffman successfully converted from a sql dump.
---
### Author:  [@neil](https://meta.discourse.org/u/neil)
#### Post date: [October 23, 2017, 4:19pm UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578/6 "2017-10-23T16:19:57Z")
I see. So user id 123 from phpBB “A” is not the same user as user id 123 from ancient forum “B”, but the importers are going to see them as the same and not create possible duplicate users. That’s normally the behaviour we want, but not in this case. We need a way to force treat all incoming users as new.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [October 23, 2017, 4:21pm UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578/7 "2017-10-23T16:21:19Z")
OH!
In that case you (or I) should run that script on top of your new data rather than try to merge the two Discourses. I didn’t know (or didn’t fully appreciate) that was what you were trying to do.
I think it might “just work” to run it on top of your forum, but it might need some tweaks.
Send me an email with the details and we’ll figure out how to proceed.
---
### Author:  [@omarfilip](https://meta.discourse.org/u/omarfilip)
#### Post date: [October 23, 2017, 4:24pm UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578/8 "2017-10-23T16:24:28Z")
> [@neil](#):
>
> user id 123 from phpBB “A” is not the same user as user id 123 from ancient forum “B”
Correct.
However, username johndoe from “A” is the same person with username johndoe in “B.” In my case it would not matter much if during the conversion (or in a prep step before) johndoe from B became johndoe\_2.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [October 23, 2017, 4:27pm UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578/9 "2017-10-23T16:27:34Z")
> [@omarfilip](#):
>
> However, username johndoe from “A” is the same person with username johndoe in “B.” In my case it would not matter much if during the conversion (or in a prep step before) johndoe from B became johndoe\_2.
That should happen automatically if the import script runs on top of your existing Discourse database.
---
### Author:  [@omarfilip](https://meta.discourse.org/u/omarfilip)
#### Post date: [October 23, 2017, 4:30pm UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578/10 "2017-10-23T16:30:51Z")
> [@pfaffman](#):
>
> run that script on top of your new data rather than try to merge the two Discourses.
Would an incremental import from phpBB work after that script was run? (To catch up with posts that were made after I made the initial import from phpBB.)
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [October 23, 2017, 4:44pm UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578/11 "2017-10-23T16:44:48Z")
I’m pretty sure that this will work:
- freeze the live formerly phpbb Discourse,
- backup
- restore that backup to a development machine
- import the ancient data on dev machine
- backup
- restore to prodcution
- profit
The only question is whether there are any clashes between import ID on the phpBB data and the ancient data. This should be pretty easy to fix in the importer (e.g., add “X” before all of the import IDs).
---
### Author:  [@neil](https://meta.discourse.org/u/neil)
#### Post date: [October 23, 2017, 6:19pm UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578/12 "2017-10-23T18:19:37Z")
> [@pfaffman](#):
>
> This should be pretty easy to fix in the importer (e.g., add “X” before all of the import IDs).
I fixed a bug where this wouldn’t work because it was being converted to an integer. `"x123".to_i == 0`. I also added a way to treat the output files from the topic exporter as coming from different places, which essentially does the same trick of appending an arbitrary string to the id.
```plaintext
IMPORT_SOURCE=A bundle exec ruby script/discourse import_topics filename-1.json
IMPORT_SOURCE=B bundle exec ruby script/discourse import_topics filename-2.json
```
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [October 23, 2017, 6:33pm UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578/13 "2017-10-23T18:33:30Z")
OH! You’re back on the import\_topics solution. I was talking about re-running the importer second importer. Several importers count on import\_id being a string.
It’s not immediately clear which solution is easier/preferable. I suppose if the import\_topics solution works, that would be great!
---
### Author:  [@omarfilip](https://meta.discourse.org/u/omarfilip)
#### Post date: [October 23, 2017, 6:35pm UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578/14 "2017-10-23T18:35:30Z")
Does this mean if I re-run Topic and Category Export/Import it should now work?
---
### Author:  [@neil](https://meta.discourse.org/u/neil)
#### Post date: [October 23, 2017, 6:38pm UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578/15 "2017-10-23T18:38:41Z")
Every case is a special unique snowflake. ❄ You worked on the migration for @omarfilip , so you would know better than me.
> [@pfaffman](#):
>
> Several importers count on import\_id being a string.
The group and user id’s [were being converted to integers](https://github.com/discourse/discourse/commit/aec5de451b85e0d83ea3f8d4793c8d60c6552cd0), so that could have been causing problems. Everything in the LookupContainer uses strings as it should though.
> [@omarfilip](#):
>
> Does this mean if I re-run Topic and Category Export/Import it should now work?
Only if you can get today’s code to use, but wait until the tests-passed branch has it.
---
### Author:  [@omarfilip](https://meta.discourse.org/u/omarfilip)
#### Post date: [October 25, 2017, 5:21pm UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578/16 "2017-10-25T17:21:26Z")
I re-ran the export/import with Discourse ([1612818)](https://github.com/discourse/discourse/compare/1612818...tests-passed).
The only apparent difference was that the importer reported which emails it was invalidating during the import. The result of the process was still the same - only 5 posts got imported, and post ownership is still mismatched.
Checking the status of the code - does this mean not everything made it into tests-passed?
---
### Author:  [@neil](https://meta.discourse.org/u/neil)
#### Post date: [October 25, 2017, 5:42pm UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578/17 "2017-10-25T17:42:58Z")
It’s in test-passed. The Travis [build was cancelled](https://travis-ci.org/discourse/discourse/builds/291676379?utm_source=github_status&utm_medium=notification) for some reason.
If IMPORT\_SOURCE still only created 5 posts, then I don’t know what’s happening. At this point someone needs to compare the data (post id’s, user emails, etc.) in the exported data with what’s in the target db and figure out why there’s a collision (user\_custom\_fields, post\_custom\_fields).
---
### Author:  [@omarfilip](https://meta.discourse.org/u/omarfilip)
#### Post date: [October 25, 2017, 7:26pm UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578/18 "2017-10-25T19:26:30Z")
Does the category importer care if a `name` has a duplicate, but the `username` and `email` are unique?
In the forum that’s being exported/imported there are several that are like that.
---
### Author:  [@neil](https://meta.discourse.org/u/neil)
#### Post date: [October 25, 2017, 8:49pm UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578/19 "2017-10-25T20:49:36Z")
No, it never looks at name. If you’re using the new IMPORT\_SOURCE support, then email address is the only thing that it looks at. If there’s a duplicate username but different email address, then a number will be put on the end, like “omar1”.
---
### Author:  [@omarfilip](https://meta.discourse.org/u/omarfilip)
#### Post date: [October 26, 2017, 8:57pm UTC](https://meta.discourse.org/t/merging-two-discourse-forums/72578/20 "2017-10-26T20:57:58Z")
Success! 🎉
With @pfaffman’s guidance, deleting `import_id` from both Discourses before export/import worked:
```
rails c
UserCustomField.where(name: 'import_id').delete_all
GroupCustomField.where(name: 'import_id').delete_all
CategoryCustomField.where(name: 'import_id').delete_all
PostCustomField.where(name: 'import_id').delete_all
```
This did not work:
`discourse import_category /category-export-2017-10-25-225306.json`
but this did:
`bundle exec discourse import_category /category-export-2017-10-25-225306.json`
Having to delete `import_id` from the Discourse that was converted with the phpBB importer prevents me from doing a leisurely initial import first and an incremental import later, but I can live with that.
Many thanks to @neil and @pfaffman!
[Next page](https://meta.discourse.org/t/merging-two-discourse-forums/72578.md?page=2)