Merge two Discourse sites into one

[Discourse]

If you have two Discourse sites that you wish were one, this guide is for you.

There’s a tool called discourse_merger that can take one Discourse site and merge it into another.

Prereqs

This is not an easy task, and should be treated like any other migration to Discourse. You will not be running discourse_merger on a live production site. You will perform the merge in another environment where you can review the output before moving the result to production.

Copy vs Merge

Almost everything will be copied from one site to the other, but Categories and Users can be merged, which will avoid duplication.

• Users will be merged if a user on both sites has the same email address.
• Categories will be merged if they have the same name.

If you want to do any reorganization of your data, do it before merging.

Choose the destination site

Choose which site will be the destination for the data. This is the one that will retain all its styling and settings. The other site will have its users, categories, topics, posts, uploads, etc. copied/merged into the destination site.

How to do it

Take backups of both sites including files and copy them to the environment where you’ll perform the merge. It’s possible that they’re from different versions of Discourse, so we need them to be at the same version. I would choose to use the most recent version of Discourse while performing the merge.

Restore the destination site to the merger environment. If doing this from the command line:

bundle exec ruby script/discourse restore destination-2018-08-02-134227-v2018xxx.tar.gz

Next we’ll extract the other site.

cd /path/to/data
tar xvzf other-2018-08-02-134227-v2018xxx.tar.gz

The output will include the database dump and the upload files.

Create a database with the data:

psql
CREATE DATABASE "copyme" ENCODING = 'utf8';
\q
gunzip < /path/to/data/other-2018-08-02-134227-v2018xxx.tar.gz | psql -d copyme

If you’re running the import in an official Docker container (recommended), you will need to reset the postgres password to provide it to the script, otherwise you may encounter an error that the postgres user cannot access the database.

To change the password:

sudo -u postgres psql
\password postgres
(enter the new password)
\q

Now it’s time to run the script. Some env variables you’ll set:

DB_NAME: name of database being merged into the destination site.
DB_HOST: (optional) hostname of database being merged. leave blank if it’s local.
DB_PASS: password for the postgres user to access the database
UPLOADS_PATH: absolute path (site being merged) of the directory containing “original” and “optimized” dirs. e.g. /path/to/data/uploads/default
SOURCE_BASE_URL: base url of the site being merged. e.g. https://meta.discourse.org
SOURCE_CDN: (optional) base url of the CDN of the site being merged.

You may need to run a bundle install prior to running the import script to avoid errors. To do so:

su discourse -c 'bundle config set --local with generic_import && bundle install'

On the first run, you might need to install some extra dependencies for the gems required in import.

Once the bundle is complete, run the import.

su discourse -c 'DB_NAME=copyme DB_PASS=password SOURCE_BASE_URL=http://copy.othersite.com UPLOADS_PATH=/shared/import/data/uploads/default bundle exec ruby script/bulk_import/discourse_merger.rb'

When it’s done, review the output in a web browser.

You can use the remap tool to update links from the old forum.

bundle exec ruby script/discourse remap 'copy.othersite.com' 'hot.newsite.com'

Also rebake all posts with uploads:

rake posts:rebake_match["upload:"]

If everything looks good, take a backup of the result and restore it to your production server.

bundle exec ruby script/discourse backup

Last edited by @italo 2024-11-27T00:52:10Z

Last checked by @JammyDodger 2024-05-26T21:20:00Z

Check document

Perform check on document:

[pfaffman (Jay Pfaffman)]

동작하는 것 같지만, 백업을 실행하면 다음 오류가 발생합니다:

pg_dump: error: query failed: ERROR:  permission denied for table migration_mappings                                                                                   

정말 이상하네요.

수정: 아래 명령어로 해결되었습니다.

ALTER USER discourse WITH SUPERUSER;

[AstonJ]

요즘 이 기능을 사용해 보신 분 계신가요? 어떻게 작동했나요?

또한, 사용자가 원래 있던 포럼별로 자동으로 그룹에 배정되는 기능도 가능한지 아시는 분 계신가요? (그렇게 하면 사용자가 온 포럼의 주제에 대한 열람 권한을 부여하는 것이 더 쉬워집니다.)

[pfaffman (Jay Pfaffman)]

조금 거칠었습니다. 몇 가지 주석 처리를 해야 했을 것 같아요. 또한 머저 이미지 문제도 있었어요.

새 사이트의 모든 사용자를 특정 그룹에 추가하는 것이 좋다고 생각합니다. 이렇게 하면 머지할 때 해당 그룹에 포함되겠죠. 머지 후나 머지 과정에서 처리하는 것보다 훨씬 쉬울 거예요.

[pfaffman (Jay Pfaffman)]

지난번에 이 작업을 했을 때 병합된 사이트의 업로드 파일이 전부 누락되어 있었습니다. tar tf backupfile.tar.gz 명령으로 업로드 파일 목록을 가져와 allfiles.txtx 파일에 넣고, 이를 업로드 디렉토리로 복사했습니다. 이 스크립트(수정 없이 여러분 환경에서 작동하지 않을 가능성이 높음)는 해당 파일 각각에 대해 업로드를 생성했으며, 이후 게시물을 다시 베이킹(rebake)하니 누락된 이미지 대부분(또는 전부)이 복구되었습니다.

def process_uploads
  begin
    # Read the list of filenames
    filenames = File.readlines('/shared/uploads/allfiles.txt').map(&:strip)
    count = 0

    filenames.each do |filename|
      # Prepend /shared to the filename
      filename.gsub!(/\.\//,"")
      full_path = File.join('/shared/uploads/default/original/', filename)

      begin
        # Check if path exists and is a regular file (not a directory)
        count += 1
        
        if File.exist?(full_path) && File.file?(full_path)
          # Open the file
          File.open(full_path, 'r') do |tempfile|
            # Create upload using the specified parameters
            u = UploadCreator.new(tempfile, 'imported', {}).create_for(-1)
            puts "#{count} -- #{u.id}: #{u.url}"
          end
        else
          puts "Warning: Path not found or is not a regular file: #{full_path}"
        end
      rescue => e
        puts "Error processing file #{full_path}: #{e.message}"
        # Continue with next file even if current one fails
        next
      end
    end
  rescue Errno::ENOENT
    puts "Error: Could not find files.txt"
  rescue => e
    puts "Error reading files.txt: #{e.message}"
  end
end

# Execute the processing
process_uploads;

문제 있는 게시물은 다음과 같이 조회했습니다:

 bad=Post.where("cooked like '%/images/transparent.png%'")

그 후 다시 베이킹이 필요하도록 표시하기 위해 다음을 실행했습니다:

bad.update_all(baked_version: nil)

조급해서 다음 명령을 사용하여 다시 베이킹을 수행했습니다.

rake posts:rebake_uncooked_posts

[AstonJ]

결국 더 쉬운 방법은, 병합하려는 디스커스 포럼을 먼저 Xenforo로 변환하는 것일지 궁금합니다(그들의 가져오기 도구는 보통 매우 훌륭하니까요). 그런 다음, 병합에 포함시키고 싶은 다른 포럼들과 병합하는 것입니다(이 포럼들은 VBulletin에서 Xenforo로 변환될 예정이죠). 그리고 마지막으로, 새로 병합된 Xenforo 포럼을 다시 디스커스로 가져오는 것이죠..