将带有MongoDB的NodeBB论坛迁移到Discourse

如您所知,NodeBB 支持两种数据库后端:Redis 和 MongoDB。Discourse 导入脚本同时支持这两种数据库。本教程将介绍如何迁移使用 MongoDB 作为数据库后端的 NodeBB。我们将使用带有 mongo 适配器的 NodeBB 导入器。如果您的 NodeBB 论坛使用 Redis 作为后端,请遵循 本教程,其中演示了 redis 适配器的使用方法。

计划

  • 准备开发环境。
  • 从生产环境导出数据库。
  • 将生产数据库导入到 Discourse 实例中。
  • 运行导入脚本。

可迁移的内容

  • 用户组
  • 附件
  • 分类
    • 根分类 => 根分类
    • 子分类 & 子子分类 => 子分类
  • 主题与帖子
    • 置顶主题 => 置顶主题
    • 锁定主题 => 关闭主题
    • 主题浏览量
    • 点赞用户
    • 样式、提及、表情符号和附件。
  • 用户(包含以下属性)
    • 个人资料背景
    • 头像
    • 封禁状态
    • 用户名
    • 姓名
    • 电子邮件
    • 管理员状态
    • 个人简介
    • 用户组
    • 网站
    • 所在地
    • 注册时间

准备本地开发环境

如计划所述,我们首先需要准备开发环境。请按照以下指南之一安装 Discourse:

:bulb: 如果您在设置 Discourse 服务器 时遇到任何问题,请参阅 本指南

然后安装 MongoDB 数据库服务器。

Ubuntu 18.04:

$ wget -qO - https://www.mongodb.org/static/pgp/server-4.0.asc | sudo apt-key add -
$ echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
$ sudo apt-get update
$ sudo apt-get install -y mongodb-org
$ sudo service mongod status

更多详情,请参阅 官方指南

Mac OS:

$ brew tap mongodb/brew
$ brew install mongodb-community@4.0
$ brew services start mongodb-community@4.0
$ brew services status mongodb-community@4.0

更多详情,请参阅 官方指南

Windows 10:

下载安装程序并将 MongoDB 安装为 Windows 服务:

https://fastdl.mongodb.org/win32/mongodb-win32-x86_64-2008plus-ssl-4.0.12-signed.msi

以管理员权限运行 cmd 以检查 MongoDB 服务器是否正常工作:

"C:\Program Files\MongoDB\Server\4.0\bin\mongo.exe"

更多详情,请参阅 官方指南

此环境将作为我们的 Discourse 服务器

导出生产数据库转储

关闭您的 NodeBB 论坛(生产服务器)。

$ cd /path_to_nodebb
$ ./nodebb stop

关闭您的数据库:

$ sudo service mongodb stop

备份您的数据库:

$ mongodump --out ~/my_backup_path/

上一条命令的输出可能如下所示:

2019-08-16T15:17:09.845+0300 done dumping admin.system.users (1 document)
2019-08-16T15:17:09.846+0300 done dumping admin.system.version (2 documents)
2019-08-16T15:17:09.849+0300 done dumping nodebb.sessions (10 documents)
2019-08-16T15:17:09.850+0300 done dumping nodebb.socket.io (215 documents)
2019-08-16T15:17:09.854+0300 done dumping nodebb.objects (1488 documents)

请注意,备份实际上是一个目录,而不仅仅是一个文件。

:bulb: 您可以通过在 mongo CLI 中运行 use myDatabase 然后运行 db.stats().dataSize; 来检查数据库的大小。返回的值将以字节为单位。

备份您的论坛资源:

$ cd /path_to_nodebb_root_directory/
$ tar -czf ./uploads.tar.gz ./public/uploads

在拥有数据库和资源后,请将它们复制到 Discourse 服务器

导入数据库

现在我们已经有了数据库,可以将其导入到本地 MongoDB 实例中:

$ mongorestore ~/path_of_my_backup_directory/

更多选项,请参阅 官方指南

接下来,您需要解压 uploads.tar.gz,以便导入器能够导入资源:

$ tar xvzf uploads.tar.gz

运行导入脚本

现在数据库和资源已就位,我们可以准备运行导入脚本。在此之前,我们需要编辑 NodeBB 导入脚本以适应我们的需求。

这是您的 NodeBB 上传文件夹的路径:

ATTACHMENT_DIR = '/absolute_path/uploads'

接下来,我们需要告诉导入器使用 mongo 适配器而不是 redis 适配器:

adapter = NodeBB::Mongo
@client = adapter.new('mongodb://127.0.0.1:27017/nodebb')

# adapter = NodeBB::Redis
# @client = adapter.new(
# host: "localhost",
# port: "6379",
# db: 14
# )

在干净的 Discourse 环境中并支持 mongo gem 的情况下运行导入器:

$ cd ~/discourse
$ echo "gem 'mongo'" >> Gemfile
$ bundle install
$ bundle exec rake db:drop db:create db:migrate
$ bundle exec ruby script/import_scripts/nodebb/nodebb.rb

导入器将连接到 MongoDB 实例,并将所有内容迁移到 Discourse。

导入完成后,启动 Discourse:

$ bundle exec rails server

启动 Sidekiq 以处理迁移的数据:

$ bundle exec sidekiq

您可以在 http://localhost:3000/sidekiq/queues 监控进度。

按照此 教程 执行 Discourse 备份,并将其上传到您的 Discourse 生产服务器。

:tada:

如果您对该过程有任何疑问,我很乐意提供帮助。

祝迁移顺利 :grinning:

10 个赞

Great work really. While I was trying, I am getting below error. Any idea what I might be doing wrong here?

importing groups
   10 / 10 (100.0%)  [474765 items/min]    
importing top level categories...
    8 / 8 (100.0%)  [437896 items/min]    
importing child categories...
   68 / 68 (100.0%)  [774048 items/min]  
importing users
 5534 / 5534 (100.0%)  [355520 items/min]    
adding users to groups...

importing topics...
Traceback (most recent call last):
	12: from script/import_scripts/nodebb/nodebb.rb:532:in `<main>'
	11: from /home/fsuzer/discourse/script/import_scripts/base.rb:47:in `perform'
	10: from script/import_scripts/nodebb/nodebb.rb:52:in `execute'
	 9: from script/import_scripts/nodebb/nodebb.rb:336:in `import_topics'
	 8: from /home/fsuzer/discourse/script/import_scripts/base.rb:877:in `batches'
	 7: from /home/fsuzer/discourse/script/import_scripts/base.rb:877:in `loop'
	 6: from /home/fsuzer/discourse/script/import_scripts/base.rb:878:in `block in batches'
	 5: from script/import_scripts/nodebb/nodebb.rb:337:in `block in import_topics'
	 4: from /home/fsuzer/discourse/script/import_scripts/nodebb/mongo.rb:71:in `topics'
	 3: from /home/fsuzer/discourse/script/import_scripts/nodebb/mongo.rb:71:in `map'
	 2: from /home/fsuzer/discourse/script/import_scripts/nodebb/mongo.rb:71:in `block in topics'
	 1: from /home/fsuzer/discourse/script/import_scripts/nodebb/mongo.rb:79:in `topic'
/home/fsuzer/discourse/script/import_scripts/nodebb/mongo.rb:100:in `post': undefined method `[]' for nil:NilClass (NoMethodError)

Some field from the topic is empty. Or perhaps the topic query returned nothing.

1 个赞

Can someone help me?

Error: Could not find gem ‘mongo’ in any of the gem sources listed in your Gemfile.

After $ bundle install

This will happen if you skipped this:

Make sure you Gemfile has this line:

gem 'mongo'

Then run:

$ bundle install
1 个赞

Gem

Thanks for your help. But this line already exists in my file. I also did the previous steps informed.

I don’t know if that helps, but when I run gem list mongo it’s not listed.

Your Gemfile sounds weird to me. Please, can you share the whole content?

Your Gemfile is just wrong. It should have another content, just like this:

In addition to:

gem 'mongo'

嘿,

感谢这份指南!当我运行迁移脚本时,在脚本执行到导入用户这一步,我得到了以下输出:

Traceback (most recent call last):
        21: from script/import_scripts/nodebb/nodebb.rb:532:in `<main>'
        20: from /home/odyslam/forum/discourse/script/import_scripts/base.rb:47:in `perform'
        19: from script/import_scripts/nodebb/nodebb.rb:50:in `execute'
        18: from script/import_scripts/nodebb/nodebb.rb:130:in `import_users'
        17: from /home/odyslam/forum/discourse/script/import_scripts/base.rb:262:in `create_users'
        16: from /home/odyslam/forum/discourse/script/import_scripts/base.rb:262:in `each'
        15: from /home/odyslam/forum/discourse/script/import_scripts/base.rb:274:in `block in create_users'
        14: from /home/odyslam/forum/discourse/script/import_scripts/base.rb:391:in `create_user'
        13: from /home/odyslam/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/core_ext/object/try.rb:15:in `try'
        12: from /home/odyslam/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/core_ext/object/try.rb:15:in `public_send'
        11: from script/import_scripts/nodebb/nodebb.rb:164:in `block (2 levels) in import_users'
        10: from script/import_scripts/nodebb/nodebb.rb:211:in `import_profile_picture'
         9: from /home/odyslam/forum/discourse/lib/upload_creator.rb:45:in `create_for'
         8: from /home/odyslam/forum/discourse/lib/distributed_mutex.rb:14:in `synchronize'
         7: from /home/odyslam/forum/discourse/lib/distributed_mutex.rb:29:in `synchronize'
         6: from /home/odyslam/forum/discourse/lib/distributed_mutex.rb:29:in `synchronize'
         5: from /home/odyslam/forum/discourse/lib/distributed_mutex.rb:33:in `block in synchronize'
         4: from /home/odyslam/forum/discourse/lib/upload_creator.rb:66:in `block in create_for'
         3: from /home/odyslam/forum/discourse/lib/upload_creator.rb:381:in `optimize!'
         2: from /home/odyslam/forum/discourse/app/models/optimized_image.rb:178:in `ensure_safe_paths!'
         1: from /home/odyslam/forum/discourse/app/models/optimized_image.rb:178:in `each'
/home/odyslam/forum/discourse/app/models/optimized_image.rb:179:in `block in ensure_safe_paths!': Discourse::InvalidAccess (Discourse::InvalidAccess)

我通过注释掉上传头像的部分绕过了这个问题,这倒不是问题。

现在我在导入话题时遇到了 nil 错误。:frowning:

Traceback (most recent call last):
        12: from script/import_scripts/nodebb/nodebb.rb:532:in `<main>'
        11: from /home/odyslam/forum/discourse/script/import_scripts/base.rb:47:in `perform'
        10: from script/import_scripts/nodebb/nodebb.rb:52:in `execute'
         9: from script/import_scripts/nodebb/nodebb.rb:336:in `import_topics'
         8: from /home/odyslam/forum/discourse/script/import_scripts/base.rb:862:in `batches'
         7: from /home/odyslam/forum/discourse/script/import_scripts/base.rb:862:in `loop'
         6: from /home/odyslam/forum/discourse/script/import_scripts/base.rb:863:in `block in batches'
         5: from script/import_scripts/nodebb/nodebb.rb:337:in `block in import_topics'
         4: from /home/odyslam/forum/discourse/script/import_scripts/nodebb/mongo.rb:71:in `topics'
         3: from /home/odyslam/forum/discourse/script/import_scripts/nodebb/mongo.rb:71:in `map'
         2: from /home/odyslam/forum/discourse/script/import_scripts/nodebb/mongo.rb:71:in `block in topics'
         1: from /home/odyslam/forum/discourse/script/import_scripts/nodebb/mongo.rb:79:in `topic'
/home/odyslam/forum/discourse/script/import_scripts/nodebb/mongo.rb:100:in `post': undefined method `[]' for nil:NilClass (NoMethodError)

如果将来有人对此感兴趣,我已经成功完成了迁移。以下是我的一些经验总结;过程中有几个需要注意的陷阱。

https://odyslam.com/blog/Migrating-from-Nodebb-to-Discourse/

2 个赞
21: from script/import_scripts/nodebb/nodebb.rb:532:in `main`
        20: from /home/nodebb/discourse/script/import_scripts/base.rb:47:in `perform`
        19: from script/import_scripts/nodebb/nodebb.rb:50:in `execute`
        18: from script/import_scripts/nodebb/nodebb.rb:130:in `import_users`
        17: from /home/nodebb/discourse/script/import_scripts/base.rb:264:in `create_users`
        16: from /home/nodebb/discourse/script/import_scripts/base.rb:264:in `each`
        15: from /home/nodebb/discourse/script/import_scripts/base.rb:276:in `block in create_users`
        14: from /home/nodebb/discourse/script/import_scripts/base.rb:393:in `create_user`
        13: from /home/nodebb/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/core_ext/object/try.rb:15:in `try`
        12: from /home/nodebb/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/core_ext/object/try.rb:15:in `public_send`
        11: from script/import_scripts/nodebb/nodebb.rb:164:in `block (2 levels) in import_users`
        10: from script/import_scripts/nodebb/nodebb.rb:211:in `import_profile_picture`
         9: from /home/nodebb/discourse/lib/upload_creator.rb:64:in `create_for`
         8: from /home/nodebb/discourse/lib/distributed_mutex.rb:14:in `synchronize`
         7: from /home/nodebb/discourse/lib/distributed_mutex.rb:29:in `synchronize`
         6: from /home/nodebb/discourse/lib/distributed_mutex.rb:29:in `synchronize`
         5: from /home/nodebb/discourse/lib/distributed_mutex.rb:33:in `block in synchronize`
         4: from /home/nodebb/discourse/lib/upload_creator.rb:83:in `block in create_for`
         3: from /home/nodebb/discourse/lib/upload_creator.rb:501:in `optimize!`
         2: from /home/nodebb/discourse/app/models/optimized_image.rb:181:in `ensure_safe_paths!`
         1: from /home/nodebb/discourse/app/models/optimized_image.rb:181:in `each`
/home/nodebb/discourse/app/models/optimized_image.rb:182:in `block in ensure_safe_paths!': Discourse::InvalidAccess (Discourse::InvalidAccess)

@enigmaty 我遇到了和 @OdysLam @FreeWorLD 一样的错误。

您好!

有人尝试过从最新的 NodeBB 4.9.x 迁移到当前的 Discourse 开发版本吗?

/home/dev/discourse/script/import_scripts/nodebb/mongo.rb:100:in 'NodeBB::Mongo#post': undefined method '[]' for nil (NoMethodError)

      post["timestamp"] = timestamp_to_date(post["timestamp"])
                                                ^^^^^^^^^^^^^
        from /home/dev/discourse/script/import_scripts/nodebb/mongo.rb:95:in 'block in NodeBB::Mongo#posts'
        from /home/dev/discourse/script/import_scripts/nodebb/mongo.rb:95:in 'Array#map'
        from /home/dev/discourse/script/import_scripts/nodebb/mongo.rb:95:in 'NodeBB::Mongo#posts'
        from script/import_scripts/nodebb/nodebb.rb:413:in 'block in ImportScripts::NodeBB#import_posts'
        from /home/dev/discourse/script/import_scripts/base.rb:943:in 'block in ImportScripts::Base#batches'
        from 内部:kernel:168:in 'Kernel#loop'
        from /home/dev/discourse/script/import_scripts/base.rb:942:in 'ImportScripts::Base#batches'
        from script/import_scripts/nodebb/nodebb.rb:412:in 'ImportScripts::NodeBB#import_posts'
        from script/import_scripts/nodebb/nodebb.rb:52:in 'ImportScripts::NodeBB#execute'
        from /home/dev/discourse/script/import_scripts/base.rb:47:in 'ImportScripts::Base#perform'
        from script/import_scripts/nodebb/nodebb.rb:568:in '<main>'

我在开发环境中迁移时遇到了这个错误。
有什么想法吗?

迁移脚本在 2026 年是否仍然受社区支持?
字段 timestamp 存在并且包含有效数据

谢谢。

[quote=“Twissell, post:17, topic:126553”]'NodeBB::Mongo#post': undefined method '[]' for nil

  post["timestamp"] = timestamp_to_date(post["timestamp"])

[/quote]

问题在于 post 是 nil。所以你的配置不正确。它导入用户了吗?

我们可以尝试一下,但这非常有限。你可能需要了解很多工作原理,这些内容可能超出了论坛容易支持的范围。

3 个赞
正在导入用户
      182 / 182 (100.0%)  [437953 items/min]
正在将用户添加到组...

正在导入主题...
      132 / 132 (100.0%)  [3567036 items/min]
正在导入帖子...
/home/dev/discourse/script/import_scripts/nodebb/mongo.rb:100:in 'NodeBB::Mongo#post': undefined method '[]' for nil (NoMethodError)

      post["timestamp"] = timestamp_to_date(post["timestamp"])
                                                ^^^^^^^^^^^^^
        from /home/dev/discourse/script/import_scripts/nodebb/mongo.rb:95:in 'block in NodeBB::Mongo#posts'
        from /home/dev/discourse/script/import_scripts/nodebb/mongo.rb:95:in 'Array#map'
        from /home/dev/discourse/script/import_scripts/nodebb/mongo.rb:95:in 'NodeBB::Mongo#posts'
        from script/import_scripts/nodebb/nodebb.rb:413:in 'block in ImportScripts::NodeBB#import_posts'
        from /home/dev/discourse/script/import_scripts/base.rb:943:in 'block in ImportScripts::Base#batches'
        from 内部:kernel:168:in 'Kernel#loop'
        from /home/dev/discourse/script/import_scripts/base.rb:942:in 'ImportScripts::Base#batches'
        from script/import_scripts/nodebb/nodebb.rb:412:in 'ImportScripts::NodeBB#import_posts'
        from script/import_scripts/nodebb/nodebb.rb:52:in 'ImportScripts::NodeBB#execute'
        from /home/dev/discourse/script/import_scripts/base.rb:47:in 'ImportScripts::Base#perform'
        from script/import_scripts/nodebb/nodebb.rb:568:in '<main>'


根据脚本输出,用户似乎已正确导入,请参见上文。

抱歉,您是真人还是足够聪明的 AI 助手?

那么问题就出在它没有读取帖子数据上。

我猜问题在这里,但不知何故它没有找到帖子。

    def posts(offset = 0, page_size = 2000)
      post_keys = mongo.find(_key: "posts:pid").skip(offset).limit(page_size).pluck(:value)

      post_keys.map { |post_key| post(post_key) }
    end

我是一个真人,我靠支持 Discourse 为生已有十年,编写了许多导入程序,并从各种其他平台导入了一百多个论坛。

如果你想从 AI 助手那里获得帮助,请参阅 https://ask.discourse.com/

6 个赞

我设法让脚本函数正常工作了。

这是我的修改(得益于 Claude Code 的一点帮助 :slight_smile:

 >mongo.rb
   
   def posts(offset = 0, page_size = 1000)
      post_keys = mongo.find(_key: "posts:pid").skip(offset).limit(page_size).pluck(:value)
      post_keys
          .map { |pid| post(pid) }
          .compact  # <-- 丢弃任何 nil 结果(孤立的 pid)
      post_keys.map { |post_key| post(post_key) }
    end

    def post(id)
    post = mongo.find(_key: "post:#{id}").first
    return nil if post.nil? # <-- null 检查
    post["timestamp"] = timestamp_to_date(post["timestamp"])
    if post["upvoted_by"] = mongo.find(_key: "pid:#{id}:upvote").first
        post["upvoted_by"] = mongo.find(_key: "pid:#{id}:upvote").first[:members]
      else
        post["upvoted_by"] = []
      end

      post["pid"] = post["pid"].to_s
      post["deleted"] = post["deleted"].to_s

      post
    end
	
>nodebb.rb

 create_posts(posts, total: post_count, offset: offset) do |post|
        # 如果 post 为 null 则跳过
		# 如果是 merged_post 则跳过
        next if post.nil?
        next if @merged_posts_map[post["pid"]]

        # 如果已删除则跳过
        next if post["deleted"] == "1"

        raw = post["content"]
        post_id = "p#{post["pid"]}"

        next if raw.blank?
        topic = topic_lookup_from_imported_post_id("t#{post["tid"]}")

        unless topic
          puts "未找到 ID 为 #{post["tid"]} 的主题,跳过"
          next
        end	

它现在似乎可以正常工作了。

虽然我不知道从 Discourse 内部架构的角度来看这有多正确,但乍一看它似乎是可行的。

非常欢迎任何改进和优化方面的建议。

1 个赞

提示:不要让它完全静默失败——将此更改为

if post.nil?
  puts "!!! Could not find post #{id}"
  return nil
end

否则,您会想知道为什么您的一半帖子丢失了,并且在数小时的沮丧之后,结果却是这个。

对于以下两个也一样

        next if post.nil?
        next if post["deleted"] == "1"
5 个赞

重写了 mongo.rb 文件中的 posts 方法

def posts(offset = 0, page_size = 1000)
post_keys = mongo.find(_key: “posts:pid”).sort(score: 1).skip(offset).limit(page_size).pluck(:value)
post_keys.map { |pid| post(pid)}.compact
end

在此处输入引用的文本

此方法确保主题内的帖子按正确的时间顺序排序。升序。
注意 sort(score: 1) 调用。