将带有MongoDB的NodeBB论坛迁移到Discourse

如您所知,NodeBB 支持两种数据库后端:Redis 和 MongoDB。Discourse 导入脚本同时支持这两种数据库。本教程将介绍如何迁移使用 MongoDB 作为数据库后端的 NodeBB。我们将使用带有 mongo 适配器的 NodeBB 导入器。如果您的 NodeBB 论坛使用 Redis 作为后端,请遵循 本教程,其中演示了 redis 适配器的使用方法。

计划

  • 准备开发环境。
  • 从生产环境导出数据库。
  • 将生产数据库导入到 Discourse 实例中。
  • 运行导入脚本。

可迁移的内容

  • 用户组
  • 附件
  • 分类
    • 根分类 => 根分类
    • 子分类 & 子子分类 => 子分类
  • 主题与帖子
    • 置顶主题 => 置顶主题
    • 锁定主题 => 关闭主题
    • 主题浏览量
    • 点赞用户
    • 样式、提及、表情符号和附件。
  • 用户(包含以下属性)
    • 个人资料背景
    • 头像
    • 封禁状态
    • 用户名
    • 姓名
    • 电子邮件
    • 管理员状态
    • 个人简介
    • 用户组
    • 网站
    • 所在地
    • 注册时间

准备本地开发环境

如计划所述,我们首先需要准备开发环境。请按照以下指南之一安装 Discourse:

:bulb: 如果您在设置 Discourse 服务器 时遇到任何问题,请参阅 本指南

然后安装 MongoDB 数据库服务器。

Ubuntu 18.04:

$ wget -qO - https://www.mongodb.org/static/pgp/server-4.0.asc | sudo apt-key add -
$ echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
$ sudo apt-get update
$ sudo apt-get install -y mongodb-org
$ sudo service mongod status

更多详情,请参阅 官方指南

Mac OS:

$ brew tap mongodb/brew
$ brew install mongodb-community@4.0
$ brew services start mongodb-community@4.0
$ brew services status mongodb-community@4.0

更多详情,请参阅 官方指南

Windows 10:

下载安装程序并将 MongoDB 安装为 Windows 服务:

https://fastdl.mongodb.org/win32/mongodb-win32-x86_64-2008plus-ssl-4.0.12-signed.msi

以管理员权限运行 cmd 以检查 MongoDB 服务器是否正常工作:

"C:\Program Files\MongoDB\Server\4.0\bin\mongo.exe"

更多详情,请参阅 官方指南

此环境将作为我们的 Discourse 服务器

导出生产数据库转储

关闭您的 NodeBB 论坛(生产服务器)。

$ cd /path_to_nodebb
$ ./nodebb stop

关闭您的数据库:

$ sudo service mongodb stop

备份您的数据库:

$ mongodump --out ~/my_backup_path/

上一条命令的输出可能如下所示:

2019-08-16T15:17:09.845+0300 done dumping admin.system.users (1 document)
2019-08-16T15:17:09.846+0300 done dumping admin.system.version (2 documents)
2019-08-16T15:17:09.849+0300 done dumping nodebb.sessions (10 documents)
2019-08-16T15:17:09.850+0300 done dumping nodebb.socket.io (215 documents)
2019-08-16T15:17:09.854+0300 done dumping nodebb.objects (1488 documents)

请注意,备份实际上是一个目录,而不仅仅是一个文件。

:bulb: 您可以通过在 mongo CLI 中运行 use myDatabase 然后运行 db.stats().dataSize; 来检查数据库的大小。返回的值将以字节为单位。

备份您的论坛资源:

$ cd /path_to_nodebb_root_directory/
$ tar -czf ./uploads.tar.gz ./public/uploads

在拥有数据库和资源后,请将它们复制到 Discourse 服务器

导入数据库

现在我们已经有了数据库,可以将其导入到本地 MongoDB 实例中:

$ mongorestore ~/path_of_my_backup_directory/

更多选项,请参阅 官方指南

接下来,您需要解压 uploads.tar.gz,以便导入器能够导入资源:

$ tar xvzf uploads.tar.gz

运行导入脚本

现在数据库和资源已就位,我们可以准备运行导入脚本。在此之前,我们需要编辑 NodeBB 导入脚本以适应我们的需求。

这是您的 NodeBB 上传文件夹的路径:

ATTACHMENT_DIR = '/absolute_path/uploads'

接下来,我们需要告诉导入器使用 mongo 适配器而不是 redis 适配器:

adapter = NodeBB::Mongo
@client = adapter.new('mongodb://127.0.0.1:27017/nodebb')

# adapter = NodeBB::Redis
# @client = adapter.new(
# host: "localhost",
# port: "6379",
# db: 14
# )

在干净的 Discourse 环境中并支持 mongo gem 的情况下运行导入器:

$ cd ~/discourse
$ echo "gem 'mongo'" >> Gemfile
$ bundle install
$ bundle exec rake db:drop db:create db:migrate
$ bundle exec ruby script/import_scripts/nodebb/nodebb.rb

导入器将连接到 MongoDB 实例,并将所有内容迁移到 Discourse。

导入完成后,启动 Discourse:

$ bundle exec rails server

启动 Sidekiq 以处理迁移的数据:

$ bundle exec sidekiq

您可以在 http://localhost:3000/sidekiq/queues 监控进度。

按照此 教程 执行 Discourse 备份,并将其上传到您的 Discourse 生产服务器。

:tada:

如果您对该过程有任何疑问,我很乐意提供帮助。

祝迁移顺利 :grinning:

10 个赞

Great work really. While I was trying, I am getting below error. Any idea what I might be doing wrong here?

importing groups
   10 / 10 (100.0%)  [474765 items/min]    
importing top level categories...
    8 / 8 (100.0%)  [437896 items/min]    
importing child categories...
   68 / 68 (100.0%)  [774048 items/min]  
importing users
 5534 / 5534 (100.0%)  [355520 items/min]    
adding users to groups...

importing topics...
Traceback (most recent call last):
	12: from script/import_scripts/nodebb/nodebb.rb:532:in `<main>'
	11: from /home/fsuzer/discourse/script/import_scripts/base.rb:47:in `perform'
	10: from script/import_scripts/nodebb/nodebb.rb:52:in `execute'
	 9: from script/import_scripts/nodebb/nodebb.rb:336:in `import_topics'
	 8: from /home/fsuzer/discourse/script/import_scripts/base.rb:877:in `batches'
	 7: from /home/fsuzer/discourse/script/import_scripts/base.rb:877:in `loop'
	 6: from /home/fsuzer/discourse/script/import_scripts/base.rb:878:in `block in batches'
	 5: from script/import_scripts/nodebb/nodebb.rb:337:in `block in import_topics'
	 4: from /home/fsuzer/discourse/script/import_scripts/nodebb/mongo.rb:71:in `topics'
	 3: from /home/fsuzer/discourse/script/import_scripts/nodebb/mongo.rb:71:in `map'
	 2: from /home/fsuzer/discourse/script/import_scripts/nodebb/mongo.rb:71:in `block in topics'
	 1: from /home/fsuzer/discourse/script/import_scripts/nodebb/mongo.rb:79:in `topic'
/home/fsuzer/discourse/script/import_scripts/nodebb/mongo.rb:100:in `post': undefined method `[]' for nil:NilClass (NoMethodError)

Some field from the topic is empty. Or perhaps the topic query returned nothing.

1 个赞

有人能帮我吗?

错误:在您的 Gemfile 列出的任何 gem 源中找不到 gem ‘mongo’。

执行 $ bundle install

如果你跳过了以下步骤,就会发生这种情况:

请确保你的 Gemfile 中包含这一行:

gem 'mongo'

然后运行:

$ bundle install
1 个赞

感谢您的帮助。但这行内容在我的文件中已经存在了。我也已经执行了之前告知的步骤。

不知这是否有帮助,但当我运行 gem list mongo 时,它并未被列出。

你的 Gemfile 听起来有点奇怪。请问能分享一下完整内容吗?

你的 Gemfile 有问题。它应该包含如下内容:

此外,还应包括:

gem 'mongo'

嘿,

感谢这份指南!当我运行迁移脚本时,在脚本执行到导入用户这一步,我得到了以下输出:

Traceback (most recent call last):
        21: from script/import_scripts/nodebb/nodebb.rb:532:in `<main>'
        20: from /home/odyslam/forum/discourse/script/import_scripts/base.rb:47:in `perform'
        19: from script/import_scripts/nodebb/nodebb.rb:50:in `execute'
        18: from script/import_scripts/nodebb/nodebb.rb:130:in `import_users'
        17: from /home/odyslam/forum/discourse/script/import_scripts/base.rb:262:in `create_users'
        16: from /home/odyslam/forum/discourse/script/import_scripts/base.rb:262:in `each'
        15: from /home/odyslam/forum/discourse/script/import_scripts/base.rb:274:in `block in create_users'
        14: from /home/odyslam/forum/discourse/script/import_scripts/base.rb:391:in `create_user'
        13: from /home/odyslam/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/core_ext/object/try.rb:15:in `try'
        12: from /home/odyslam/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/core_ext/object/try.rb:15:in `public_send'
        11: from script/import_scripts/nodebb/nodebb.rb:164:in `block (2 levels) in import_users'
        10: from script/import_scripts/nodebb/nodebb.rb:211:in `import_profile_picture'
         9: from /home/odyslam/forum/discourse/lib/upload_creator.rb:45:in `create_for'
         8: from /home/odyslam/forum/discourse/lib/distributed_mutex.rb:14:in `synchronize'
         7: from /home/odyslam/forum/discourse/lib/distributed_mutex.rb:29:in `synchronize'
         6: from /home/odyslam/forum/discourse/lib/distributed_mutex.rb:29:in `synchronize'
         5: from /home/odyslam/forum/discourse/lib/distributed_mutex.rb:33:in `block in synchronize'
         4: from /home/odyslam/forum/discourse/lib/upload_creator.rb:66:in `block in create_for'
         3: from /home/odyslam/forum/discourse/lib/upload_creator.rb:381:in `optimize!'
         2: from /home/odyslam/forum/discourse/app/models/optimized_image.rb:178:in `ensure_safe_paths!'
         1: from /home/odyslam/forum/discourse/app/models/optimized_image.rb:178:in `each'
/home/odyslam/forum/discourse/app/models/optimized_image.rb:179:in `block in ensure_safe_paths!': Discourse::InvalidAccess (Discourse::InvalidAccess)

我通过注释掉上传头像的部分绕过了这个问题,这倒不是问题。

现在我在导入话题时遇到了 nil 错误。:frowning:

Traceback (most recent call last):
        12: from script/import_scripts/nodebb/nodebb.rb:532:in `<main>'
        11: from /home/odyslam/forum/discourse/script/import_scripts/base.rb:47:in `perform'
        10: from script/import_scripts/nodebb/nodebb.rb:52:in `execute'
         9: from script/import_scripts/nodebb/nodebb.rb:336:in `import_topics'
         8: from /home/odyslam/forum/discourse/script/import_scripts/base.rb:862:in `batches'
         7: from /home/odyslam/forum/discourse/script/import_scripts/base.rb:862:in `loop'
         6: from /home/odyslam/forum/discourse/script/import_scripts/base.rb:863:in `block in batches'
         5: from script/import_scripts/nodebb/nodebb.rb:337:in `block in import_topics'
         4: from /home/odyslam/forum/discourse/script/import_scripts/nodebb/mongo.rb:71:in `topics'
         3: from /home/odyslam/forum/discourse/script/import_scripts/nodebb/mongo.rb:71:in `map'
         2: from /home/odyslam/forum/discourse/script/import_scripts/nodebb/mongo.rb:71:in `block in topics'
         1: from /home/odyslam/forum/discourse/script/import_scripts/nodebb/mongo.rb:79:in `topic'
/home/odyslam/forum/discourse/script/import_scripts/nodebb/mongo.rb:100:in `post': undefined method `[]' for nil:NilClass (NoMethodError)

如果将来有人对此感兴趣,我已经成功完成了迁移。以下是我的一些经验总结;过程中有几个需要注意的陷阱。

https://odyslam.com/blog/Migrating-from-Nodebb-to-Discourse/

2 个赞
21: from script/import_scripts/nodebb/nodebb.rb:532:in `main`
        20: from /home/nodebb/discourse/script/import_scripts/base.rb:47:in `perform`
        19: from script/import_scripts/nodebb/nodebb.rb:50:in `execute`
        18: from script/import_scripts/nodebb/nodebb.rb:130:in `import_users`
        17: from /home/nodebb/discourse/script/import_scripts/base.rb:264:in `create_users`
        16: from /home/nodebb/discourse/script/import_scripts/base.rb:264:in `each`
        15: from /home/nodebb/discourse/script/import_scripts/base.rb:276:in `block in create_users`
        14: from /home/nodebb/discourse/script/import_scripts/base.rb:393:in `create_user`
        13: from /home/nodebb/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/core_ext/object/try.rb:15:in `try`
        12: from /home/nodebb/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/core_ext/object/try.rb:15:in `public_send`
        11: from script/import_scripts/nodebb/nodebb.rb:164:in `block (2 levels) in import_users`
        10: from script/import_scripts/nodebb/nodebb.rb:211:in `import_profile_picture`
         9: from /home/nodebb/discourse/lib/upload_creator.rb:64:in `create_for`
         8: from /home/nodebb/discourse/lib/distributed_mutex.rb:14:in `synchronize`
         7: from /home/nodebb/discourse/lib/distributed_mutex.rb:29:in `synchronize`
         6: from /home/nodebb/discourse/lib/distributed_mutex.rb:29:in `synchronize`
         5: from /home/nodebb/discourse/lib/distributed_mutex.rb:33:in `block in synchronize`
         4: from /home/nodebb/discourse/lib/upload_creator.rb:83:in `block in create_for`
         3: from /home/nodebb/discourse/lib/upload_creator.rb:501:in `optimize!`
         2: from /home/nodebb/discourse/app/models/optimized_image.rb:181:in `ensure_safe_paths!`
         1: from /home/nodebb/discourse/app/models/optimized_image.rb:181:in `each`
/home/nodebb/discourse/app/models/optimized_image.rb:182:in `block in ensure_safe_paths!': Discourse::InvalidAccess (Discourse::InvalidAccess)

@enigmaty 我遇到了和 @OdysLam @FreeWorLD 一样的错误。

您好!

有人尝试过从最新的 NodeBB 4.9.x 迁移到当前的 Discourse 开发版本吗?

/home/dev/discourse/script/import_scripts/nodebb/mongo.rb:100:in 'NodeBB::Mongo#post': undefined method '[]' for nil (NoMethodError)

      post["timestamp"] = timestamp_to_date(post["timestamp"])
                                                ^^^^^^^^^^^^^
        from /home/dev/discourse/script/import_scripts/nodebb/mongo.rb:95:in 'block in NodeBB::Mongo#posts'
        from /home/dev/discourse/script/import_scripts/nodebb/mongo.rb:95:in 'Array#map'
        from /home/dev/discourse/script/import_scripts/nodebb/mongo.rb:95:in 'NodeBB::Mongo#posts'
        from script/import_scripts/nodebb/nodebb.rb:413:in 'block in ImportScripts::NodeBB#import_posts'
        from /home/dev/discourse/script/import_scripts/base.rb:943:in 'block in ImportScripts::Base#batches'
        from 内部:kernel:168:in 'Kernel#loop'
        from /home/dev/discourse/script/import_scripts/base.rb:942:in 'ImportScripts::Base#batches'
        from script/import_scripts/nodebb/nodebb.rb:412:in 'ImportScripts::NodeBB#import_posts'
        from script/import_scripts/nodebb/nodebb.rb:52:in 'ImportScripts::NodeBB#execute'
        from /home/dev/discourse/script/import_scripts/base.rb:47:in 'ImportScripts::Base#perform'
        from script/import_scripts/nodebb/nodebb.rb:568:in '<main>'

我在开发环境中迁移时遇到了这个错误。
有什么想法吗?

迁移脚本在 2026 年是否仍然受社区支持?
字段 timestamp 存在并且包含有效数据

谢谢。

[quote=“Twissell, post:17, topic:126553”]'NodeBB::Mongo#post': undefined method '[]' for nil

  post["timestamp"] = timestamp_to_date(post["timestamp"])

[/quote]

问题在于 post 是 nil。所以你的配置不正确。它导入用户了吗?

我们可以尝试一下,但这非常有限。你可能需要了解很多工作原理,这些内容可能超出了论坛容易支持的范围。

3 个赞
正在导入用户
      182 / 182 (100.0%)  [437953 items/min]
正在将用户添加到组...

正在导入主题...
      132 / 132 (100.0%)  [3567036 items/min]
正在导入帖子...
/home/dev/discourse/script/import_scripts/nodebb/mongo.rb:100:in 'NodeBB::Mongo#post': undefined method '[]' for nil (NoMethodError)

      post["timestamp"] = timestamp_to_date(post["timestamp"])
                                                ^^^^^^^^^^^^^
        from /home/dev/discourse/script/import_scripts/nodebb/mongo.rb:95:in 'block in NodeBB::Mongo#posts'
        from /home/dev/discourse/script/import_scripts/nodebb/mongo.rb:95:in 'Array#map'
        from /home/dev/discourse/script/import_scripts/nodebb/mongo.rb:95:in 'NodeBB::Mongo#posts'
        from script/import_scripts/nodebb/nodebb.rb:413:in 'block in ImportScripts::NodeBB#import_posts'
        from /home/dev/discourse/script/import_scripts/base.rb:943:in 'block in ImportScripts::Base#batches'
        from 内部:kernel:168:in 'Kernel#loop'
        from /home/dev/discourse/script/import_scripts/base.rb:942:in 'ImportScripts::Base#batches'
        from script/import_scripts/nodebb/nodebb.rb:412:in 'ImportScripts::NodeBB#import_posts'
        from script/import_scripts/nodebb/nodebb.rb:52:in 'ImportScripts::NodeBB#execute'
        from /home/dev/discourse/script/import_scripts/base.rb:47:in 'ImportScripts::Base#perform'
        from script/import_scripts/nodebb/nodebb.rb:568:in '<main>'


根据脚本输出,用户似乎已正确导入,请参见上文。

抱歉,您是真人还是足够聪明的 AI 助手?

那么问题就出在它没有读取帖子数据上。

我猜问题在这里,但不知何故它没有找到帖子。

    def posts(offset = 0, page_size = 2000)
      post_keys = mongo.find(_key: "posts:pid").skip(offset).limit(page_size).pluck(:value)

      post_keys.map { |post_key| post(post_key) }
    end

我是一个真人,我靠支持 Discourse 为生已有十年,编写了许多导入程序,并从各种其他平台导入了一百多个论坛。

如果你想从 AI 助手那里获得帮助,请参阅 https://ask.discourse.com/

6 个赞

我设法让脚本函数正常工作了。

这是我的修改(得益于 Claude Code 的一点帮助 :slight_smile:

 >mongo.rb
   
   def posts(offset = 0, page_size = 1000)
      post_keys = mongo.find(_key: "posts:pid").skip(offset).limit(page_size).pluck(:value)
      post_keys
          .map { |pid| post(pid) }
          .compact  # <-- 丢弃任何 nil 结果(孤立的 pid)
      post_keys.map { |post_key| post(post_key) }
    end

    def post(id)
    post = mongo.find(_key: "post:#{id}").first
    return nil if post.nil? # <-- null 检查
    post["timestamp"] = timestamp_to_date(post["timestamp"])
    if post["upvoted_by"] = mongo.find(_key: "pid:#{id}:upvote").first
        post["upvoted_by"] = mongo.find(_key: "pid:#{id}:upvote").first[:members]
      else
        post["upvoted_by"] = []
      end

      post["pid"] = post["pid"].to_s
      post["deleted"] = post["deleted"].to_s

      post
    end
	
>nodebb.rb

 create_posts(posts, total: post_count, offset: offset) do |post|
        # 如果 post 为 null 则跳过
		# 如果是 merged_post 则跳过
        next if post.nil?
        next if @merged_posts_map[post["pid"]]

        # 如果已删除则跳过
        next if post["deleted"] == "1"

        raw = post["content"]
        post_id = "p#{post["pid"]}"

        next if raw.blank?
        topic = topic_lookup_from_imported_post_id("t#{post["tid"]}")

        unless topic
          puts "未找到 ID 为 #{post["tid"]} 的主题,跳过"
          next
        end	

它现在似乎可以正常工作了。

虽然我不知道从 Discourse 内部架构的角度来看这有多正确,但乍一看它似乎是可行的。

非常欢迎任何改进和优化方面的建议。

1 个赞

提示:不要让它完全静默失败——将此更改为

if post.nil?
  puts "!!! Could not find post #{id}"
  return nil
end

否则,您会想知道为什么您的一半帖子丢失了,并且在数小时的沮丧之后,结果却是这个。

对于以下两个也一样

        next if post.nil?
        next if post["deleted"] == "1"
5 个赞

重写了 mongo.rb 文件中的 posts 方法

def posts(offset = 0, page_size = 1000)
post_keys = mongo.find(_key: “posts:pid”).sort(score: 1).skip(offset).limit(page_size).pluck(:value)
post_keys.map { |pid| post(pid)}.compact
end

在此处输入引用的文本

此方法确保主题内的帖子按正确的时间顺序排序。升序。
注意 sort(score: 1) 调用。