PHPBB 到 Discourse 迁移速度优化

支持 迁移
1

嘿，我从 2004 年就开始使用 phpBB，现在正在迁移一个规模不小的论坛。目前迁移进行了两天，进度为 303139/2167314（14%），速度约为每分钟 446 条。有什么方法可以加快速度吗？我使用的是 DigitalOcean，CPU 使用率仅为 4%。Redis 一直超时。我已经关闭了附件功能。

2

一些有用的细节将有助于缩小问题范围：

  • Droplet 规格：内存、vCPU 数量、磁盘类型/大小
  • 您正在运行的确切导入命令
  • 您的 app.yml 中与资源相关的设置（已移除机密信息），特别是：
    • UNICORN_WORKERS
    • db_shared_buffers
    • db_work_mem
  • 导入是否在标准的 Discourse Docker 容器内运行
  • 任何 Redis 超时消息或日志片段
  • 数据库是位于同一容器内的本地数据库，还是外部数据库

需要说明的一点是：UNICORN_WORKERS 可能不会显著加快导入速度，因为 phpBB 导入器并非主要通过 Web 工作进程提供服务。增加该值可能只会消耗更多内存。PostgreSQL 的内存设置和磁盘 I/O 更可能是关键因素。

由于 CPU 使用率仅为 4% 左右，问题可能是受限于 I/O、数据库性能，或正在等待 Redis 响应，而非 CPU 不足。Redis 超时值得单独调查。

Mattash (Matt) 3

image

4

我的更新日志
script/import_scripts/phpbb3.rb:15:in ‘’
302570 / 2167314 ( 14.0%) [448 项/分钟] 创建帖子 304683 时发生异常。已跳过。
等待了 1.0 秒
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/redis-client-0.28.0/lib/redis_client/ruby_connection/buffered_io.rb:214:in ‘block in RedisClient::RubyConnection::BufferedIO#fill_buffer’
internal:kernel:168:in ‘Kernel#loop’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/redis-client-0.28.0/lib/redis_client/ruby_connection/buffered_io.rb:197:in ‘RedisClient::RubyConnection::BufferedIO#fill_buffer’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/redis-client-0.28.0/lib/redis_client/ruby_connection/buffered_io.rb:187:in ‘RedisClient::RubyConnection::BufferedIO#ensure_remaining’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/redis-client-0.28.0/lib/redis_client/ruby_connection/buffered_io.rb:152:in ‘RedisClient::RubyConnection::BufferedIO#getbyte’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/redis-client-0.28.0/lib/redis_client/ruby_connection/resp3.rb:113:in ‘RedisClient::RESP3.parse’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/redis-client-0.28.0/lib/redis_client/ruby_connection/resp3.rb:50:in ‘RedisClient::RESP3.load’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/redis-client-0.28.0/lib/redis_client/ruby_connection.rb:97:in ‘RedisClient::RubyConnection#read’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/redis-client-0.28.0/lib/redis_client/connection_mixin.rb:37:in ‘RedisClient::ConnectionMixin#call’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/redis-client-0.28.0/lib/redis_client.rb:374:in ‘block (2 levels) in RedisClient#call_v’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/redis-client-0.28.0/lib/redis_client/middlewares.rb:16:in ‘RedisClient::BasicMiddleware#call’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/redis-client-0.28.0/lib/redis_client.rb:373:in ‘block in RedisClient#call_v’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/redis-client-0.28.0/lib/redis_client.rb:781:in ‘RedisClient#ensure_connected’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/redis-client-0.28.0/lib/redis_client.rb:372:in ‘RedisClient#call_v’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/redis-5.4.0/lib/redis/client.rb:90:in ‘Redis::Client#call_v’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-mini-profiler-4.0.1/lib/mini_profiler/profiling_methods.rb:90:in ‘block in Redis::Client#profile_method’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/redis-5.4.0/lib/redis.rb:152:in ‘block in Redis#send_command’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/redis-5.4.0/lib/redis.rb:151:in ‘Monitor#synchronize’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/redis-5.4.0/lib/redis.rb:151:in ‘Redis#send_command’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/redis-5.4.0/lib/redis/commands/scripting.rb:110:in ‘Redis::Commands::Scripting#_eval’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/redis-5.4.0/lib/redis/commands/scripting.rb:97:in ‘Redis::Commands::Scripting#evalsha’
/var/www/discourse/lib/discourse_redis.rb:38:in ‘Kernel#public_send’
/var/www/discourse/lib/discourse_redis.rb:38:in ‘block in DiscourseRedis#method_missing’
/var/www/discourse/lib/discourse_redis.rb:29:in ‘DiscourseRedis.ignore_readonly’
/var/www/discourse/lib/discourse_redis.rb:38:in ‘DiscourseRedis#method_missing’
/var/www/discourse/lib/discourse_redis.rb:270:in ‘DiscourseRedis::EvalHelper#eval’
/var/www/discourse/lib/distributed_mutex.rb:82:in ‘DistributedMutex#get_lock’
/var/www/discourse/lib/distributed_mutex.rb:50:in ‘block in DistributedMutex#synchronize’
/var/www/discourse/lib/distributed_mutex.rb:49:in ‘Thread::Mutex#synchronize’
/var/www/discourse/lib/distributed_mutex.rb:49:in ‘DistributedMutex#synchronize’
/var/www/discourse/lib/distributed_mutex.rb:34:in ‘DistributedMutex.synchronize’
/var/www/discourse/lib/post_creator.rb:410:in ‘PostCreator#transaction’
/var/www/discourse/lib/post_creator.rb:200:in ‘PostCreator#create’
/var/www/discourse/script/import_scripts/base.rb:611:in ‘ImportScripts::Base#create_post’
/var/www/discourse/script/import_scripts/base.rb:555:in ‘block in ImportScripts::Base#create_posts’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/mysql2/alias_method.rb:8:in ‘Mysql2::Result#each’
/var/www/discourse/vendor/bundle/ruby/3.4.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/mysql2/alias_method.rb:8:in ‘Mysql2::Result#each’
/var/www/discourse/script/import_scripts/base.rb:542:in ‘ImportScripts::Base#create_posts’
/var/www/discourse/script/import_scripts/phpbb3/importer.rb:212:in ‘block in ImportScripts::PhpBB3::Importer#import_posts’
/var/www/discourse/script/import_scripts/base.rb:943:in ‘block in ImportScripts::Base#batches’
internal:kernel:168:in ‘Kernel#loop’
/var/www/discourse/script/import_scripts/base.rb:942:in ‘ImportScripts::Base#batches’
/var/www/discourse/script/import_scripts/phpbb3/importer.rb:293:in ‘ImportScripts::PhpBB3::Importer#batches’
/var/www/discourse/script/import_scripts/phpbb3/importer.rb:208:in ‘ImportScripts::PhpBB3::Importer#import_posts’
/var/www/discourse/script/import_scripts/phpbb3/importer.rb:38:in ‘ImportScripts::PhpBB3::Importer#execute’
/var/www/discourse/script/import_scripts/base.rb:47:in ‘ImportScripts::Base#perform’
/var/www/discourse/script/import_scripts/phpbb3/importer.rb:22:in ‘ImportScripts::PhpBB3::Importer#perform’
script/import_scripts/phpbb3.rb:35:in ‘module:PhpBB3’
script/import_scripts/phpbb3.rb:16:in ‘module:ImportScripts’
script/import_scripts/phpbb3.rb:15:in ‘’
333919 / 2167314 ( 15.4%) [441 项/分钟]

5

看起来确实能正常工作，没什么可抱怨的。
到目前为止

已停止 Unicorn

已禁用附件和头像

6

该 Droplet 的规格对于此类导入任务来说看起来相当充足，因此我不认为 UNICORN_WORKERS 会是主要的瓶颈。

几点观察：

  • 对于仅用于导入或预演的容器来说，停止 Unicorn 可能没问题，但这不太可能显著提升导入器本身的速度。
  • 禁用附件和头像应该会移除两个较慢的环节，因此如果导入速度仍仅为每分钟约 440 条帖子，瓶颈可能在于数据库 I/O、源端 phpBB 的 MySQL 查询侧，或是 Redis/PostgreSQL 的等待时间。
  • 需要仔细关注以下日志行：

Exception while creating post 304683. Skipping.

尽管导入仍在继续，但这意味着至少该条帖子被跳过了。如果这种情况反复发生，最终迁移结果可能会缺失部分帖子。

Redis 部分看起来是在 Discourse 的分布式互斥锁期间，PostCreator 创建帖子时发生了 1 秒的 Redis 客户端超时。建议检查 Redis 是否真的过载，或者在长时间导入过程中超时设置是否过于激进。

接下来有用的检查包括：

free -h

df -h

iostat -xz 1

vmstat 1

redis-cli INFO memory

redis-cli INFO stats

redis-cli SLOWLOG GET 20

另外，phpBB 的 MySQL 数据库是位于同一台 Droplet 上的本地数据库，还是通过网络读取？由于堆栈跟踪显示正在遍历 mysql2，源数据库缓慢或磁盘性能差可能导致 CPU 使用率偏低，而导入器却在等待。

按当前速度，从 333919 / 2167314 以每分钟约 441 项的速度计算，您大约还有 69 小时，因此导入可能会完成，但我主要担心的是这些 Redis 超时异常是否导致了帖子被跳过。

我不建议提高 UNICORN_WORKERS 的数量。在仅用于导入的运行中，Unicorn 基本无关紧要。关键问题在于导入虽然在继续，但正在跳过帖子，这才是需要重点关注的地方。