Rebake 失败:如何诊断和修复?

你好,

在禁用论坛的 CDN 后,我运行了 rake posts:rebake 命令。
在重新烘焙了几万条帖子后,该命令失败了。

搜索其中一条错误消息(“PQsocket() can’t get socket descriptor”)后,我找到了 Rake:rebake crashes with errors: PG::ConnectionBad: PQsocket - #8 by j127 这个链接,其中提到增加交换空间(swap)可以解决问题。
我不确定是否属于同样的问题,但我尝试了该方法,重新烘焙仍然失败。增加交换空间后,也许在失败前能处理的帖子数量比以前更多,但我没有记录之前失败时具体处理了多少条帖子。

具体情况如下:

  1. 重新烘焙开始后,在处理约 20 万条帖子(总共 160 万条)后崩溃。
    root@Unicyclist-app:/var/www/discourse# rake posts:rebake
    Rebaking post markdown for 'default'
       202930 / 1634878 ( 12.4%)
    Failed to rebake (topic_id: 196639, post_id: 1470519)
    PQconsumeInput() server closed the connection unexpectedly
            This probably means the server terminated abnormally
            before or while processing the request.
    
  2. 出现该消息后,随后的数百条帖子会立即显示以下错误:
       202931 / 1634878 ( 12.4%)
    Failed to rebake (topic_id: 196638, post_id: 1470518)
    PQsocket() can't get socket descriptor
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `async_exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:167:in `perform_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:556:in `block (2 levels) in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1017:in `block in with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:986:in `with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:555:in `block in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1137:in `log'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:554:in `raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:591:in `internal_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:547:in `internal_exec_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:693:in `select'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:73:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/query_cache.rb:262:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:68:in `_query_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:53:in `block in find_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:52:in `find_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/statement_cache.rb:152:in `execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/core.rb:454:in `block in cached_find_by'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/core.rb:442:in `cached_find_by'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/core.rb:327:in `find_by'
    /var/www/discourse/lib/pretty_text/helpers.rb:20:in `avatar_template'
    /var/www/discourse/lib/pretty_text.rb:255:in `call'
    /var/www/discourse/lib/pretty_text.rb:255:in `eval'
    /var/www/discourse/lib/pretty_text.rb:255:in `block in markdown'
    /var/www/discourse/lib/pretty_text.rb:695:in `block in protect'
    /var/www/discourse/lib/pretty_text.rb:695:in `synchronize'
    /var/www/discourse/lib/pretty_text.rb:695:in `protect'
    /var/www/discourse/lib/pretty_text.rb:192:in `markdown'
    /var/www/discourse/lib/pretty_text.rb:318:in `cook'
    /var/www/discourse/app/models/post_analyzer.rb:30:in `cook'
    /var/www/discourse/app/models/post.rb:359:in `cook'
    /var/www/discourse/app/models/post.rb:836:in `rebake!'
    /var/www/discourse/lib/tasks/posts.rake:145:in `rebake_post'
    /var/www/discourse/lib/tasks/posts.rake:132:in `block (2 levels) in rebake_posts'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
    /var/www/discourse/lib/tasks/posts.rake:131:in `block in rebake_posts'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/lib/tasks/posts.rake:126:in `rebake_posts'
    /var/www/discourse/lib/tasks/posts.rake:111:in `block in rebake_posts_all_sites'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `each_connection'
    /var/www/discourse/lib/tasks/posts.rake:111:in `rebake_posts_all_sites'
    /var/www/discourse/lib/tasks/posts.rake:7:in `block in <main>'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `block in execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `each'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:199:in `synchronize'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:188:in `invoke'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:188:in `invoke_task'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:138:in `block (2 levels) in top_level'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:138:in `each'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:138:in `block in top_level'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:147:in `run_with_threads'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:132:in `top_level'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:83:in `block in run'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:214:in `standard_exception_handling'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:80:in `run'
    bin/rake:13:in `<top (required)>'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:59:in `load'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:59:in `kernel_load'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:23:in `run'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:452:in `exec'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor.rb:538:in `dispatch'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:35:in `dispatch'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:29:in `start'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/exe/bundle:28:in `block in <top (required)>'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/exe/bundle:20:in `<top (required)>'
    /usr/local/bin/bundle:25:in `load'
    /usr/local/bin/bundle:25:in `<main>'
    
  3. 最终多次出现以下错误并结束:
    Caused by:
    ActiveRecord::ConnectionNotEstablished: PQsocket() can't get socket descriptor (ActiveRecord::ConnectionNotEstablished)
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `async_exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:167:in `perform_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:556:in `block (2 levels) in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1017:in `block in with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:986:in `with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:555:in `block in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1137:in `log'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:554:in `raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:591:in `internal_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:547:in `internal_exec_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:693:in `select'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:73:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/query_cache.rb:262:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:68:in `_query_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1459:in `block (2 levels) in exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1458:in `block in exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1442:in `exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1420:in `block in exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1414:in `exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/goldiloader-6.0.0/lib/goldiloader/active_record_patches.rb:40:in `exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1191:in `load'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:353:in `records'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
    /var/www/discourse/lib/tasks/posts.rake:131:in `block in rebake_posts'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/lib/tasks/posts.rake:126:in `rebake_posts'
    /var/www/discourse/lib/tasks/posts.rake:111:in `block in rebake_posts_all_sites'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `each_connection'
    /var/www/discourse/lib/tasks/posts.rake:111:in `rebake_posts_all_sites'
    /var/www/discourse/lib/tasks/posts.rake:7:in `block in <main>'
    /usr/local/bin/bundle:25:in `load'
    /usr/local/bin/bundle:25:in `<main>'
    
    Caused by:
    PG::ConnectionBad: PQsocket() can't get socket descriptor (PG::ConnectionBad)
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `async_exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:167:in `perform_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:556:in `block (2 levels) in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1017:in `block in with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:986:in `with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:555:in `block in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1137:in `log'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:554:in `raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:591:in `internal_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:547:in `internal_exec_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:693:in `select'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:73:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/query_cache.rb:262:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:68:in `_query_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1459:in `block (2 levels) in exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1458:in `block in exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1442:in `exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1420:in `block in exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1414:in `exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/goldiloader-6.0.0/lib/goldiloader/active_record_patches.rb:40:in `exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1191:in `load'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:353:in `records'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
    /var/www/discourse/lib/tasks/posts.rake:131:in `block in rebake_posts'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/lib/tasks/posts.rake:126:in `rebake_posts'
    /var/www/discourse/lib/tasks/posts.rake:111:in `block in rebake_posts_all_sites'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `each_connection'
    /var/www/discourse/lib/tasks/posts.rake:111:in `rebake_posts_all_sites'
    /var/www/discourse/lib/tasks/posts.rake:7:in `block in <main>'
    /usr/local/bin/bundle:25:in `load'
    /usr/local/bin/bundle:25:in `<main>'
    Tasks: TOP => posts:rebake
    (See full trace by running task with --trace)
    root@Unicyclist-app:/var/www/discourse#
    

以下是重新烘焙期间 free -h 的输出(重新烘焙过程中任何时刻的数值大致相同):

               total        used        free      shared  buff/cache   available
Mem:           7.6Gi       4.6Gi       231Mi       2.1Gi       2.8Gi       709Mi
Swap:          2.0Gi       1.8Gi       194Mi

我还有大量可用空间(之前几次没有,但我特意释放了一些空间以防万一,但这并未改变任何情况)。

过去几年里,我曾多次对该论坛的所有帖子进行过重新烘焙,从未遇到过任何问题。论坛规模增长不大,服务器规格也与之前相同。

我还在寻求这方面的帮助 :face_with_tongue:

我将交换空间从 2 GB 增加到 4 GB 后,重新烘焙成功了。

所以,我猜是内存问题,尽管我对它需要的内存量感到有些惊讶(8 GB 内存 + 4 GB 交换空间),特别是考虑到以前的重新烘焙在没有交换空间和相同内存量的情况下都没有问题。

如果我想监控内存使用情况并检查是否存在任何问题,我应该怎么做? 哪个指标对估算内存需求最重要? 每日页面浏览量(5000-8000 + 每月一两次机器人高峰)? 数据库大小(23 GB)? :person_shrugging: :face_with_raised_eyebrow:

在重建期间的内存使用量远大于正常操作期间。看起来重新烘焙(rebaking)也会产生很大的需求。如果是这样,任何类型的定期监控都不会增加太多价值:监控需要在那些高峰期进行,而幸运的是,这些高峰期恰好发生在管理员执行某些特定操作时。

当我运行在较小、更边缘的机器配置上时,我会打开第二个终端窗口,通过 ssh 连接到我的服务器并运行
vmstat 5
它会记录内存使用量的起伏情况。观察 swpd 列并与您配置的交换空间进行比较。通常故障会突然发生,而不是逐渐发生,所以即使查看短期趋势也帮助不大。

如果您有足够的磁盘空间,拥有大量的交换空间是完全没有坏处的——可以是 RAM 的一半,甚至是与 RAM 一样多。它在这里是为了应对高峰。您不希望在正常使用期间看到交换/分页活动。同样,您可以使用 vmstat 5 5 来获取分页活动的短期快照(在 siso 列中)。

这是一个例子:

# vmstat 5 5
procs -----------memory----------   ---swap--  -----io---- -system--  ------cpu-----
 r  b   swpd   free   buff  cache     si    so    bi    bo   in    cs us sy id wa st
 3  0 1392140  61200  11632  76432    41    32   117    93    0     1  2  1 97  0  0
 1  1 1467220  63416    324  67284  8786 20499 13178 20567 2539  8924 77 13  0 10  0
 0  2 1593340  57916   1096  53832 24262 46868 29986 46889 5377 18534 44 22  0 34  0
 4  0 1155632 120680   2772  86280 39111 35424 54768 37824 6987 25174 38 27  0 35  0
 3  0 1102988  74096   2852  85276 11261   246 12610   271 1879  6365 86  6  0  8  0

您可以看到 swpd 列的峰值超过了 1.5G,而我配置的是 2.0G。您可以看到交换出(so)活动在同一 5 秒窗口内达到峰值,而交换入(si)活动在下一个窗口达到峰值。

(编辑:我可以看到我配置了 2.0G 交换空间,因为我之前运行了 free

# free
              total        used        free      shared  buff/cache   available
Mem:        1009140      696504       78544       51784      234092      118436
Swap:       2097144      154628     1942516

我们还看到当时我设法只用 1G RAM 运行 discourse。)

有意思。我磁盘空间不足已经有一段时间了。我目前正试图调整到其他方案(我会为此创建一个主题)并比较性能差异,但与这样做相比,你认为——至少在理论上——保持我的当前方案,将备份移动到一个便宜的附加卷[1],然后增加几 GB 的交换空间会是更好的主意吗?


  1. 我需要看看增加一个卷与更改我的方案以获得更多磁盘空间的方案之间到底有多大区别。 ↩︎

那是一个可能的权衡。我做的第一件事之一就是限制备份的数量(并调整频率)。我还没有将备份或上传移动到块存储,但这可能是一个优势,如果它能延长有限内存配置的可用寿命的话。

(在我任职于最小配置机器的最后阶段,我做的另一件事是暂时扩展到具有更大内存的实例,仅在执行升级所需的时间内,然后再缩减回原来的配置。即使较大的实例每月费用可能是两倍,但这只花费了分文。但这确实需要一些停机时间,以及一些时间和精力。它还要求您的提供商有一个具有相同磁盘容量的更大机器——扩展到具有更大磁盘的配置通常是单向的。)

我的提供商允许在保持相同磁盘空间的情况下进行升级,这使得有需要时可以降级 :slight_smile: