Rebakeが失敗する:診断と修正の方法は?

こんにちは、

フォーラムの CDN を無効化した後、rake posts:rebake を実行しました。
数万件のリベイク後に失敗しました。

エラーメッセージの 1 つ(“PQsocket() can’t get socket descriptor”)を検索すると、Rake:rebake crashes with errors: PG::ConnectionBad: PQsocket - #8 by j127 というページが見つかり、そこでスワップ領域を追加することで問題が解決したとありました。
しかし、今回の問題が同じものかどうかは確信が持てませんが、試してみたところ、リベイクは依然として失敗しています。スワップを追加した後は、以前よりも多くのリベイク後に失敗する可能性はありますが、最初の失敗時に何件リベイクしたかは記録していませんでした。

発生している現象:

  1. リベイクが開始され、160 万件の投稿のうち約 20 万件でクラッシュします。
    root@Unicyclist-app:/var/www/discourse# rake posts:rebake
    Rebaking post markdown for 'default'
       202930 / 1634878 ( 12.4%)
    Failed to rebake (topic_id: 196639, post_id: 1470519)
    PQconsumeInput() server closed the connection unexpectedly
            This probably means the server terminated abnormally
            before or while processing the request.
    
  2. このメッセージの後、以降の数百件の投稿で即座に以下のエラーが表示されます:
       202931 / 1634878 ( 12.4%)
    Failed to rebake (topic_id: 196638, post_id: 1470518)
    PQsocket() can't get socket descriptor
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `async_exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:167:in `perform_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:556:in `block (2 levels) in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1017:in `block in with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:986:in `with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:555:in `block in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1137:in `log'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:554:in `raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:591:in `internal_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:547:in `internal_exec_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:693:in `select'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:73:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/query_cache.rb:262:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:68:in `_query_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:53:in `block in find_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:52:in `find_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/statement_cache.rb:152:in `execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/core.rb:454:in `block in cached_find_by'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/core.rb:442:in `cached_find_by'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/core.rb:327:in `find_by'
    /var/www/discourse/lib/pretty_text/helpers.rb:20:in `avatar_template'
    /var/www/discourse/lib/pretty_text.rb:255:in `call'
    /var/www/discourse/lib/pretty_text.rb:255:in `eval'
    /var/www/discourse/lib/pretty_text.rb:255:in `block in markdown'
    /var/www/discourse/lib/pretty_text.rb:695:in `block in protect'
    /var/www/discourse/lib/pretty_text.rb:695:in `synchronize'
    /var/www/discourse/lib/pretty_text.rb:695:in `protect'
    /var/www/discourse/lib/pretty_text.rb:192:in `markdown'
    /var/www/discourse/lib/pretty_text.rb:318:in `cook'
    /var/www/discourse/app/models/post_analyzer.rb:30:in `cook'
    /var/www/discourse/app/models/post.rb:359:in `cook'
    /var/www/discourse/app/models/post.rb:836:in `rebake!'
    /var/www/discourse/lib/tasks/posts.rake:145:in `rebake_post'
    /var/www/discourse/lib/tasks/posts.rake:132:in `block (2 levels) in rebake_posts'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
    /var/www/discourse/lib/tasks/posts.rake:131:in `block in rebake_posts'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/lib/tasks/posts.rake:126:in `rebake_posts'
    /var/www/discourse/lib/tasks/posts.rake:111:in `block in rebake_posts_all_sites'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `each_connection'
    /var/www/discourse/lib/tasks/posts.rake:111:in `rebake_posts_all_sites'
    /var/www/discourse/lib/tasks/posts.rake:7:in `block in <main>'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `block in execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `each'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:199:in `synchronize'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:188:in `invoke'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:188:in `invoke_task'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:138:in `block (2 levels) in top_level'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:138:in `each'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:138:in `block in top_level'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:147:in `run_with_threads'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:132:in `top_level'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:83:in `block in run'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:214:in `standard_exception_handling'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:80:in `run'
    bin/rake:13:in `<top (required)>'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:59:in `load'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:59:in `kernel_load'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:23:in `run'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:452:in `exec'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor.rb:538:in `dispatch'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:35:in `dispatch'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:29:in `start'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/exe/bundle:28:in `block in <top (required)>'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/exe/bundle:20:in `<top (required)>'
    /usr/local/bin/bundle:25:in `load'
    /usr/local/bin/bundle:25:in `<main>'
    
  3. その後、以下のようなエラーが複数回表示されて終了します:
    Caused by:
    ActiveRecord::ConnectionNotEstablished: PQsocket() can't get socket descriptor (ActiveRecord::ConnectionNotEstablished)
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `async_exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:167:in `perform_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:556:in `block (2 levels) in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1017:in `block in with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:986:in `with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:555:in `block in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1137:in `log'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:554:in `raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:591:in `internal_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:547:in `internal_exec_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:693:in `select'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:73:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/query_cache.rb:262:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:68:in `_query_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1459:in `block (2 levels) in exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1458:in `block in exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1442:in `exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1420:in `block in exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1414:in `exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/goldiloader-6.0.0/lib/goldiloader/active_record_patches.rb:40:in `exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1191:in `load'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:353:in `records'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
    /var/www/discourse/lib/tasks/posts.rake:131:in `block in rebake_posts'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/lib/tasks/posts.rake:126:in `rebake_posts'
    /var/www/discourse/lib/tasks/posts.rake:111:in `block in rebake_posts_all_sites'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `each_connection'
    /var/www/discourse/lib/tasks/posts.rake:111:in `rebake_posts_all_sites'
    /var/www/discourse/lib/tasks/posts.rake:7:in `block in <main>'
    /usr/local/bin/bundle:25:in `load'
    /usr/local/bin/bundle:25:in `<main>'
    
    Caused by:
    PG::ConnectionBad: PQsocket() can't get socket descriptor (PG::ConnectionBad)
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `async_exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:167:in `perform_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:556:in `block (2 levels) in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1017:in `block in with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:986:in `with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:555:in `block in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1137:in `log'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:554:in `raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:591:in `internal_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:547:in `internal_exec_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:693:in `select'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:73:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/query_cache.rb:262:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:68:in `_query_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1459:in `block (2 levels) in exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1458:in `block in exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1442:in `exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1420:in `block in exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1414:in `exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/goldiloader-6.0.0/lib/goldiloader/active_record_patches.rb:40:in `exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1191:in `load'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:353:in `records'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
    /var/www/discourse/lib/tasks/posts.rake:131:in `block in rebake_posts'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/lib/tasks/posts.rake:126:in `rebake_posts'
    /var/www/discourse/lib/tasks/posts.rake:111:in `block in rebake_posts_all_sites'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `each_connection'
    /var/www/discourse/lib/tasks/posts.rake:111:in `rebake_posts_all_sites'
    /var/www/discourse/lib/tasks/posts.rake:7:in `block in <main>'
    /usr/local/bin/bundle:25:in `load'
    /usr/local/bin/bundle:25:in `<main>'
    Tasks: TOP => posts:rebake
    (See full trace by running task with --trace)
    root@Unicyclist-app:/var/www/discourse#
    

リベイク中の free -h の出力は以下の通りです(リベイク中は値がほぼ一定でした):

               total        used        free      shared  buff/cache   available
Mem:           7.6Gi       4.6Gi       231Mi       2.1Gi       2.8Gi       709Mi
Swap:          2.0Gi       1.8Gi       194Mi

また、ディスク容量も十分に残っています(最初は足りませんでしたが、念のため空き容量を増やしましたが、状況は変わりませんでした)。

このフォーラムでは過去数年間に何度か全投稿のリベイクを行いましたが、これまで問題に直面したことはありませんでした。フォーラムの規模も大きく成長しておらず、サーバーの仕様も以前と同じです。

まだこれについての助けを探しています :face_with_tongue:

スワップサイズを2GBから4GBに増やしたところ、リベイクが成功しました。

メモリが原因だったようですが、要求される量(RAM 8GB + スワップ4GB)には少し驚いています。特に、以前のリベイクではスワップなし、同じRAM量でも問題がなかったからです。

メモリ使用量を監視し、問題がないか確認するにはどうすればよいですか?メモリ要件を見積もる上で重要となる指標は何でしょうか?1日のページビュー数(5000〜8000 + 月に1〜2回のボットスパイク)?データベースサイズ(23GB)? :person_shrugging: :face_with_raised_eyebrow:

リビルド中は、通常の動作時よりもメモリ使用量がはるかに大きくなります。同様に、リベイク(rebaking)も大きな負荷をかけるようです。もしそうであれば、定期的な監視は何の価値もありません。監視が必要なのは、幸いにも管理者が特定のアクションを実行したときに発生するピーク時だからです。

より小さく、より限界的なマシンの構成で実行していた頃は、2つ目のターミナルウィンドウを開き、サーバーにssh接続して以下を実行していました。
vmstat 5
これはメモリ使用量がどのように増減するかを記録します。swpd列を見て、設定したスワップ領域と比較してください。通常、障害は徐々にではなく突然発生するため、短期的な傾向を見てもあまり役に立ちません。

ディスク容量があれば、多くのスワップ(RAMの半分、あるいはRAMと同量)があっても全く害はありません。これはピークに対処するためにそこにあります。通常の動作中にスワッピング/ページングアクティビティを見るべきではありません。ここでも、vmstat 5 5を使用して、ページングアクティビティの短期的な状況(si列とso列)を確認できます。

例を以下に示します。

# vmstat 5 5
procs -----------memory----------   ---swap--  -----io---- -system--  ------cpu-----
 r  b   swpd   free   buff  cache     si    so    bi    bo   in    cs us sy id wa st
 3  0 1392140  61200  11632  76432    41    32   117    93    0     1  2  1 97  0  0
 1  1 1467220  63416    324  67284  8786 20499 13178 20567 2539  8924 77 13  0 10  0
 0  2 1593340  57916   1096  53832 24262 46868 29986 46889 5377 18534 44 22  0 34  0
 4  0 1155632 120680   2772  86280 39111 35424 54768 37824 6987 25174 38 27  0 35  0
 3  0 1102988  74096   2852  85276 11261   246 12610   271 1879  6365 86  6  0  8  0

swpd列が設定した2.0Gに対して1.5Gを超えてピークに達しているのがわかります。同じ5秒間のウィンドウでスワップアウト(so)アクティビティがピークに達し、次のウィンドウでスワップイン(si)がピークに達しているのがわかります。

(追記:以前にfreeを実行したため、2.0Gのスワップを設定していたことがわかります。

# free
              total        used        free      shared  buff/cache   available
Mem:        1009140      696504       78544       51784      234092      118436
Swap:       2097144      154628     1942516

この時点で、1GのRAMしかなくてもDiscourseを実行できていたこともわかります。)

興味深いですね。しばらくディスク容量が不足しています。現在、他のプランへのスケール変更を試みており(それについてはトピックを作成します)、パフォーマンスの比較を見ていますが、そうする代わりに、プランを維持し、バックアップを安価な追加ボリューム[1]に移動し、数GBのスワップを追加する方が良いとお考えでしょうか?


  1. 追加のボリュームを追加することと、より多くのディスク容量を提供する別のプランに変更することの間に、それほど大きな違いがあるかどうかを確認する必要があります) ↩︎

それはトレードオフの可能性があります。私が最初に行ったことの一つは、バックアップの数(と頻度)を制限することでした。まだバックアップやアップロードにブロックストレージを移行していませんが、RAMが限られた構成の有用な寿命を延ばすことができれば、それは有益かもしれません。

(私が、最小構成のマシンで在籍していた頃の終盤に行ったもう一つのことは、アップグレードにかかる時間だけ一時的にRAMの大きいインスタンスにスケールアップし、その後スケールダウンするというものでした。より大きなインスタンスは月額コストが2倍になるかもしれませんが、それでも数セントしかかかりませんでした。しかし、少しのダウンタイムと、時間と注意が必要です。また、プロバイダーが同じディスク容量を持つより大きなマシンを持っている必要があります。ディスク容量の大きい構成へのスケールアップは、通常一方通行です。)

私のプロバイダーは、必要に応じてスケールダウンできるように、同じディスク容量を維持しながらスケールアップすることを許可しています :slight_smile: