Rebakeが失敗する:診断と修正の方法は?

Hi,

After disabling the CDN on my forum, I ran rake posts:rebake.
It failed after a few dozen thousand rebake.

Searching for on of the error messages (“PQsocket() can’t get socket descriptor”) led me to Rake:rebake crashes with errors: PG::ConnectionBad: PQsocket - #8 by j127 where adding swap solved the issue.
I’m not sure the issue is the same tho, but I tried it, and rebaking still fail. It’s possible that it fails after more rebaking than before after adding swap but I didn’t write down how many rebakes it failed the first times.

What happens:

  1. It starts rebaking, and crashes after about 200000 posts out of 1.6 million.
    root@Unicyclist-app:/var/www/discourse# rake posts:rebake
    Rebaking post markdown for 'default'
       202930 / 1634878 ( 12.4%)
    Failed to rebake (topic_id: 196639, post_id: 1470519)
    PQconsumeInput() server closed the connection unexpectedly
            This probably means the server terminated abnormally
            before or while processing the request.
    
  2. After this message, this error is instantly shown for hundreds of subsequent posts :
       202931 / 1634878 ( 12.4%)
    Failed to rebake (topic_id: 196638, post_id: 1470518)
    PQsocket() can't get socket descriptor
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `async_exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:167:in `perform_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:556:in `block (2 levels) in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1017:in `block in with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:986:in `with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:555:in `block in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1137:in `log'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:554:in `raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:591:in `internal_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:547:in `internal_exec_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:693:in `select'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:73:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/query_cache.rb:262:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:68:in `_query_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:53:in `block in find_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:52:in `find_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/statement_cache.rb:152:in `execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/core.rb:454:in `block in cached_find_by'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/core.rb:442:in `cached_find_by'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/core.rb:327:in `find_by'
    /var/www/discourse/lib/pretty_text/helpers.rb:20:in `avatar_template'
    /var/www/discourse/lib/pretty_text.rb:255:in `call'
    /var/www/discourse/lib/pretty_text.rb:255:in `eval'
    /var/www/discourse/lib/pretty_text.rb:255:in `block in markdown'
    /var/www/discourse/lib/pretty_text.rb:695:in `block in protect'
    /var/www/discourse/lib/pretty_text.rb:695:in `synchronize'
    /var/www/discourse/lib/pretty_text.rb:695:in `protect'
    /var/www/discourse/lib/pretty_text.rb:192:in `markdown'
    /var/www/discourse/lib/pretty_text.rb:318:in `cook'
    /var/www/discourse/app/models/post_analyzer.rb:30:in `cook'
    /var/www/discourse/app/models/post.rb:359:in `cook'
    /var/www/discourse/app/models/post.rb:836:in `rebake!'
    /var/www/discourse/lib/tasks/posts.rake:145:in `rebake_post'
    /var/www/discourse/lib/tasks/posts.rake:132:in `block (2 levels) in rebake_posts'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
    /var/www/discourse/lib/tasks/posts.rake:131:in `block in rebake_posts'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/lib/tasks/posts.rake:126:in `rebake_posts'
    /var/www/discourse/lib/tasks/posts.rake:111:in `block in rebake_posts_all_sites'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `each_connection'
    /var/www/discourse/lib/tasks/posts.rake:111:in `rebake_posts_all_sites'
    /var/www/discourse/lib/tasks/posts.rake:7:in `block in <main>'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `block in execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `each'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:199:in `synchronize'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:188:in `invoke'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:188:in `invoke_task'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:138:in `block (2 levels) in top_level'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:138:in `each'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:138:in `block in top_level'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:147:in `run_with_threads'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:132:in `top_level'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:83:in `block in run'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:214:in `standard_exception_handling'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:80:in `run'
    bin/rake:13:in `<top (required)>'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:59:in `load'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:59:in `kernel_load'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:23:in `run'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:452:in `exec'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor.rb:538:in `dispatch'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:35:in `dispatch'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:29:in `start'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/exe/bundle:28:in `block in <top (required)>'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/exe/bundle:20:in `<top (required)>'
    /usr/local/bin/bundle:25:in `load'
    /usr/local/bin/bundle:25:in `<main>'
    
  3. Until it ends with this multiple times:
    Caused by:
    ActiveRecord::ConnectionNotEstablished: PQsocket() can't get socket descriptor (ActiveRecord::ConnectionNotEstablished)
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `async_exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:167:in `perform_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:556:in `block (2 levels) in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1017:in `block in with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:986:in `with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:555:in `block in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1137:in `log'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:554:in `raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:591:in `internal_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:547:in `internal_exec_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:693:in `select'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:73:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/query_cache.rb:262:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:68:in `_query_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1459:in `block (2 levels) in exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1458:in `block in exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1442:in `exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1420:in `block in exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1414:in `exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/goldiloader-6.0.0/lib/goldiloader/active_record_patches.rb:40:in `exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1191:in `load'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:353:in `records'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
    /var/www/discourse/lib/tasks/posts.rake:131:in `block in rebake_posts'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/lib/tasks/posts.rake:126:in `rebake_posts'
    /var/www/discourse/lib/tasks/posts.rake:111:in `block in rebake_posts_all_sites'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `each_connection'
    /var/www/discourse/lib/tasks/posts.rake:111:in `rebake_posts_all_sites'
    /var/www/discourse/lib/tasks/posts.rake:7:in `block in <main>'
    /usr/local/bin/bundle:25:in `load'
    /usr/local/bin/bundle:25:in `<main>'
    
    Caused by:
    PG::ConnectionBad: PQsocket() can't get socket descriptor (PG::ConnectionBad)
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `async_exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:167:in `perform_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:556:in `block (2 levels) in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1017:in `block in with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:986:in `with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:555:in `block in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1137:in `log'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:554:in `raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:591:in `internal_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:547:in `internal_exec_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:693:in `select'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:73:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/query_cache.rb:262:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:68:in `_query_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1459:in `block (2 levels) in exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1458:in `block in exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1442:in `exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1420:in `block in exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1414:in `exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/goldiloader-6.0.0/lib/goldiloader/active_record_patches.rb:40:in `exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1191:in `load'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:353:in `records'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
    /var/www/discourse/lib/tasks/posts.rake:131:in `block in rebake_posts'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/lib/tasks/posts.rake:126:in `rebake_posts'
    /var/www/discourse/lib/tasks/posts.rake:111:in `block in rebake_posts_all_sites'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `each_connection'
    /var/www/discourse/lib/tasks/posts.rake:111:in `rebake_posts_all_sites'
    /var/www/discourse/lib/tasks/posts.rake:7:in `block in <main>'
    /usr/local/bin/bundle:25:in `load'
    /usr/local/bin/bundle:25:in `<main>'
    Tasks: TOP => posts:rebake
    (See full trace by running task with --trace)
    root@Unicyclist-app:/var/www/discourse#
    

Here’s the output of free -h during the rebake (the values are roughly the same at any time during the rebake):

               total        used        free      shared  buff/cache   available
Mem:           7.6Gi       4.6Gi       231Mi       2.1Gi       2.8Gi       709Mi
Swap:          2.0Gi       1.8Gi       194Mi

I also have a lot of free space (I didn’t the first times, but freed space just in case, and it didn’t change anything).

I have rebaked all posts on this forum a few times over the last years and I never faced any issues. The forum hasn’t grown much, the server’s specs are the same as before.

まだこれについての助けを探しています :face_with_tongue:

スワップサイズを2GBから4GBに増やしたところ、リベイクが成功しました。

メモリが原因だったようですが、要求される量(RAM 8GB + スワップ4GB)には少し驚いています。特に、以前のリベイクではスワップなし、同じRAM量でも問題がなかったからです。

メモリ使用量を監視し、問題がないか確認するにはどうすればよいですか?メモリ要件を見積もる上で重要となる指標は何でしょうか?1日のページビュー数(5000〜8000 + 月に1〜2回のボットスパイク)?データベースサイズ(23GB)? :person_shrugging: :face_with_raised_eyebrow:

「いいね!」 1

リビルド中は、通常の動作時よりもメモリ使用量がはるかに大きくなります。同様に、リベイク(rebaking)も大きな負荷をかけるようです。もしそうであれば、定期的な監視は何の価値もありません。監視が必要なのは、幸いにも管理者が特定のアクションを実行したときに発生するピーク時だからです。

より小さく、より限界的なマシンの構成で実行していた頃は、2つ目のターミナルウィンドウを開き、サーバーにssh接続して以下を実行していました。
vmstat 5
これはメモリ使用量がどのように増減するかを記録します。swpd列を見て、設定したスワップ領域と比較してください。通常、障害は徐々にではなく突然発生するため、短期的な傾向を見てもあまり役に立ちません。

ディスク容量があれば、多くのスワップ(RAMの半分、あるいはRAMと同量)があっても全く害はありません。これはピークに対処するためにそこにあります。通常の動作中にスワッピング/ページングアクティビティを見るべきではありません。ここでも、vmstat 5 5を使用して、ページングアクティビティの短期的な状況(si列とso列)を確認できます。

例を以下に示します。

# vmstat 5 5
procs -----------memory----------   ---swap--  -----io---- -system--  ------cpu-----
 r  b   swpd   free   buff  cache     si    so    bi    bo   in    cs us sy id wa st
 3  0 1392140  61200  11632  76432    41    32   117    93    0     1  2  1 97  0  0
 1  1 1467220  63416    324  67284  8786 20499 13178 20567 2539  8924 77 13  0 10  0
 0  2 1593340  57916   1096  53832 24262 46868 29986 46889 5377 18534 44 22  0 34  0
 4  0 1155632 120680   2772  86280 39111 35424 54768 37824 6987 25174 38 27  0 35  0
 3  0 1102988  74096   2852  85276 11261   246 12610   271 1879  6365 86  6  0  8  0

swpd列が設定した2.0Gに対して1.5Gを超えてピークに達しているのがわかります。同じ5秒間のウィンドウでスワップアウト(so)アクティビティがピークに達し、次のウィンドウでスワップイン(si)がピークに達しているのがわかります。

(追記:以前にfreeを実行したため、2.0Gのスワップを設定していたことがわかります。

# free
              total        used        free      shared  buff/cache   available
Mem:        1009140      696504       78544       51784      234092      118436
Swap:       2097144      154628     1942516

この時点で、1GのRAMしかなくてもDiscourseを実行できていたこともわかります。)

「いいね!」 1

興味深いですね。しばらくディスク容量が不足しています。現在、他のプランへのスケール変更を試みており(それについてはトピックを作成します)、パフォーマンスの比較を見ていますが、そうする代わりに、プランを維持し、バックアップを安価な追加ボリューム[1]に移動し、数GBのスワップを追加する方が良いとお考えでしょうか?


  1. 追加のボリュームを追加することと、より多くのディスク容量を提供する別のプランに変更することの間に、それほど大きな違いがあるかどうかを確認する必要があります) ↩︎

「いいね!」 1

それはトレードオフの可能性があります。私が最初に行ったことの一つは、バックアップの数(と頻度)を制限することでした。まだバックアップやアップロードにブロックストレージを移行していませんが、RAMが限られた構成の有用な寿命を延ばすことができれば、それは有益かもしれません。

(私が、最小構成のマシンで在籍していた頃の終盤に行ったもう一つのことは、アップグレードにかかる時間だけ一時的にRAMの大きいインスタンスにスケールアップし、その後スケールダウンするというものでした。より大きなインスタンスは月額コストが2倍になるかもしれませんが、それでも数セントしかかかりませんでした。しかし、少しのダウンタイムと、時間と注意が必要です。また、プロバイダーが同じディスク容量を持つより大きなマシンを持っている必要があります。ディスク容量の大きい構成へのスケールアップは、通常一方通行です。)

「いいね!」 1

私のプロバイダーは、必要に応じてスケールダウンできるように、同じディスク容量を維持しながらスケールアップすることを許可しています :slight_smile:

「いいね!」 1