Rebake 실패: 진단 및 해결 방법?

CDN을 비활성화한 후 포럼에서 rake posts:rebake를 실행했습니다.
수만 건의 rebake를 진행하던 중 실패했습니다.

에러 메시지 중 하나인 (“PQsocket() can’t get socket descriptor”)를 검색하여 Rake:rebake crashes with errors: PG::ConnectionBad: PQsocket - #8 by j127 링크를 찾았고, 여기서 swap을 추가하면 문제가 해결된다고 하더군요.
정확히 같은 문제인지 확신은 없었지만, 시도해 보았으나 rebake는 여전히 실패했습니다. swap을 추가한 후 이전보다 더 많은 rebake가 진행된 후에 실패하는 것일 수도 있지만, 처음 몇 번 실패했을 때 정확히 몇 건까지 진행되었는지 기록해 두지 않았습니다.

발생하는 현상:

  1. rebake가 시작되어 약 160만 건 중 200,000건 정도에서 충돌합니다.
    root@Unicyclist-app:/var/www/discourse# rake posts:rebake
    Rebaking post markdown for 'default'
       202930 / 1634878 ( 12.4%)
    Failed to rebake (topic_id: 196639, post_id: 1470519)
    PQconsumeInput() server closed the connection unexpectedly
            This probably means the server terminated abnormally
            before or while processing the request.
    
  2. 이 메시지 이후, 수백 건의 후속 게시물에 대해 즉시 다음과 같은 에러가 표시됩니다:
       202931 / 1634878 ( 12.4%)
    Failed to rebake (topic_id: 196638, post_id: 1470518)
    PQsocket() can't get socket descriptor
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `async_exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:167:in `perform_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:556:in `block (2 levels) in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1017:in `block in with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:986:in `with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:555:in `block in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1137:in `log'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:554:in `raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:591:in `internal_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:547:in `internal_exec_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:693:in `select'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:73:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/query_cache.rb:262:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:68:in `_query_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:53:in `block in find_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:52:in `find_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/statement_cache.rb:152:in `execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/core.rb:454:in `block in cached_find_by'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/core.rb:442:in `cached_find_by'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/core.rb:327:in `find_by'
    /var/www/discourse/lib/pretty_text/helpers.rb:20:in `avatar_template'
    /var/www/discourse/lib/pretty_text.rb:255:in `call'
    /var/www/discourse/lib/pretty_text.rb:255:in `eval'
    /var/www/discourse/lib/pretty_text.rb:255:in `block in markdown'
    /var/www/discourse/lib/pretty_text.rb:695:in `block in protect'
    /var/www/discourse/lib/pretty_text.rb:695:in `synchronize'
    /var/www/discourse/lib/pretty_text.rb:695:in `protect'
    /var/www/discourse/lib/pretty_text.rb:192:in `markdown'
    /var/www/discourse/lib/pretty_text.rb:318:in `cook'
    /var/www/discourse/app/models/post_analyzer.rb:30:in `cook'
    /var/www/discourse/app/models/post.rb:359:in `cook'
    /var/www/discourse/app/models/post.rb:836:in `rebake!'
    /var/www/discourse/lib/tasks/posts.rake:145:in `rebake_post'
    /var/www/discourse/lib/tasks/posts.rake:132:in `block (2 levels) in rebake_posts'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
    /var/www/discourse/lib/tasks/posts.rake:131:in `block in rebake_posts'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/lib/tasks/posts.rake:126:in `rebake_posts'
    /var/www/discourse/lib/tasks/posts.rake:111:in `block in rebake_posts_all_sites'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `each_connection'
    /var/www/discourse/lib/tasks/posts.rake:111:in `rebake_posts_all_sites'
    /var/www/discourse/lib/tasks/posts.rake:7:in `block in <main>'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `block in execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `each'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:199:in `synchronize'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:188:in `invoke'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:188:in `invoke_task'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:138:in `block (2 levels) in top_level'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:138:in `each'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:138:in `block in top_level'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:147:in `run_with_threads'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:132:in `top_level'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:83:in `block in run'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:214:in `standard_exception_handling'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:80:in `run'
    bin/rake:13:in `<top (required)>'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:59:in `load'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:59:in `kernel_load'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:23:in `run'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:452:in `exec'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor.rb:538:in `dispatch'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:35:in `dispatch'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:29:in `start'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/exe/bundle:28:in `block in <top (required)>'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/exe/bundle:20:in `<top (required)>'
    /usr/local/bin/bundle:25:in `load'
    /usr/local/bin/bundle:25:in `<main>'
    
  3. 여러 번 반복된 후 다음과 같은 내용으로 끝납니다:
    Caused by:
    ActiveRecord::ConnectionNotEstablished: PQsocket() can't get socket descriptor (ActiveRecord::ConnectionNotEstablished)
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `async_exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:167:in `perform_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:556:in `block (2 levels) in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1017:in `block in with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:986:in `with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:555:in `block in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1137:in `log'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:554:in `raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:591:in `internal_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:547:in `internal_exec_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:693:in `select'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:73:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/query_cache.rb:262:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:68:in `_query_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1459:in `block (2 levels) in exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1458:in `block in exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1442:in `exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1420:in `block in exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1414:in `exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/goldiloader-6.0.0/lib/goldiloader/active_record_patches.rb:40:in `exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1191:in `load'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:353:in `records'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
    /var/www/discourse/lib/tasks/posts.rake:131:in `block in rebake_posts'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/lib/tasks/posts.rake:126:in `rebake_posts'
    /var/www/discourse/lib/tasks/posts.rake:111:in `block in rebake_posts_all_sites'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `each_connection'
    /var/www/discourse/lib/tasks/posts.rake:111:in `rebake_posts_all_sites'
    /var/www/discourse/lib/tasks/posts.rake:7:in `block in <main>'
    /usr/local/bin/bundle:25:in `load'
    /usr/local/bin/bundle:25:in `<main>'
    
    Caused by:
    PG::ConnectionBad: PQsocket() can't get socket descriptor (PG::ConnectionBad)
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `async_exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:167:in `perform_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:556:in `block (2 levels) in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1017:in `block in with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:986:in `with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:555:in `block in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1137:in `log'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:554:in `raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:591:in `internal_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:547:in `internal_exec_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:693:in `select'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:73:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/query_cache.rb:262:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:68:in `_query_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1459:in `block (2 levels) in exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1458:in `block in exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1442:in `exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1420:in `block in exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1414:in `exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/goldiloader-6.0.0/lib/goldiloader/active_record_patches.rb:40:in `exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1191:in `load'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:353:in `records'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
    /var/www/discourse/lib/tasks/posts.rake:131:in `block in rebake_posts'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/lib/tasks/posts.rake:126:in `rebake_posts'
    /var/www/discourse/lib/tasks/posts.rake:111:in `block in rebake_posts_all_sites'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `each_connection'
    /var/www/discourse/lib/tasks/posts.rake:111:in `rebake_posts_all_sites'
    /var/www/discourse/lib/tasks/posts.rake:7:in `block in <main>'
    /usr/local/bin/bundle:25:in `load'
    /usr/local/bin/bundle:25:in `<main>'
    Tasks: TOP => posts:rebake
    (See full trace by running task with --trace)
    root@Unicyclist-app:/var/www/discourse#
    

reake 진행 중 free -h 명령의 출력입니다 (reake 중 시점에 관계없이 값은 대략적으로 동일합니다):

               total        used        free      shared  buff/cache   available
Mem:           7.6Gi       4.6Gi       231Mi       2.1Gi       2.8Gi       709Mi
Swap:          2.0Gi       1.8Gi       194Mi

또한 여유 공간이 많이 있습니다 (처음에는 없었지만, 혹시 모를 경우를 대비해 공간을 확보해 두었으나 아무런 변화가 없었습니다).

지난 몇 년간 이 포럼의 모든 게시물을 여러 번 rebake해 보았지만, 이런 문제는 겪어본 적이 없습니다. 포럼 크기가 크게 증가하지 않았으며, 서버 사양도 이전과 동일합니다.

여전히 이 문제에 대해 도움을 찾고 있어요 :face_with_tongue:

스와프(Swap) 크기를 2GB에서 4GB로 늘렸더니 리베이킹(rebake)이 정상적으로 작동했습니다.

따라서 메모리 문제였을 것으로 추측되지만, 이전 리베이킹에서는 스와프 없이 동일한 RAM 용량으로 아무런 문제가 없었던 점을 고려하면, 8GB RAM + 4GB 스와프라는 요구량이 다소 놀랍습니다.

RAM 사용량을 모니터링하고 문제를 확인하고 싶다면 무엇을 해야 하나요? 메모리 요구량을 추정하는 데 중요한 지표는 무엇일까요? 일일 페이지 뷰(5,000~8,000회 + 한 달에 한두 번 발생하는 봇 스파이크)? 데이터베이스 크기(23GB)? :person_shrugging: :face_with_raised_eyebrow:

재빌드(rebuild) 중에는 정상 운영 중보다 메모리 사용량이 훨씬 더 많이 증가합니다. 비슷하게, rebaking 역시 상당한 메모리 요구 사항을 발생시킵니다. 만약 그렇다면, 정기적인 모니터링은 큰 가치를 추가하지 못할 것입니다. 모니터링은 그러한 피크(peak) 시점에 필요하며, 다행히도 이는 관리자가 특정 작업을 수행할 때 발생합니다.

저는 더 작고 사양이 낮은 머신을 사용할 때, 두 번째 터미널 창을 열어 서버에 ssh로 접속하고 vmstat 5를 실행했습니다. 이 명령어는 메모리 사용량이 증감하는 과정을 기록해 줍니다. swpd 컬럼을 관찰하여 구성된 스왑 공간과 비교해 보세요. 일반적으로 장애는 점진적으로 발생하기보다는 갑자기 발생하므로, 단기적인 추이를 살펴보는 것만으로는 큰 도움이 되지 않습니다.

디스크 공간이 있다면, 스왑을 많이 두는 데는 전혀 해가 없습니다. RAM의 절반 정도, 또는 RAM과 같은 크기까지도 가능합니다. 이 경우 스왑은 피크를 감당하기 위해 존재합니다. 정상 사용 중에 스왑/페이지ング 활동이 나타나지 않는 것이 좋습니다. 마찬가지로, vmstat 5 5를 사용하여 페이지ング 활동의 단기적인 상황을 파악할 수 있습니다(si 및 so 컬럼).

다음은 예시입니다:

# vmstat 5 5
procs -----------memory----------   ---swap--  -----io---- -system--  ------cpu-----
 r  b   swpd   free   buff  cache     si    so    bi    bo   in    cs us sy id wa st
 3  0 1392140  61200  11632  76432    41    32   117    93    0     1  2  1 97  0  0
 1  1 1467220  63416    324  67284  8786 20499 13178 20567 2539  8924 77 13  0 10  0
 0  2 1593340  57916   1096  53832 24262 46868 29986 46889 5377 18534 44 22  0 34  0
 4  0 1155632 120680   2772  86280 39111 35424 54768 37824 6987 25174 38 27  0 35  0
 3  0 1102988  74096   2852  85276 11261   246 12610   271 1879  6365 86  6  0  8  0

swpd 컬럼이 구성된 2.0G에 비해 1.5G를 초과하는 피크에 도달했음을 볼 수 있습니다. 스왑아웃(so) 활동이 같은 5초 창에서 피크에 도달했고, 스왑인(si)이 다음 창에서 피크에 도달했음을 볼 수 있습니다.

(편집: 이전에 free를 실행했기 때문에 2.0G의 스왑이 구성된 것을 알 수 있습니다:

# free
              total        used        free      shared  buff/cache   available
Mem:        1009140      696504       78544       51784      234092      118436
Swap:       2097144      154628     1942516

또한 당시 RAM 1G만으로 discourse를 실행하고 있었던 것도 볼 수 있습니다.)

흥미롭네요. 저는 이미有一段 동안 디스크 공간이 부족했습니다. 현재 다른 플랜으로 전환(rescale)을 시도 중이며(이 부분은 별도 토픽으로 만들 예정입니다) 성능 차이가 어떻게 나타나는지 확인하고 있습니다. 하지만 이렇게 하는 대신, 제 생각에는 현재 플랜을 유지하고 백업을 저렴한 추가 볼륨[1]으로 옮긴 뒤, 스왑 공간을 수 GB 추가하는 것이 적어도 이론적으로는 더 나은 아이디어가 아닐까요?


  1. 사실 추가 볼륨을 붙이는 것과 더 많은 디스크 공간을 제공하는 다른 플랜으로 변경하는 것 사이에 차이가 얼마나 나는지 확인해 봐야 합니다.) ↩︎

그것은 가능한 트레이드오프입니다. 제가 처음에 한 일 중 하나는 백업 횟수를 제한하고(주기를 조정하는 것)였습니다. 아직 백업이나 업로드를 위해 블록 스토리지로 전환하지는 않았지만, 제한된 RAM 구성의 수명을 연장할 수 있다면 좋은 선택이 될 수 있습니다.

(제가 가장 작은 구성의 머신을 사용하던 기간 말미에 한 또 다른 일은, 업그레이드를 수행하는 데 걸리는 시간 동안만 RAM이 더 큰 인스턴스로 일시적으로 리스케일링을 한 뒤, 다시 원래대로 되돌리는 것이었습니다. 더 큰 인스턴스의 월 비용이 두 배가 될지라도, 실제로는 몇 센트밖에 들지 않았습니다. 다만 약간의 다운타임과 시간, 그리고 주의가 필요합니다. 또한 제공업체가 동일한 디스크 용량을 가진 더 큰 머신을 갖추고 있어야 합니다. 디스크 용량이 더 큰 구성으로 업스케일링하는 것은 일반적으로 되돌릴 수 없는 일방통행이기 때문입니다.)

제 프로바이더는 동일한 디스크 공간을 유지하면서 업스케일링을 허용하므로, 필요할 경우 다운스케일링도 가능합니다 :slight_smile: