فشل إعادة الخبز: كيف يتم التشخيص والإصلاح؟

Hi,

After disabling the CDN on my forum, I ran rake posts:rebake.
It failed after a few dozen thousand rebake.

Searching for on of the error messages (“PQsocket() can’t get socket descriptor”) led me to Rake:rebake crashes with errors: PG::ConnectionBad: PQsocket - #8 by j127 where adding swap solved the issue.
I’m not sure the issue is the same tho, but I tried it, and rebaking still fail. It’s possible that it fails after more rebaking than before after adding swap but I didn’t write down how many rebakes it failed the first times.

What happens:

  1. It starts rebaking, and crashes after about 200000 posts out of 1.6 million.
    root@Unicyclist-app:/var/www/discourse# rake posts:rebake
    Rebaking post markdown for 'default'
       202930 / 1634878 ( 12.4%)
    Failed to rebake (topic_id: 196639, post_id: 1470519)
    PQconsumeInput() server closed the connection unexpectedly
            This probably means the server terminated abnormally
            before or while processing the request.
    
  2. After this message, this error is instantly shown for hundreds of subsequent posts :
       202931 / 1634878 ( 12.4%)
    Failed to rebake (topic_id: 196638, post_id: 1470518)
    PQsocket() can't get socket descriptor
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `async_exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:167:in `perform_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:556:in `block (2 levels) in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1017:in `block in with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:986:in `with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:555:in `block in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1137:in `log'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:554:in `raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:591:in `internal_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:547:in `internal_exec_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:693:in `select'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:73:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/query_cache.rb:262:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:68:in `_query_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:53:in `block in find_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:52:in `find_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/statement_cache.rb:152:in `execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/core.rb:454:in `block in cached_find_by'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/core.rb:442:in `cached_find_by'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/core.rb:327:in `find_by'
    /var/www/discourse/lib/pretty_text/helpers.rb:20:in `avatar_template'
    /var/www/discourse/lib/pretty_text.rb:255:in `call'
    /var/www/discourse/lib/pretty_text.rb:255:in `eval'
    /var/www/discourse/lib/pretty_text.rb:255:in `block in markdown'
    /var/www/discourse/lib/pretty_text.rb:695:in `block in protect'
    /var/www/discourse/lib/pretty_text.rb:695:in `synchronize'
    /var/www/discourse/lib/pretty_text.rb:695:in `protect'
    /var/www/discourse/lib/pretty_text.rb:192:in `markdown'
    /var/www/discourse/lib/pretty_text.rb:318:in `cook'
    /var/www/discourse/app/models/post_analyzer.rb:30:in `cook'
    /var/www/discourse/app/models/post.rb:359:in `cook'
    /var/www/discourse/app/models/post.rb:836:in `rebake!'
    /var/www/discourse/lib/tasks/posts.rake:145:in `rebake_post'
    /var/www/discourse/lib/tasks/posts.rake:132:in `block (2 levels) in rebake_posts'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
    /var/www/discourse/lib/tasks/posts.rake:131:in `block in rebake_posts'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/lib/tasks/posts.rake:126:in `rebake_posts'
    /var/www/discourse/lib/tasks/posts.rake:111:in `block in rebake_posts_all_sites'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `each_connection'
    /var/www/discourse/lib/tasks/posts.rake:111:in `rebake_posts_all_sites'
    /var/www/discourse/lib/tasks/posts.rake:7:in `block in <main>'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `block in execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `each'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:199:in `synchronize'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:188:in `invoke'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:188:in `invoke_task'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:138:in `block (2 levels) in top_level'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:138:in `each'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:138:in `block in top_level'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:147:in `run_with_threads'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:132:in `top_level'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:83:in `block in run'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:214:in `standard_exception_handling'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:80:in `run'
    bin/rake:13:in `<top (required)>'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:59:in `load'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:59:in `kernel_load'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:23:in `run'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:452:in `exec'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor.rb:538:in `dispatch'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:35:in `dispatch'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:29:in `start'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/exe/bundle:28:in `block in <top (required)>'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
    /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/exe/bundle:20:in `<top (required)>'
    /usr/local/bin/bundle:25:in `load'
    /usr/local/bin/bundle:25:in `<main>'
    
  3. Until it ends with this multiple times:
    Caused by:
    ActiveRecord::ConnectionNotEstablished: PQsocket() can't get socket descriptor (ActiveRecord::ConnectionNotEstablished)
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `async_exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:167:in `perform_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:556:in `block (2 levels) in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1017:in `block in with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:986:in `with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:555:in `block in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1137:in `log'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:554:in `raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:591:in `internal_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:547:in `internal_exec_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:693:in `select'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:73:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/query_cache.rb:262:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:68:in `_query_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1459:in `block (2 levels) in exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1458:in `block in exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1442:in `exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1420:in `block in exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1414:in `exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/goldiloader-6.0.0/lib/goldiloader/active_record_patches.rb:40:in `exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1191:in `load'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:353:in `records'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
    /var/www/discourse/lib/tasks/posts.rake:131:in `block in rebake_posts'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/lib/tasks/posts.rake:126:in `rebake_posts'
    /var/www/discourse/lib/tasks/posts.rake:111:in `block in rebake_posts_all_sites'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `each_connection'
    /var/www/discourse/lib/tasks/posts.rake:111:in `rebake_posts_all_sites'
    /var/www/discourse/lib/tasks/posts.rake:7:in `block in <main>'
    /usr/local/bin/bundle:25:in `load'
    /usr/local/bin/bundle:25:in `<main>'
    
    Caused by:
    PG::ConnectionBad: PQsocket() can't get socket descriptor (PG::ConnectionBad)
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `async_exec'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:167:in `perform_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:556:in `block (2 levels) in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1017:in `block in with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:986:in `with_raw_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:555:in `block in raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1137:in `log'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:554:in `raw_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:591:in `internal_execute'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:547:in `internal_exec_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:693:in `select'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:73:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/query_cache.rb:262:in `select_all'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:68:in `_query_by_sql'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1459:in `block (2 levels) in exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1458:in `block in exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1442:in `exec_main_query'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1420:in `block in exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1414:in `exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/goldiloader-6.0.0/lib/goldiloader/active_record_patches.rb:40:in `exec_queries'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1191:in `load'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:353:in `records'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
    /var/www/discourse/lib/tasks/posts.rake:131:in `block in rebake_posts'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
    /var/www/discourse/lib/tasks/posts.rake:126:in `rebake_posts'
    /var/www/discourse/lib/tasks/posts.rake:111:in `block in rebake_posts_all_sites'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
    /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `each_connection'
    /var/www/discourse/lib/tasks/posts.rake:111:in `rebake_posts_all_sites'
    /var/www/discourse/lib/tasks/posts.rake:7:in `block in <main>'
    /usr/local/bin/bundle:25:in `load'
    /usr/local/bin/bundle:25:in `<main>'
    Tasks: TOP => posts:rebake
    (See full trace by running task with --trace)
    root@Unicyclist-app:/var/www/discourse#
    

Here’s the output of free -h during the rebake (the values are roughly the same at any time during the rebake):

               total        used        free      shared  buff/cache   available
Mem:           7.6Gi       4.6Gi       231Mi       2.1Gi       2.8Gi       709Mi
Swap:          2.0Gi       1.8Gi       194Mi

I also have a lot of free space (I didn’t the first times, but freed space just in case, and it didn’t change anything).

I have rebaked all posts on this forum a few times over the last years and I never faced any issues. The forum hasn’t grown much, the server’s specs are the same as before.

ما زلت أبحث عن مساعدة بخصوص هذا الأمر :face_with_tongue:

لقد قمت بزيادة حجم المبادلة (swap) من 2 إلى 4 غيغابايت، ونجحت عملية إعادة الخبز (rebake).

لذا، أعتقد أن الأمر كان يتعلق بالذاكرة، على الرغم من أنني مندهش بعض الشيء من مقدار ما يطلبه (8 غيغابايت من ذاكرة الوصول العشوائي + 4 غيغابايت من المبادلة)، خاصة وأن عمليات إعادة الخبز السابقة لم تكن تواجه أي مشاكل بدون مبادلة وبنفس مقدار ذاكرة الوصول العشوائي.

ماذا يجب أن أفعل إذا أردت مراقبة استخدام ذاكرة الوصول العشوائي والتحقق من وجود أي مشاكل؟ ما هو المقياس الذي يجب أن يكون مهمًا لتقدير متطلبات الذاكرة؟ عدد مشاهدات الصفحة يوميًا (5000-8000 + زيادات مفاجئة من الروبوتات مرة أو مرتين شهريًا)؟ حجم قاعدة البيانات (23 غيغابايت)؟ :person_shrugging: :face_with_raised_eyebrow:

إعجاب واحد (1)

يكون استخدام الذاكرة أكبر بكثير أثناء إعادة البناء منه أثناء التشغيل العادي. ويبدو أن إعادة الخبز (rebaking)، بالمثل، تتطلب عبئًا كبيرًا. إذا كان الأمر كذلك، فإن أي نوع من المراقبة المنتظمة لن يضيف قيمة كبيرة: ستكون المراقبة مطلوبة خلال تلك الفترات القصوى، والتي تحدث لحسن الحظ عندما يقوم المسؤول باتخاذ إجراء معين.

عندما كنت أعمل على تكوين جهاز أصغر وأكثر هامشية، كنت أفتح نافذة طرفية ثانية، متصلاً عبر ssh بخادمي وأقوم بتشغيل
vmstat 5
مما يوفر سجلاً لاستخدام الذاكرة وهي تتذبذب. راقب عمود swpd وقارنه بمساحة المبادلة (swap space) المكونة لديك. غالبًا ما يحدث الفشل فجأة، وليس تدريجيًا، لذا فإن النظر إلى الاتجاهات قصيرة المدى ليس ذا فائدة كبيرة.

إذا كان لديك مساحة قرص كافية، فلا ضرر على الإطلاق من وجود الكثير من المبادلة - نصف مساحة ذاكرة الوصول العشوائي (RAM)، أو حتى بقدر ذاكرة الوصول العشوائي. إنها موجودة في هذه الحالة للتعامل مع الفترات القصوى. لا تريد رؤية نشاط المبادلة/الترحيل أثناء الاستخدام العادي. مرة أخرى، يمكن للمرء استخدام vmstat 5 5 للحصول على صورة قصيرة المدى لنشاط الترحيل (في أعمدة si و so)

إليك مثال:

# vmstat 5 5
procs -----------memory----------   ---swap--  -----io---- -system--  ------cpu-----
 r  b   swpd   free   buff  cache     si    so    bi    bo   in    cs us sy id wa st
 3  0 1392140  61200  11632  76432    41    32   117    93    0     1  2  1 97  0  0
 1  1 1467220  63416    324  67284  8786 20499 13178 20567 2539  8924 77 13  0 10  0
 0  2 1593340  57916   1096  53832 24262 46868 29986 46889 5377 18534 44 22  0 34  0
 4  0 1155632 120680   2772  86280 39111 35424 54768 37824 6987 25174 38 27  0 35  0
 3  0 1102988  74096   2852  85276 11261   246 12610   271 1879  6365 86  6  0  8  0

ترى أن عمود swpd بلغ ذروته عند أكثر من 1.5 جيجابايت، مقارنة بـ 2.0 جيجابايت المكونة لدي. ترى أن نشاط الترحيل إلى الخارج (swapout) (so) بلغ ذروته في نفس النافذة الزمنية البالغة 5 ثوانٍ، وبلغ نشاط الترحيل إلى الداخل (swapin) (si) ذروته في النافذة التالية.

(تعديل: يمكنني أن أرى أنني قمت بتكوين 2.0 جيجابايت من المبادلة لأنني كنت قد قمت سابقًا بتشغيل free:

# free
              total        used        free      shared  buff/cache   available
Mem:        1009140      696504       78544       51784      234092      118436
Swap:       2097144      154628     1942516

نرى أيضًا أنني كنت في ذلك الوقت أدير ديسكورس (discourse) باستخدام 1 جيجابايت فقط من ذاكرة الوصول العشوائي.)

إعجاب واحد (1)

مثير للاهتمام. لقد كنت أعاني من نقص في مساحة القرص لبعض الوقت. أحاول حاليًا إعادة القياس إلى خطط أخرى (سأقوم بإنشاء موضوع حول ذلك) وأرى كيف تختلف الأداءات، ولكن بدلاً من القيام بذلك، هل تعتقد أنه سيكون من الأفضل - على الورق، على الأقل - الاحتفاظ بخطتي، ونقل النسخ الاحتياطية الخاصة بي إلى وحدة تخزين إضافية رخيصة [1]، وإضافة عدة غيغابايت من المبادلة (swap)؟


  1. حسنًا، أحتاج إلى معرفة ما إذا كان هناك هذا القدر الكبير من الاختلاف بين إضافة وحدة تخزين أو تغيير خطتي إلى أخرى توفر أيضًا مساحة قرص أكبر) ↩︎

إعجاب واحد (1)

هذه مقايضة محتملة. أحد أول الأشياء التي قمت بها هو تحديد عدد النسخ الاحتياطية (وتعديل تكرارها). لم أنتقل بعد إلى تخزين الكتل (block storage) للنسخ الاحتياطية أو التحميلات، ولكن قد يكون ذلك مكسبًا إذا أدى إلى إطالة العمر الافتراضي لتكوين ذاكرة وصول عشوائي (RAM) محدودة.

(شيء آخر قمت به، في نهاية فترة عملي على أجهزة التكوين الأصغر، هو تغيير حجمها مؤقتًا إلى مثيل ذاكرة وصول عشوائي أكبر فقط للمدة التي يستغرقها إجراء الترقية، ثم تغيير الحجم مرة أخرى. لم يكلف الأمر سوى سنتات، على الرغم من أن المثيل الأكبر قد يكلف ضعف التكلفة الشهرية. ولكنه يتطلب بعض وقت التوقف عن العمل، وبعض الوقت والعناية. ويتطلب أن يكون لدى مزود الخدمة الخاص بك جهاز أكبر بنفس سعة القرص - الترقية إلى تكوين بقرص أكبر هي عادةً طريق باتجاه واحد.)

إعجاب واحد (1)

مزود الخدمة الخاص بي يسمح بالتوسيع مع الحفاظ على نفس مساحة القرص، مما يسمح بالتصغير عند الحاجة :slight_smile:

إعجاب واحد (1)