Rebake schlägt fehl: wie diagnostizieren und beheben?

[Canapin (Coin-coin le Canapin)]

Hi,

After disabling the CDN on my forum, I ran rake posts:rebake.
It failed after a few dozen thousand rebake.

Searching for on of the error messages (“PQsocket() can’t get socket descriptor”) led me to Rake:rebake crashes with errors: PG::ConnectionBad: PQsocket - #8 by j127 where adding swap solved the issue.
I’m not sure the issue is the same tho, but I tried it, and rebaking still fail. It’s possible that it fails after more rebaking than before after adding swap but I didn’t write down how many rebakes it failed the first times.

What happens:

1. It starts rebaking, and crashes after about 200000 posts out of 1.6 million.

   root@Unicyclist-app:/var/www/discourse# rake posts:rebake
   Rebaking post markdown for 'default'
      202930 / 1634878 ( 12.4%)
   Failed to rebake (topic_id: 196639, post_id: 1470519)
   PQconsumeInput() server closed the connection unexpectedly
           This probably means the server terminated abnormally
           before or while processing the request.
   

2. After this message, this error is instantly shown for hundreds of subsequent posts :

      202931 / 1634878 ( 12.4%)
   Failed to rebake (topic_id: 196638, post_id: 1470518)
   PQsocket() can't get socket descriptor
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `exec'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `async_exec'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:167:in `perform_query'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:556:in `block (2 levels) in raw_execute'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1017:in `block in with_raw_connection'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:986:in `with_raw_connection'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:555:in `block in raw_execute'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1137:in `log'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:554:in `raw_execute'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:591:in `internal_execute'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:547:in `internal_exec_query'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:693:in `select'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:73:in `select_all'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/query_cache.rb:262:in `select_all'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:68:in `_query_by_sql'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:53:in `block in find_by_sql'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:52:in `find_by_sql'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/statement_cache.rb:152:in `execute'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/core.rb:454:in `block in cached_find_by'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/core.rb:442:in `cached_find_by'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/core.rb:327:in `find_by'
   /var/www/discourse/lib/pretty_text/helpers.rb:20:in `avatar_template'
   /var/www/discourse/lib/pretty_text.rb:255:in `call'
   /var/www/discourse/lib/pretty_text.rb:255:in `eval'
   /var/www/discourse/lib/pretty_text.rb:255:in `block in markdown'
   /var/www/discourse/lib/pretty_text.rb:695:in `block in protect'
   /var/www/discourse/lib/pretty_text.rb:695:in `synchronize'
   /var/www/discourse/lib/pretty_text.rb:695:in `protect'
   /var/www/discourse/lib/pretty_text.rb:192:in `markdown'
   /var/www/discourse/lib/pretty_text.rb:318:in `cook'
   /var/www/discourse/app/models/post_analyzer.rb:30:in `cook'
   /var/www/discourse/app/models/post.rb:359:in `cook'
   /var/www/discourse/app/models/post.rb:836:in `rebake!'
   /var/www/discourse/lib/tasks/posts.rake:145:in `rebake_post'
   /var/www/discourse/lib/tasks/posts.rake:132:in `block (2 levels) in rebake_posts'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
   /var/www/discourse/lib/tasks/posts.rake:131:in `block in rebake_posts'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
   /var/www/discourse/lib/tasks/posts.rake:126:in `rebake_posts'
   /var/www/discourse/lib/tasks/posts.rake:111:in `block in rebake_posts_all_sites'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `each_connection'
   /var/www/discourse/lib/tasks/posts.rake:111:in `rebake_posts_all_sites'
   /var/www/discourse/lib/tasks/posts.rake:7:in `block in <main>'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `block in execute'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `each'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:281:in `execute'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:199:in `synchronize'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/task.rb:188:in `invoke'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:188:in `invoke_task'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:138:in `block (2 levels) in top_level'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:138:in `each'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:138:in `block in top_level'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:147:in `run_with_threads'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:132:in `top_level'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:83:in `block in run'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:214:in `standard_exception_handling'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.3.1/lib/rake/application.rb:80:in `run'
   bin/rake:13:in `<top (required)>'
   /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:59:in `load'
   /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:59:in `kernel_load'
   /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli/exec.rb:23:in `run'
   /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:452:in `exec'
   /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
   /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
   /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor.rb:538:in `dispatch'
   /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:35:in `dispatch'
   /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
   /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/cli.rb:29:in `start'
   /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/exe/bundle:28:in `block in <top (required)>'
   /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
   /usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.4/exe/bundle:20:in `<top (required)>'
   /usr/local/bin/bundle:25:in `load'
   /usr/local/bin/bundle:25:in `<main>'
   

3. Until it ends with this multiple times:

   Caused by:
   ActiveRecord::ConnectionNotEstablished: PQsocket() can't get socket descriptor (ActiveRecord::ConnectionNotEstablished)
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `exec'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `async_exec'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:167:in `perform_query'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:556:in `block (2 levels) in raw_execute'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1017:in `block in with_raw_connection'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:986:in `with_raw_connection'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:555:in `block in raw_execute'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1137:in `log'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:554:in `raw_execute'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:591:in `internal_execute'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:547:in `internal_exec_query'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:693:in `select'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:73:in `select_all'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/query_cache.rb:262:in `select_all'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:68:in `_query_by_sql'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1459:in `block (2 levels) in exec_main_query'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1458:in `block in exec_main_query'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1442:in `exec_main_query'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1420:in `block in exec_queries'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1414:in `exec_queries'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/goldiloader-6.0.0/lib/goldiloader/active_record_patches.rb:40:in `exec_queries'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1191:in `load'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:353:in `records'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
   /var/www/discourse/lib/tasks/posts.rake:131:in `block in rebake_posts'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
   /var/www/discourse/lib/tasks/posts.rake:126:in `rebake_posts'
   /var/www/discourse/lib/tasks/posts.rake:111:in `block in rebake_posts_all_sites'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `each_connection'
   /var/www/discourse/lib/tasks/posts.rake:111:in `rebake_posts_all_sites'
   /var/www/discourse/lib/tasks/posts.rake:7:in `block in <main>'
   /usr/local/bin/bundle:25:in `load'
   /usr/local/bin/bundle:25:in `<main>'
   
   Caused by:
   PG::ConnectionBad: PQsocket() can't get socket descriptor (PG::ConnectionBad)
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `exec'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.1/lib/patches/db/pg/alias_method.rb:109:in `async_exec'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:167:in `perform_query'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:556:in `block (2 levels) in raw_execute'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1017:in `block in with_raw_connection'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:986:in `with_raw_connection'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:555:in `block in raw_execute'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:1137:in `log'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:554:in `raw_execute'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:591:in `internal_execute'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:547:in `internal_exec_query'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:693:in `select'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:73:in `select_all'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/query_cache.rb:262:in `select_all'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/querying.rb:68:in `_query_by_sql'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1459:in `block (2 levels) in exec_main_query'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/connection_handling.rb:312:in `with_connection'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1458:in `block in exec_main_query'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1442:in `exec_main_query'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1420:in `block in exec_queries'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1480:in `skip_query_cache_if_necessary'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1414:in `exec_queries'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/goldiloader-6.0.0/lib/goldiloader/active_record_patches.rb:40:in `exec_queries'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:1191:in `load'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation.rb:353:in `records'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-8.0.4/lib/active_record/relation/delegation.rb:101:in `each'
   /var/www/discourse/lib/tasks/posts.rake:131:in `block in rebake_posts'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-8.0.4/lib/active_support/core_ext/range/each.rb:14:in `step'
   /var/www/discourse/lib/tasks/posts.rake:126:in `rebake_posts'
   /var/www/discourse/lib/tasks/posts.rake:111:in `block in rebake_posts_all_sites'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'
   /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-7.0.0/lib/rails_multisite/connection_management.rb:17:in `each_connection'
   /var/www/discourse/lib/tasks/posts.rake:111:in `rebake_posts_all_sites'
   /var/www/discourse/lib/tasks/posts.rake:7:in `block in <main>'
   /usr/local/bin/bundle:25:in `load'
   /usr/local/bin/bundle:25:in `<main>'
   Tasks: TOP => posts:rebake
   (See full trace by running task with --trace)
   root@Unicyclist-app:/var/www/discourse#
   

Here’s the output of free -h during the rebake (the values are roughly the same at any time during the rebake):

               total        used        free      shared  buff/cache   available
Mem:           7.6Gi       4.6Gi       231Mi       2.1Gi       2.8Gi       709Mi
Swap:          2.0Gi       1.8Gi       194Mi

I also have a lot of free space (I didn’t the first times, but freed space just in case, and it didn’t change anything).

I have rebaked all posts on this forum a few times over the last years and I never faced any issues. The forum hasn’t grown much, the server’s specs are the same as before.

[Canapin (Coin-coin le Canapin)]

Ich suche immer noch nach Hilfe dabei

[Canapin (Coin-coin le Canapin)]

Ich habe die Swap-Größe von 2 auf 4 GB erhöht, und das erneute Backen funktionierte.

Ich vermute also, es ging um den Speicher, obwohl ich etwas überrascht bin, wie viel er verlangt (8 GB RAM + 4 GB Swap), besonders da ältere erneute Backvorgänge ohne Swap und mit der gleichen Menge an RAM keine Probleme hatten.

Was soll ich tun, wenn ich die RAM-Auslastung überwachen und auf Probleme prüfen möchte? Welche Metrik sollte für die Schätzung des Speicherbedarfs wichtig sein? Seitenaufrufe pro Tag (5000–8000 + Bot-Spitzen ein- oder zweimal im Monat)? Datenbankgröße (23 GB)?

[Ed_S]

Der Speicherverbrauch ist während eines Rebuilds viel höher als im normalen Betrieb. Es scheint, dass auch das erneute Backen (rebaking) eine große Belastung darstellt. Wenn ja, würde eine regelmäßige Überwachung nicht viel Wert bringen: Die Überwachung wäre während dieser Spitzen erforderlich, die glücklicherweise dann auftreten, wenn der Administrator eine bestimmte Aktion ausführt.

Als ich auf einer kleineren, marginaleren Maschinenkonfiguration lief, hatte ich ein zweites Terminalfenster geöffnet, über SSH mit meinem Server verbunden und führte aus:
vmstat 5
was eine Aufzeichnung der Speichernutzung liefert, wie sie sich hebt und senkt. Beobachten Sie die Spalte swpd und vergleichen Sie sie mit Ihrem konfigurierten Swap-Speicher. Normalerweise tritt der Fehler plötzlich auf, nicht allmählich, sodass selbst die Betrachtung kurzfristiger Trends nicht viel hilft.

Wenn Sie über genügend Festplattenspeicher verfügen, schadet es überhaupt nicht, viel Swap zu haben – halb so viel wie RAM oder sogar so viel wie RAM. Er ist in diesem Fall da, um Spitzen abzufangen. Sie möchten während des normalen Gebrauchs keine Swap-/Paging-Aktivität sehen. Auch hier kann man vmstat 5 5 verwenden, um ein kurzfristiges Bild der Paging-Aktivität (in den Spalten si und so) zu erhalten.

Hier ist ein Beispiel:

# vmstat 5 5
procs -----------memory----------   ---swap--  -----io---- -system--  ------cpu-----
 r  b   swpd   free   buff  cache     si    so    bi    bo   in    cs us sy id wa st
 3  0 1392140  61200  11632  76432    41    32   117    93    0     1  2  1 97  0  0
 1  1 1467220  63416    324  67284  8786 20499 13178 20567 2539  8924 77 13  0 10  0
 0  2 1593340  57916   1096  53832 24262 46868 29986 46889 5377 18534 44 22  0 34  0
 4  0 1155632 120680   2772  86280 39111 35424 54768 37824 6987 25174 38 27  0 35  0
 3  0 1102988  74096   2852  85276 11261   246 12610   271 1879  6365 86  6  0  8  0

Sie sehen, dass die Spalte swpd auf über 1,5G anstieg, gegenüber meinen konfigurierten 2,0G. Sie sehen, dass die Swap-Out-Aktivität (so) im selben 5-Sekunden-Fenster ihren Höhepunkt erreichte, und die Swap-In-Aktivität (si) im nächsten Fenster.

(Bearbeitung: Ich sehe, dass ich 2,0G Swap konfiguriert hatte, weil ich zuvor free ausgeführt hatte:

# free
              total        used        free      shared  buff/cache   available
Mem:        1009140      696504       78544       51784      234092      118436
Swap:       2097144      154628     1942516

Wir sehen auch, dass ich zu dieser Zeit Discourse mit nur 1G RAM betreiben konnte.)

[Canapin (Coin-coin le Canapin)]

@Ed_S

Wenn Sie den Festplattenspeicher haben, schadet es überhaupt nicht, viel Swap zu haben – halb so viel wie RAM oder sogar so viel wie RAM

Interessant. Ich habe seit einiger Zeit wenig Festplattenspeicher. Ich versuche gerade, auf andere Pläne umzusteigen (ich werde dazu ein Thema erstellen) und zu sehen, wie sich die Leistung vergleicht. Aber anstatt das zu tun, denken Sie, es wäre – zumindest auf dem Papier – eine bessere Idee, meinen Plan beizubehalten, meine Backups auf ein günstiges zusätzliches Volume^[1] zu verschieben und mehrere GB Swap hinzuzufügen?

---

1. Nun, ich muss sehen, ob es einen großen Unterschied macht, ein Volume hinzuzufügen oder meinen Plan in einen anderen zu ändern, der auch mehr Festplattenspeicher bietet) ↩︎

[Ed_S]

Das ist ein möglicher Kompromiss. Eines der ersten Dinge, die ich getan habe, war, die Anzahl der Backups zu begrenzen (und die Häufigkeit anzupassen). Ich bin noch nicht auf Block-Speicher für Backups oder Uploads umgestiegen, aber das könnte ein Gewinn sein, wenn es die Nutzungsdauer einer Konfiguration mit begrenztem RAM verlängert.

(Eine andere Sache, die ich gegen Ende meiner Zeit auf den Maschinen mit der kleinsten Konfiguration getan habe, war, vorübergehend auf eine Instanz mit mehr RAM zu skalieren, nur für die Zeit, die für ein Upgrade benötigt wird, und dann wieder zurückzuskalieren. Es kostete nur Pfennige, obwohl die größere Instanz doppelt so hohe monatliche Kosten verursachen kann. Es erfordert jedoch etwas Ausfallzeit und etwas Zeit und Sorgfalt. Und es setzt voraus, dass Ihr Anbieter eine größere Maschine mit der gleichen Festplattenkapazität hat – das Hochskalieren auf eine Konfiguration mit größerer Festplatte ist normalerweise eine Einbahnstraße.)

[Canapin (Coin-coin le Canapin)]

@Ed_S

Und es erfordert, dass Ihr Anbieter eine größere Maschine mit der gleichen Festplattenkapazität hat

Mein Anbieter erlaubt das Hochskalieren bei Beibehaltung des gleichen Speicherplatzes, was bei Bedarf auch ein Herunterskalieren ermöglicht