Sidekiq job (Calculate Scores) are constantly failing after PG15 migration

Product Support

AI-generated summary

After migrating to PostgreSQL 15, a Sidekiq job (Calculate Scores) is failing with errors indicating missing or corrupted database files, such as "could not open file “base/16384/544114.1"”. The issue is linked to file segments being shorter than expected, and no such files are present in the data folder. Additionally, some users are encountering 500 errors when accessing their public profiles via /u/username/summary.json. Dmitry_Krasnoperov reports the error and backtrace, while zogstrip suggests investigating the database files as the root cause.

1

Greetings. All of a sudden after migration to PSQL15 one of my jobs (Calculate Scores) are constantly failing:

Job exception: ERROR: could not open file "base/16384/544114.1" (target block 655360): previous segment is only 20325 blocks
Job exception: ERROR: could not open file "base/16384/544113.1" (target block 134235414): previous segment is only 41880 blocks
(no such files present in psql data folder)

Backtrace:
rack-mini-profiler-3.3.1/lib/patches/db/pg.rb:110:in `exec' 
rack-mini-profiler-3.3.1/lib/patches/db/pg.rb:110:in `async_exec' 
mini_sql-1.6.0/lib/mini_sql/postgres/connection.rb:217:in `run' 
mini_sql-1.6.0/lib/mini_sql/active_record_postgres/connection.rb:38:in `block in run' 
mini_sql-1.6.0/lib/mini_sql/active_record_postgres/connection.rb:34:in `block in with_lock' 
activesupport-7.2.2.1/lib/active_support/concurrency/null_lock.rb:9:in `synchronize' 
mini_sql-1.6.0/lib/mini_sql/active_record_postgres/connection.rb:34:in `with_lock' 
mini_sql-1.6.0/lib/mini_sql/active_record_postgres/connection.rb:38:in `run' 
/var/www/discourse/lib/mini_sql_multisite_connection.rb:109:in `run' 
mini_sql-1.6.0/lib/mini_sql/postgres/connection.rb:196:in `exec' 
(eval at /var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mini_sql-1.6.0/lib/mini_sql/builder.rb:62):2:in `exec'
/var/www/discourse/lib/score_calculator.rb:73:in `update_posts_rank' 
/var/www/discourse/lib/score_calculator.rb:15:in `calculate' 
/var/www/discourse/app/jobs/scheduled/calculate_scores.rb:8:in `execute' 
/var/www/discourse/app/jobs/base.rb:316:in `block (2 levels) in perform' 
rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
rails_multisite-6.1.0/lib/rails_multisite/connection_management.rb:21:in `with_connection'
/var/www/discourse/app/jobs/base.rb:303:in `block in perform' 
/var/www/discourse/app/jobs/base.rb:299:in `each' 
/var/www/discourse/app/jobs/base.rb:299:in `perform' 
/var/www/discourse/app/jobs/base.rb:379:in `perform' 
mini_scheduler-0.18.0/lib/mini_scheduler/manager.rb:137:in `process_queue' 
mini_scheduler-0.18.0/lib/mini_scheduler/manager.rb:77:in `worker_loop' 
mini_scheduler-0.18.0/lib/mini_scheduler/manager.rb:63:in `block (2 levels) in ensure_worker_threads'

Also, two of my users, considering no issues with their profiles, reporting 500 error while accessing their public profiles:

/u/username/summary.json

Any advise on fixing the situation?

2

Something’s wrong with your database file(s). You might want to investigate those first.