Corrupted item pointer during backup process

Support Self-hosting

AI-generated summary

Corrupted item pointer during backup process

RikoDEV encountered an error while backing up their Discourse database (version 2.5.0.beta4) due to a corrupted item pointer in the public.scheduler_stats table. codinghorror suspected a corrupt database, prompting RikoDEV to ask for a way to repair the database without losing data.

Falco suggested removing the rows in the affected table, but this failed due to the server crashing. However, Falco later suggested using TRUNCATE on the table, which worked.

After resolving the initial issue, another error occurred while backing up the post_search_data table. RikoDEV eventually fixed the problem by following a solution from a Stack Overflow post that helped resolve the invalid page in block error.

1

Discourse version: 2.5.0.beta4 (e81a4048e3)

When trying to backup, I get the error given below.

[2020-05-16 10:09:19] pg_dump: processing data for table "public.reviewable_claimed_topics"
[2020-05-16 10:09:19] pg_dump: dumping contents of table "public.reviewable_claimed_topics"
[2020-05-16 10:09:19] pg_dump: processing data for table "public.reviewable_histories"
[2020-05-16 10:09:19] pg_dump: dumping contents of table "public.reviewable_histories"
[2020-05-16 10:09:19] pg_dump: processing data for table "public.reviewable_scores"
[2020-05-16 10:09:19] pg_dump: dumping contents of table "public.reviewable_scores"
[2020-05-16 10:09:19] pg_dump: processing data for table "public.reviewables"
[2020-05-16 10:09:19] pg_dump: dumping contents of table "public.reviewables"
[2020-05-16 10:09:19] pg_dump: processing data for table "public.scheduler_stats"
[2020-05-16 10:09:19] pg_dump: dumping contents of table "public.scheduler_stats"
[2020-05-16 10:09:19] pg_dump: Dumping the contents of table "scheduler_stats" failed: PQgetResult() failed.
[2020-05-16 10:09:19] pg_dump: Error message from server: PANIC:  corrupted item pointer: 10992
[2020-05-16 10:09:19] pg_dump: The command was: COPY public.scheduler_stats (id, name, hostname, pid, duration_ms, live_slots_start, live_slots_finish, started_at, success, error) TO stdout;
[2020-05-16 10:09:19] Unpausing sidekiq...
[2020-05-16 10:09:19] EXCEPTION: pg_dump failed
[2020-05-16 10:09:19] /var/www/discourse/lib/backup_restore/backuper.rb:183:in `dump_public_schema'
/var/www/discourse/lib/backup_restore/backuper.rb:40:in `run'
/var/www/discourse/lib/backup_restore.rb:184:in `block in start!'
/var/www/discourse/lib/backup_restore.rb:181:in `fork'
/var/www/discourse/lib/backup_restore.rb:181:in `start!'
/var/www/discourse/lib/backup_restore.rb:15:in `backup!'
/var/www/discourse/app/controllers/admin/backups_controller.rb:40:in `create'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/abstract_controller/base.rb:195:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_controller/metal/rendering.rb:30:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3/lib/active_support/callbacks.rb:135:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/abstract_controller/callbacks.rb:41:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_controller/metal/rescue.rb:22:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3/lib/active_support/notifications.rb:180:in `block in instrument'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3/lib/active_support/notifications.rb:180:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/abstract_controller/base.rb:136:in `process'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionview-6.0.3/lib/action_view/rendering.rb:39:in `process'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-2.0.1/lib/mini_profiler/profiling_methods.rb:104:in `block in profile_method'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_controller/metal.rb:190:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_controller/metal.rb:254:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/routing/route_set.rb:33:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/journey/router.rb:49:in `block in serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/journey/router.rb:32:in `each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/journey/router.rb:32:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/routing/route_set.rb:834:in `call'
/var/www/discourse/lib/middleware/omniauth_bypass_middleware.rb:68:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/tempfile_reaper.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/conditional_get.rb:38:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/head.rb:12:in `call'
/var/www/discourse/lib/content_security_policy/middleware.rb:12:in `call'
/var/www/discourse/lib/middleware/anonymous_cache.rb:318:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/session/abstract/id.rb:259:in `context'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/session/abstract/id.rb:253:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/middleware/cookies.rb:648:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3/lib/active_support/callbacks.rb:101:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/middleware/actionable_exceptions.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/logster-2.8.0/lib/logster/middleware/reporter.rb:43:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3/lib/rails/rack/logger.rb:37:in `call_app'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3/lib/rails/rack/logger.rb:28:in `call'
/var/www/discourse/config/initializers/100-quiet_logger.rb:18:in `call'
/var/www/discourse/config/initializers/100-silence_logger.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/var/www/discourse/lib/middleware/enforce_hostname.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/method_override.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/sendfile.rb:111:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-2.0.1/lib/mini_profiler/profiler.rb:312:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus/rack/middleware.rb:57:in `call'
/var/www/discourse/lib/middleware/request_tracker.rb:172:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3/lib/rails/engine.rb:527:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3/lib/rails/railtie.rb:190:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3/lib/rails/railtie.rb:190:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/urlmap.rb:68:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/urlmap.rb:53:in `each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/urlmap.rb:53:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.5/lib/unicorn/http_server.rb:605:in `process_client'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.5/lib/unicorn/http_server.rb:701:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.5/lib/unicorn/http_server.rb:548:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.5/lib/unicorn/http_server.rb:144:in `start'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.5/bin/unicorn:128:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `<main>'
[2020-05-16 10:09:19] Deleting old backups...
[2020-05-16 10:09:19] Cleaning stuff up...
[2020-05-16 10:09:19] Removing '.tar' leftovers...
[2020-05-16 10:09:19] Marking backup as finished...
[2020-05-16 10:09:19] Refreshing disk stats...
[2020-05-16 10:09:19] Notifying 'RikoDEV' of the end of the backup...
[2020-05-16 10:09:29] Something went wrong while notifying user.
[2020-05-16 10:09:29] Finished!
2

Looks like a corrupt database to me based on the error message?

3

Is there any way to repair this database without losing data? I have the last backup 2 months ago and I don’t want to lose the created topics.

4

You got lucky that this table contains just sidekiq stats and nothing really important. You can try to remove this table rows and then try to trigger another backup. Hope only this table was affected :crossed_fingers:

5

The problem is that postgresql crashes when I try to clean this table.

discourse=# DELETE FROM scheduler_stats;
PANIC:  corrupted item pointer: 10992
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
The connection to the server was lost. Attempting reset: Succeeded.
6

Oh noes. Does a TRUNCATE on this table also fails?

7

TRUNCATE works, thanks. :smile:. I will now check if the backup is successful.

EDIT:

Oh no :frowning: !

[2020-05-17 16:56:39] pg_dump: Dumping the contents of table "post_search_data" failed: PQgetResult() failed.
[2020-05-17 16:56:39] pg_dump: Error message from server: ERROR:  invalid page in block 89 of relation base/16384/22213
[2020-05-17 16:56:39] pg_dump: The command was: COPY public.post_search_data (post_id, search_data, raw_data, locale, version) TO stdout;
[2020-05-17 16:56:39] Unpausing sidekiq...
[2020-05-17 16:56:39] EXCEPTION: pg_dump failed
[2020-05-17 16:56:39] /var/www/discourse/lib/backup_restore/backuper.rb:183:in `dump_public_schema'

EDIT 2 :tada:

Fixed using https://stackoverflow.com/questions/5220344/postgresql-invalid-page-header-in-block.