Corrupted item pointer during backup process

Installation
1

Discourse version: 2.5.0.beta4 (e81a4048e3)

When trying to backup, I get the error given below.

[2020-05-16 10:09:19] pg_dump: processing data for table "public.reviewable_claimed_topics"
[2020-05-16 10:09:19] pg_dump: dumping contents of table "public.reviewable_claimed_topics"
[2020-05-16 10:09:19] pg_dump: processing data for table "public.reviewable_histories"
[2020-05-16 10:09:19] pg_dump: dumping contents of table "public.reviewable_histories"
[2020-05-16 10:09:19] pg_dump: processing data for table "public.reviewable_scores"
[2020-05-16 10:09:19] pg_dump: dumping contents of table "public.reviewable_scores"
[2020-05-16 10:09:19] pg_dump: processing data for table "public.reviewables"
[2020-05-16 10:09:19] pg_dump: dumping contents of table "public.reviewables"
[2020-05-16 10:09:19] pg_dump: processing data for table "public.scheduler_stats"
[2020-05-16 10:09:19] pg_dump: dumping contents of table "public.scheduler_stats"
[2020-05-16 10:09:19] pg_dump: Dumping the contents of table "scheduler_stats" failed: PQgetResult() failed.
[2020-05-16 10:09:19] pg_dump: Error message from server: PANIC:  corrupted item pointer: 10992
[2020-05-16 10:09:19] pg_dump: The command was: COPY public.scheduler_stats (id, name, hostname, pid, duration_ms, live_slots_start, live_slots_finish, started_at, success, error) TO stdout;
[2020-05-16 10:09:19] Unpausing sidekiq...
[2020-05-16 10:09:19] EXCEPTION: pg_dump failed
[2020-05-16 10:09:19] /var/www/discourse/lib/backup_restore/backuper.rb:183:in `dump_public_schema'
/var/www/discourse/lib/backup_restore/backuper.rb:40:in `run'
/var/www/discourse/lib/backup_restore.rb:184:in `block in start!'
/var/www/discourse/lib/backup_restore.rb:181:in `fork'
/var/www/discourse/lib/backup_restore.rb:181:in `start!'
/var/www/discourse/lib/backup_restore.rb:15:in `backup!'
/var/www/discourse/app/controllers/admin/backups_controller.rb:40:in `create'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/abstract_controller/base.rb:195:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_controller/metal/rendering.rb:30:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3/lib/active_support/callbacks.rb:135:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/abstract_controller/callbacks.rb:41:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_controller/metal/rescue.rb:22:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3/lib/active_support/notifications.rb:180:in `block in instrument'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3/lib/active_support/notifications.rb:180:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/abstract_controller/base.rb:136:in `process'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionview-6.0.3/lib/action_view/rendering.rb:39:in `process'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-2.0.1/lib/mini_profiler/profiling_methods.rb:104:in `block in profile_method'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_controller/metal.rb:190:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_controller/metal.rb:254:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/routing/route_set.rb:33:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/journey/router.rb:49:in `block in serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/journey/router.rb:32:in `each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/journey/router.rb:32:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/routing/route_set.rb:834:in `call'
/var/www/discourse/lib/middleware/omniauth_bypass_middleware.rb:68:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/tempfile_reaper.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/conditional_get.rb:38:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/head.rb:12:in `call'
/var/www/discourse/lib/content_security_policy/middleware.rb:12:in `call'
/var/www/discourse/lib/middleware/anonymous_cache.rb:318:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/session/abstract/id.rb:259:in `context'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/session/abstract/id.rb:253:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/middleware/cookies.rb:648:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3/lib/active_support/callbacks.rb:101:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/middleware/actionable_exceptions.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/logster-2.8.0/lib/logster/middleware/reporter.rb:43:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3/lib/rails/rack/logger.rb:37:in `call_app'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3/lib/rails/rack/logger.rb:28:in `call'
/var/www/discourse/config/initializers/100-quiet_logger.rb:18:in `call'
/var/www/discourse/config/initializers/100-silence_logger.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/var/www/discourse/lib/middleware/enforce_hostname.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/method_override.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/sendfile.rb:111:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.3/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-2.0.1/lib/mini_profiler/profiler.rb:312:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus/rack/middleware.rb:57:in `call'
/var/www/discourse/lib/middleware/request_tracker.rb:172:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3/lib/rails/engine.rb:527:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3/lib/rails/railtie.rb:190:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3/lib/rails/railtie.rb:190:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/urlmap.rb:68:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/urlmap.rb:53:in `each'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/urlmap.rb:53:in `call'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.5/lib/unicorn/http_server.rb:605:in `process_client'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.5/lib/unicorn/http_server.rb:701:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.5/lib/unicorn/http_server.rb:548:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.5/lib/unicorn/http_server.rb:144:in `start'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.5/bin/unicorn:128:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `<main>'
[2020-05-16 10:09:19] Deleting old backups...
[2020-05-16 10:09:19] Cleaning stuff up...
[2020-05-16 10:09:19] Removing '.tar' leftovers...
[2020-05-16 10:09:19] Marking backup as finished...
[2020-05-16 10:09:19] Refreshing disk stats...
[2020-05-16 10:09:19] Notifying 'RikoDEV' of the end of the backup...
[2020-05-16 10:09:29] Something went wrong while notifying user.
[2020-05-16 10:09:29] Finished!
1 Like
2

Looks like a corrupt database to me based on the error message?

3

Is there any way to repair this database without losing data? I have the last backup 2 months ago and I don’t want to lose the created topics.

4

You got lucky that this table contains just sidekiq stats and nothing really important. You can try to remove this table rows and then try to trigger another backup. Hope only this table was affected :crossed_fingers:

5

The problem is that postgresql crashes when I try to clean this table.

discourse=# DELETE FROM scheduler_stats;
PANIC:  corrupted item pointer: 10992
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
The connection to the server was lost. Attempting reset: Succeeded.
6

Oh noes. Does a TRUNCATE on this table also fails?

2 Likes
7

TRUNCATE works, thanks. :smile:. I will now check if the backup is successful.

EDIT:

Oh no :frowning: !

[2020-05-17 16:56:39] pg_dump: Dumping the contents of table "post_search_data" failed: PQgetResult() failed.
[2020-05-17 16:56:39] pg_dump: Error message from server: ERROR:  invalid page in block 89 of relation base/16384/22213
[2020-05-17 16:56:39] pg_dump: The command was: COPY public.post_search_data (post_id, search_data, raw_data, locale, version) TO stdout;
[2020-05-17 16:56:39] Unpausing sidekiq...
[2020-05-17 16:56:39] EXCEPTION: pg_dump failed
[2020-05-17 16:56:39] /var/www/discourse/lib/backup_restore/backuper.rb:183:in `dump_public_schema'

EDIT 2 :tada:

Fixed using PostgreSQL: invalid page header in block - Stack Overflow.

5 Likes
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.