Sadly, the backup doesn’t successfully load when attempted – the databases seem to have diverged in some way. I’ll try to dig the /etc/passwd out of the container, though it’s a lot of hassle when the container’s not up.
Will it not restore or can you not get it to the server to restore it?
Won’t restore. I could try it again on a fresh install but the logs I saw before the importer stopped running talked about field mismatches. It’s an old… 2.9.x instance, I think? Don’t have my notes handy.
For the record, one of our smart guys figured out how to rsync the proper permissions the container needs in order for PostgreSQL to run:
rsync -rog --delete --perms --numeric-ids root@x.x.x.x:/var/discourse/ /var/discourse
4 « J'aime »
running into a problem with rsync all seemed to go well. nit will not rebuild? in part looks like a problem with ports maybe?
I, [2025-01-05T06:32:16.218961 #1] INFO -- : > exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf
I, [2025-01-05T06:32:16.224597 #1] INFO -- : > sleep 10
2175:C 05 Jan 2025 06:32:16.236 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2175:C 05 Jan 2025 06:32:16.236 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=2175, just started
2175:C 05 Jan 2025 06:32:16.236 # Configuration loaded
2175:M 05 Jan 2025 06:32:16.238 * monotonic clock: POSIX clock_gettime
2175:M 05 Jan 2025 06:32:16.238 # Warning: Could not create server TCP listening socket *:6379: bind: Address already in use
2175:M 05 Jan 2025 06:32:16.238 # Failed listening on port 6379 (TCP), aborting.
I, [2025-01-05T06:32:26.228550 #1] INFO -- :
I, [2025-01-05T06:32:26.228998 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
URGENT: Failed to initialize site default: ActiveRecord::ConnectionNotEstablished connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:69:in `rescue in new_client'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:57:in `new_client'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:982:in `connect'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:994:in `reconnect'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:662:in `block in reconnect!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:661:in `reconnect!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:763:in `block in verify!'/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:754:in `verify!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:771:in `connect!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:977:in `block in with_raw_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:976:in `with_raw_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql/quoting.rb:128:in `quote_string'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract/quoting.rb:76:in `quote'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql/quoting.rb:122:in `quote'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql/schema_statements.rb:1099:in `quoted_scope'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql/schema_statements.rb:1076:in `data_source_sql'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:60:in `table_exists?'
/var/www/discourse/lib/site_settings/db_provider.rb:58:in `table_exists?'
/var/www/discourse/lib/site_settings/db_provider.rb:14:in `all'
/var/www/discourse/lib/site_settings/defaults_provider.rb:30:in `db_all'
/var/www/discourse/lib/site_setting_extension.rb:360:in `block in refresh!'
/var/www/discourse/lib/site_setting_extension.rb:353:in `synchronize'
/var/www/discourse/lib/site_setting_extension.rb:353:in `refresh!'
/var/www/discourse/config/initializers/005-site_settings.rb:20:in `block (2 levels) in <main>'
/var/www/discourse/lib/freedom_patches/rails_multisite.rb:16:in `block in safe_each_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:36:in `each_connection'/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management.rb:21:in `each_connection'
/var/www/discourse/lib/freedom_patches/rails_multisite.rb:9:in `each_active_connection'
/var/www/discourse/lib/freedom_patches/rails_multisite.rb:14:in `safe_each_connection'
/var/www/discourse/config/initializers/005-site_settings.rb:18:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:407:in `instance_exec'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:407:in `block in make_lambda'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:179:in `block in call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:668:in `block (2 levels) in default_terminator'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:667:in `catch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:667:in `block in default_terminator'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:180:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:559:in `block in invoke_before'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:559:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:559:in `invoke_before'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:109:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/reloader.rb:96:in `prepare!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/application/finisher.rb:74:in `block in <module:Finisher>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/initializable.rb:32:in `instance_exec'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/initializable.rb:32:in `run'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/initializable.rb:61:in `block in run_initializers'
/usr/local/lib/ruby/3.3.0/tsort.rb:231:in `block in tsort_each'
/usr/local/lib/ruby/3.3.0/tsort.rb:353:in `block (2 levels) in each_strongly_connected_component'
/usr/local/lib/ruby/3.3.0/tsort.rb:434:in `each_strongly_connected_component_from'
/usr/local/lib/ruby/3.3.0/tsort.rb:352:in `block in each_strongly_connected_component'
/usr/local/lib/ruby/3.3.0/tsort.rb:350:in `each'
/usr/local/lib/ruby/3.3.0/tsort.rb:350:in `call'
/usr/local/lib/ruby/3.3.0/tsort.rb:350:in `each_strongly_connected_component'
/usr/local/lib/ruby/3.3.0/tsort.rb:229:in `tsort_each'
/usr/local/lib/ruby/3.3.0/tsort.rb:208:in `tsort_each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/initializable.rb:60:in `run_initializers'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/application.rb:435:in `initialize!'
/var/www/discourse/config/environment.rb:7:in `<main>'
/usr/local/lib/ruby/3.3.0/bundled_gems.rb:69:in `require'
/usr/local/lib/ruby/3.3.0/bundled_gems.rb:69:in `block (2 levels) in replace_require'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/bootsnap-1.18.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/zeitwerk-2.7.1/lib/zeitwerk/core_ext/kernel.rb:34:in `require'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/application.rb:411:in `require_environment!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/application.rb:559:in `block in run_tasks_blocks'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `block in execute'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:281:in `execute'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:199:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:243:in `block in invoke_prerequisites'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:241:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:241:in `invoke_prerequisites'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:199:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:243:in `block in invoke_prerequisites'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:241:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:241:in `invoke_prerequisites'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:199:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/task.rb:188:in `invoke'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:188:in `invoke_task'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `block (2 levels) in top_level'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:138:in `block in top_level'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:147:in `run_with_threads'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:132:in `top_level'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:83:in `block in run'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:214:in `standard_exception_handling'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/lib/rake/application.rb:80:in `run'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/bin/rake:25:in `load'
/var/www/discourse/vendor/bundle/ruby/3.3.0/bin/rake:25:in `<top (required)>'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/cli/exec.rb:58:in `load'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/cli/exec.rb:58:in `kernel_load'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/cli/exec.rb:23:in `run'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/cli.rb:455:in `exec'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/cli.rb:35:in `dispatch'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/cli.rb:29:in `start'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/exe/bundle:28:in `block in <top (required)>'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.5.18/exe/bundle:20:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
rake aborted!
ActiveRecord::ConnectionNotEstablished: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory (ActiveRecord::ConnectionNotEstablished)
Is the server running locally and accepting connections on that socket?
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:69:in `rescue in new_client'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:57:in `new_client'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:982:in `connect'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:994:in `reconnect'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:662:in `block in reconnect!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:661:in `reconnect!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:763:in `block in verify!'/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:754:in `verify!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:771:in `connect!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:977:in `block in with_raw_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:976:in `with_raw_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql/quoting.rb:128:in `quote_string'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract/quoting.rb:76:in `quote'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql/quoting.rb:122:in `quote'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql/schema_statements.rb:1099:in `quoted_scope'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql/schema_statements.rb:1076:in `data_source_sql'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:60:in `table_exists?'
/var/www/discourse/config/initializers/006-ensure_login_hint.rb:8:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:407:in `instance_exec'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:407:in `block in make_lambda'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:179:in `block in call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:668:in `block (2 levels) in default_terminator'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:667:in `catch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:667:in `block in default_terminator'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:180:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:559:in `block in invoke_before'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:559:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:559:in `invoke_before'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:109:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/reloader.rb:96:in `prepare!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/application/finisher.rb:74:in `block in <module:Finisher>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/initializable.rb:32:in `instance_exec'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/initializable.rb:32:in `run'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/initializable.rb:61:in `block in run_initializers'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/initializable.rb:60:in `run_initializers'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/application.rb:435:in `initialize!'
/var/www/discourse/config/environment.rb:7:in `<main>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/bootsnap-1.18.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/zeitwerk-2.7.1/lib/zeitwerk/core_ext/kernel.rb:34:in `require'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/application.rb:411:in `require_environment!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/application.rb:559:in `block in run_tasks_blocks'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Caused by:
PG::ConnectionBad: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory (PG::ConnectionBad)
Is the server running locally and accepting connections on that socket?
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/pg-1.5.4/lib/pg/connection.rb:819:in `connect_start'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/pg-1.5.4/lib/pg/connection.rb:819:in `connect_to_hosts'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/pg-1.5.4/lib/pg/connection.rb:759:in `new'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/pg-1.5.4/lib/pg.rb:63:in `connect'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:58:in `new_client'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:982:in `connect'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:994:in `reconnect'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:662:in `block in reconnect!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:661:in `reconnect!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:763:in `block in verify!'/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:754:in `verify!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:771:in `connect!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:977:in `block in with_raw_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:976:in `with_raw_connection'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql/quoting.rb:128:in `quote_string'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract/quoting.rb:76:in `quote'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql/quoting.rb:122:in `quote'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql/schema_statements.rb:1099:in `quoted_scope'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql/schema_statements.rb:1076:in `data_source_sql'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:60:in `table_exists?'
/var/www/discourse/config/initializers/006-ensure_login_hint.rb:8:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:407:in `instance_exec'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:407:in `block in make_lambda'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:179:in `block in call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:668:in `block (2 levels) in default_terminator'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:667:in `catch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:667:in `block in default_terminator'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:180:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:559:in `block in invoke_before'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:559:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:559:in `invoke_before'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/callbacks.rb:109:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/reloader.rb:96:in `prepare!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/application/finisher.rb:74:in `block in <module:Finisher>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/initializable.rb:32:in `instance_exec'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/initializable.rb:32:in `run'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/initializable.rb:61:in `block in run_initializers'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/initializable.rb:60:in `run_initializers'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/application.rb:435:in `initialize!'
/var/www/discourse/config/environment.rb:7:in `<main>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/bootsnap-1.18.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/zeitwerk-2.7.1/lib/zeitwerk/core_ext/kernel.rb:34:in `require'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/application.rb:411:in `require_environment!'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/railties-7.2.2.1/lib/rails/application.rb:559:in `block in run_tasks_blocks'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate => db:load_config => environment
(See full trace by running task with --trace)
I, [2025-01-05T06:32:34.909125 #1] INFO -- : gem install ruby-openid -v 2.9.2 -i /var/www/discourse/plugins/discourse-steam-login/gems/3.3.6 --no-document --ignore-dependencies --no-user-install
Successfully installed ruby-openid-2.9.2
1 gem installed
gem install rack-openid -v 1.4.2 -i /var/www/discourse/plugins/discourse-steam-login/gems/3.3.6 --no-document --ignore-dependencies --no-user-install
Successfully installed rack-openid-1.4.2
1 gem installed
gem install omniauth-openid -v 2.0.1 -i /var/www/discourse/plugins/discourse-steam-login/gems/3.3.6 --no-document --ignore-dependencies --no-user-installSuccessfully installed omniauth-openid-2.0.1
1 gem installed
gem install omniauth-steam -v 1.0.6 -i /var/www/discourse/plugins/discourse-steam-login/gems/3.3.6 --no-document --ignore-dependencies --no-user-install
Successfully installed omniauth-steam-1.0.6
1 gem installed
I, [2025-01-05T06:32:34.910102 #1] INFO -- : Terminating async processes
I, [2025-01-05T06:32:34.910159 #1] INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 87
87:signal-handler (1736058754) Received SIGTERM scheduling shutdown...
87:M 05 Jan 2025 06:32:34.971 # User requested shutdown...
87:M 05 Jan 2025 06:32:34.971 * Saving the final RDB snapshot before exiting.
87:M 05 Jan 2025 06:32:35.116 * DB saved on disk
87:M 05 Jan 2025 06:32:35.116 # Redis is now ready to exit, bye bye...
FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' failed with return #<Process::Status: pid 2178 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params {"cd"=>"$home", "tag"=>"migrate", "hook"=>"db_migrate", "cmd"=>["su discourse -c 'bundle exec rake db:migrate'"]}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
9e8de06a277d5ac57d1a7ef16b9cde5fc6dd11766c456bccc7ea22b90bf80bb9
This seems to have worked. And says app is running but not accessible from web.
Could it be not connected yet from the domain server yet?
Note:. Think I found it. May have to wait 24 to 72 hours for the IP address change to be fully in effect. 
This is the connection msg I get. Presume have to wait for nameservers to be updated.
1 « J'aime »
I just want to reiterate that --numeric-ids is 100% mandatory when rsync’ing to a new server. If you don’t, rsync will try to make usernames match up between the hosts, which will change file ownership inside of Docker. This is going to have you rolling through a million error messages to find out the first failure was Postgres refusing to start because it no longer owns a file it thinks it should, but then tons of other havoc will follow (Redis will try to bind a TCP port that it’s already listening on, dogs and cats living together, etc), making it hard to diagnose the issue.
Also, it’s absolutely terrifying that actual backups made by Discourse, to run inside of a Docker container completely controlled by Discourse, MAY NOT WORK, and apparently have failures going back for years, apparently unaddressed, which is why I came to rsync to migrate to new hardware. Rsync got it done, and now it’s sitting on a mirrored ZFS filesystem with hourly snapshots and a remote backup made daily, which is great…but I do worry for all the admins out there that think the automatic backup from Discourse will save them in a disaster and are going to be scratching their heads about corrupt SQL indexes, and what plugins they were running, when they try to use it.
I’m curious about this corrupted index question. If you did have a corrupted index, wouldn’t rsync just propagate that bad information to your new instance? Wouldn’t a backup/restore at least detect that something is wrong?
It’s not clear to me how a corrupt index is best fixed - I see PostgreSQL docs say:
If you suspect corruption of an index on a user table, you can simply rebuild that index, or all indexes on the table, using
REINDEX INDEXorREINDEX TABLE.
I didn’t dig in deeply, so I can’t say, but it’s either the official backup method generates an SQL dump that doesn’t work, or importing it causes a showstopper failure that didn’t have to be a showstopper, or the database has been corrupt for years but Postgres just keeps running anyhow if you rsync it over.
In theory, SQL indexes are just for speeding up queries, right? So it’s possible that the rsync’d system is just disregarding it and something is suboptimal in some small way that we haven’t noticed. I really don’t know.
Chatter on other threads suggests that some version of Postgres broke things, and a later version fixed it, but I don’t know the specific technical details of what went wrong.
I’m not sure that’s true or sure that it always works. At some point, I think, the user and or group IDs inside the docker container changed. I’m willing to give –numberic-ids a try, though.
For the new server migrations I do, I’ve been skipping postgres files like this:
cd /var/discourse
rsync -rav --numeric-ids =old_ip=:/var/discourse/containers/ /var/discourse/containers/
rsync -rav --numeric-ids =old_ip=:/var/discourse/shared/ /var/discourse/shared/ --exclude log --exclude postgres_* --exclude redis_data --exclude log --exclude tmp --exclude state
and restoring a backup to the new machine.
Yeah. The corrupt index issue can be a real bear. I don’t think I’ve seen one in a good while, though.
Yes, but (I think) they also see that things are unique and the corrupt index will let you have multiple records with what is supposed to be a unique value. It’s for stuff like tracking link clicks, so it’s not especially critical if it’s wrong.
I think I’ve paid closer attention to this than you have and the above pretty much sums up my understanding as well. 
Maybe the problem was with PG12? That could explain why I haven’t seen the problem in a long while, but I don’t pretend to remember, if I ever knew for sure.
1 « J'aime »
So is there a known magic incantation for fixing this, even in a blanket way? Like ./launcher enter app and then run this specific command and it’ll rebuild all the database indexes whether it was needed or not?
The amazing and terrible thing about Discourse is there is almost always some esoteric magic that will fix darn near anything, but stumbling into that knowledge can be tricky sometimes. 
(Also, yes, I suppose this might not always work, so I should speak with less confidence than I did…but mapping file ownership to whatever the new host server has will definitely only work by luck, so you’re better off gambling that the Discourse image still has the same user ID numbers when rsync’ing.)
You can
./launcher enter app
sudo su - postgres
psql
And then, (EDIT: I looked it up and this appears to be the actual command)
REINDEX CONCURRENTLY DATABASE discourse;
And that will try to rebuild them and if it succeeds, you’re in good shape. If it fails, you need to go and delete the things that have duplicate ids. That’s more complicated.
3 « J'aime »
That’s more complicated.
Lord, did this turn out to be true!
I took a run at this. Now, understand our Discourse instance has like 15-20 years worth of mailing list archives imported into it. So there are a lot of old posts that look like this…
…where you get a ton of links, and email replies that quote the entire previous email repeating those links, and email signatures with links to unrelated things, etc.
So at some point, the topic_links table screwed up, and started making new rows for links it already had:
So there might be a row like:
69534 | 18675 | 86333 | 6631 | http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org | lists.libsdl.org | f | | 2017-03-26 12:13:06.365742 | 2017-03-26 12:13:06.365742 | f | 0 | | SDL Info Page | 2017-03-26 13:18:08.491592 | f |
With that row created in 2017. But then I’ve got hundreds of dupes across all these posts, where the system came along in 2024 and added duplicate rows with id values like 200000 higher.
So it’s a lot to clean out.
Basically I’d wait for the reindexing to fail, see what it spit out, find the ID for the duplicate row, and adjust the URL string to no longer duplicate…it looks like the system already deals with this by inserting ** into the string semi-randomly. I did the same. I was afraid to delete the rows outright, because I have no idea what references these rows elsewhere.
It did seem to be a handful of topics, presumably different posts on the same topics, because I was fixing the same handful of URLs over and over. It was tedious, I don’t recommend it. I probably should have found a way to automate this.
It’s also worth noting that the “CONCURRENTLY” option on REINDEX will build a temporary index (with a _ccnew* string appended to the name), and if the process fails, it’ll leave it there. If you’re going to use CONCURRENTLY and have to clean up lots of things, expect to drop a LOT of temporary indices by the time you clean everything out. (Also, if like me, you have never used Postgres directly, also know I had to issue a \connect discourse; command at the start before the REINDEX command could be used.)
I’m still in the process of doing this; it doesn’t seem to be making things worse, so I’m continuing to plug along on it.
This is more hand-waiving than actual help, but here are a couple of ideas.
First, now that you know what index is broken, you can reindex just that one and not every index in the database. You might ask https://ask.discourse.com/ for a hint for how to do that.
Corrupt indexes in PG12, how do I fix? - #11 by sam also has some hints.
Since you don’t care that much about this link tracking (and I guess a new one has been created anyway), just deleting the duplicates is fine.
There should be a way to do a query that will find the duplicates so you could delete them all at once.
I got this done, and Postgres (and Discourse!) seem to be happy.
I cleaned them out by hand, making the URLs unique with ** patterns as appropriate. It might just be a harmless cache where I could have deleted the duplicates but I didn’t want to risk it.
In my case, it was just the one index, so rebuilding all the indices was probably overkill, but honestly I felt better knowing I caught everything.
After a few failed runs of rebuilding, which takes 30 seconds or so each time and reports a single problem, this was my SQL magic to get a complete list of problem items instantly:
discourse=# select topic_id, post_id, url, COUNT(*) from topic_links GROUP BY topic_id, post_id, url HAVING COUNT(*) > 1 order by topic_id, post_id;
topic_id | post_id | url | count
----------+---------+-------------------------------------------------------+-------
19200 | 88461 | http://hg.libsdl.org/SDL/rev/**533131e24aeb | 2
19207 | 88521 | http://hg.libsdl.org/SDL/rev/44a2e00e7c66 | 2
19255 | 88683 | http://lists.libsdl.org/__listinfo.cgi/sdl-libsdl.org | 2
19255 | 88683 | http://lists.libsdl.org/**listinfo.cgi/sdl-libsdl.org | 2
19523 | 90003 | http://twitter.com/Ironcode_Gaming | 2
(5 rows)
(5 remaining problem items in this query, for example purposes.)
Then I’d look at each post to see what was there and what to fix up:
select * from topic_links where topic_id=19255 and post_id=88683
and then fix one of them up:
update public.topic_links set url='http://lists.libsdl.org/__listinfo.cgi/**sdl-libsdl.org' where id=275100;
Until I ran out of things to fix up.
I probably could have done some inner-join magic (or maybe a little Ruby) to get this all in one query, but I’m not an expert and it turned out to not be hours of work to do it manually. But it was tedious, to be clear.
Then I did REINDEX DATABASE discourse; without the CONCURRENTLY just to keep it simple, nuked a few ccnew* indices I had missed earlier, and I was good to go.
Site was live the whole time, no downtime.
Whether this was necessary or not, I definitely feel like my data is a little safer now, and I’m not careening towards some unannounced future disaster.
Thanks for nudging me in the right direction to figure this out, @pfaffman!
2 « J'aime »