Encountered an error while restoring from a backup

[Alienazk]

Hello everyone, I need help with a failure during restoration.
I just installed Discourse and tried to restore from my backup.
Unfortunately, I encountered an error during the restoration process.
The output is as follows:

Starting restore: discourse-2024-05-04-111743-v20240425133407.tar.gz
bundler: failed to load command: script/discourse (script/discourse)
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.3.1/lib/patches/db/pg.rb:110:in `exec': PG::UndefinedTable: ERROR:  relation "users" does not exist (ActiveRecord::StatementInvalid)
LINE 9:  WHERE a.attrelid = '"users"'::regclass
                            ^

	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.3.1/lib/patches/db/pg.rb:110:in `async_exec'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:19:in `block (2 levels) in query'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.1/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:18:in `block in query'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract_adapter.rb:752:in `block in log'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract_adapter.rb:743:in `log'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:17:in `query'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql_adapter.rb:916:in `column_definitions'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:117:in `columns'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/schema_cache.rb:117:in `block in columns'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/schema_cache.rb:116:in `fetch'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/schema_cache.rb:116:in `columns'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/schema_cache.rb:125:in `block in columns_hash'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/schema_cache.rb:124:in `fetch'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/schema_cache.rb:124:in `columns_hash'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:580:in `load_schema!'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/attributes.rb:264:in `load_schema!'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/encryption/encryptable_record.rb:122:in `load_schema!'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:566:in `block in load_schema'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:563:in `synchronize'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:563:in `load_schema'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:419:in `columns_hash'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/core.rb:309:in `block in find_by'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/core.rb:293:in `each'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/core.rb:293:in `each_with_object'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/core.rb:293:in `find_by'
	from /var/www/discourse/lib/discourse.rb:884:in `system_user'
	from script/discourse:143:in `restore'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.3.1/lib/thor/command.rb:28:in `run'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.3.1/lib/thor/invocation.rb:127:in `invoke_command'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.3.1/lib/thor.rb:527:in `dispatch'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.3.1/lib/thor/base.rb:584:in `start'
	from script/discourse:290:in `<top (required)>'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli/exec.rb:58:in `load'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli/exec.rb:58:in `kernel_load'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli/exec.rb:23:in `run'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli.rb:451:in `exec'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli.rb:34:in `dispatch'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli.rb:28:in `start'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/exe/bundle:28:in `block in <top (required)>'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/exe/bundle:20:in `<top (required)>'
	from /usr/local/bin/bundle:25:in `load'
	from /usr/local/bin/bundle:25:in `<main>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.3.1/lib/patches/db/pg.rb:110:in `exec': ERROR:  relation "users" does not exist (PG::UndefinedTable)
LINE 9:  WHERE a.attrelid = '"users"'::regclass
                            ^

	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.3.1/lib/patches/db/pg.rb:110:in `async_exec'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:19:in `block (2 levels) in query'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.1/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:18:in `block in query'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract_adapter.rb:752:in `block in log'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract_adapter.rb:743:in `log'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:17:in `query'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql_adapter.rb:916:in `column_definitions'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:117:in `columns'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/schema_cache.rb:117:in `block in columns'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/schema_cache.rb:116:in `fetch'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/schema_cache.rb:116:in `columns'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/schema_cache.rb:125:in `block in columns_hash'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/schema_cache.rb:124:in `fetch'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/schema_cache.rb:124:in `columns_hash'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:580:in `load_schema!'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/attributes.rb:264:in `load_schema!'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/encryption/encryptable_record.rb:122:in `load_schema!'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:566:in `block in load_schema'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:563:in `synchronize'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:563:in `load_schema'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/model_schema.rb:419:in `columns_hash'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/core.rb:309:in `block in find_by'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/core.rb:293:in `each'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/core.rb:293:in `each_with_object'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.1/lib/active_record/core.rb:293:in `find_by'
	from /var/www/discourse/lib/discourse.rb:884:in `system_user'
	from script/discourse:143:in `restore'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.3.1/lib/thor/command.rb:28:in `run'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.3.1/lib/thor/invocation.rb:127:in `invoke_command'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.3.1/lib/thor.rb:527:in `dispatch'
	from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/thor-1.3.1/lib/thor/base.rb:584:in `start'
	from script/discourse:290:in `<top (required)>'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli/exec.rb:58:in `load'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli/exec.rb:58:in `kernel_load'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli/exec.rb:23:in `run'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli.rb:451:in `exec'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli.rb:34:in `dispatch'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli.rb:28:in `start'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/exe/bundle:28:in `block in <top (required)>'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
	from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/exe/bundle:20:in `<top (required)>'
	from /usr/local/bin/bundle:25:in `load'
	from /usr/local/bin/bundle:25:in `<main>'

Thank you very much

[pfaffman (Jay Pfaffman)]

Was this a standard install and was this installation working before you started the install?

Did you start it from the command line or the web interface?

There is a slight chance it’s a bug, but that seems unlikely.

[Alienazk]

It was a standard install and it worked well before I started the restoration.
At the first time, I start it from web interface and it failed as following pic.

Next time, I start it from command line, and the output comes the second time.

[Alienazk]

I believe it’s happening due to the ongoing restoration process.
The previous issue has been resolved.
However, we’ve encountered another problem where both our memory and swap are fully utilized, resulting in a significant slowdown of our site. I’m struggling to identify the cause.
Would anyone be able to offer some insights?
Thank you very much!

[pfaffman (Jay Pfaffman)]

My guess is that it’s as simple as you need more ram. At the very least add more swap.

How big is your database?

[Ed_S]

That’s a lot of unicorns - it’s the normal number, 2 per CPU, but for only 4G RAM it’s a lot. I’d suggest reducing the number of unicorns.

There are also some kernel configs you might usefully tweak to improve virtual memory use. See

@Ed_S

But there’s also the question of the kernel tunable as mentioned in
MKJ’s Opinionated Discourse Deployment Configuration
which I’ve got set correctly, but which maybe lots of people don’t have set correctly.

[Alienazk]

Do we need more ram?

[Ed_S]

The best way to see if you need more RAM is to watch for paging activity.

I use

vmstat 5 5

because it leaves a record that I can cut and paste. Some will use top or htop.

[Alienazk]

Thanks for the suggestion.
How about lowering the number of Unicorn processes to 4?

[Ed_S]

@Alienazk

lowering the number of Unicorn processes

Worth a try. It depends on your traffic. I have just 40-50 daily active users, and two unicorns is fine.

Another point: if swap is maxxed out you should add swap. It can’t do any harm, and it might save you from bad consequences of processes being killed.

[Alienazk]

Thank you!
By the way, could we change the number of unicorns without rebuilding container?

[Ed_S]

@Alienazk

change the number of unicorns without rebuilding container

I don’t think so.