Hangs discourse and discourse restore FAILED

yes, several times, with the same result

Was on a branch 3d050bdaa31633a954758894629c0eb9fea537d0

want to upgrade to fe71c43c57c0248a8264245cb6ff0dc114da335a

and hangs discourse !!

production.log

Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH) subscribe failed, reconnecting in 1 second. Call stack ["/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:363:in `rescue in establish_connection'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:344:in `establish_connection'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:106:in `block in connect'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:307:in `with_reconnect'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:105:in `connect'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:382:in `ensure_connected'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:231:in `block in process'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:320:in `logging'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:230:in `process'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:139:in `block in call_loop'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:293:in `with_socket_timeout'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:138:in `call_loop'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/subscribe.rb:44:in `subscription'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/subscribe.rb:13:in `subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:3468:in `_subscription'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:2301:in `block in subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:51:in `block in synchronize'", "/usr/local/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:51:in `synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:2300:in `subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus/backends/redis.rb:287:in `global_subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus.rb:741:in `global_subscribe_thread'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus.rb:689:in `block in new_subscriber_thread'"]
Job exception: Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH)

Job exception: Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH)

Job exception: Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH)

Job exception: Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH)

Job exception: Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH)

Job exception: Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH)

Job exception: Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH)

Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH) subscribe failed, reconnecting in 1 second. Call stack ["/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:363:in `rescue in establish_connection'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:344:in `establish_connection'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:106:in `block in connect'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:307:in `with_reconnect'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:105:in `connect'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:382:in `ensure_connected'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:231:in `block in process'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:320:in `logging'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:230:in `process'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:125:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:915:in `block in get'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:51:in `block in synchronize'", "/usr/local/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:51:in `synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:914:in `get'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus/backends/redis.rb:360:in `process_global_backlog'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus/backends/redis.rb:271:in `block in global_subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus/backends/redis.rb:284:in `global_subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus.rb:741:in `global_subscribe_thread'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus.rb:689:in `block in new_subscriber_thread'"]
Job exception: Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH)

Creating scope :visible_groups. Overwriting existing method Group.visible_groups.
Creating scope :visible. Overwriting existing method Notification.visible.
Creating scope :public_posts. Overwriting existing method Post.public_posts.
Creating scope :private_posts. Overwriting existing method Post.private_posts.
Creating scope :open. Overwriting existing method Poll.open.
Migrating to CreateDiscourseVotingCategorySettings (20200727220143)
Migrating to CreateDiscourseVotingVotes (20200728222920)
Migrating to CreateDiscourseVotingTopicVoteCount (20200729042607)
Creating scope :visible_groups. Overwriting existing method Group.visible_groups.
Creating scope :visible. Overwriting existing method Notification.visible.
Creating scope :public_posts. Overwriting existing method Post.public_posts.
Creating scope :private_posts. Overwriting existing method Post.private_posts.
Creating scope :open. Overwriting existing method Poll.open.
Creating scope :visible_groups. Overwriting existing method Group.visible_groups.
Creating scope :visible. Overwriting existing method Notification.visible.
Creating scope :public_posts. Overwriting existing method Post.public_posts.
Creating scope :private_posts. Overwriting existing method Post.private_posts.
Creating scope :open. Overwriting existing method Poll.open.

ubuntu 14.04 seems quite old, I have seen in the last weeks some posts where it has been an issue :
https://meta.discourse.org/search?q=14.04%20order%3Alatest

maybe a bump in a dependency between may and now broke something :roll_eyes:

3 Likes

Thanks the answers

I have the new server with ubuntu 20.04.1 LTS and docker on it.

How can I move my old prod-forum on this server?
Could you suggest step by step manuals for me?

Build from scratch using normal guide. Copy over app.yml. Rebuild. Import Discourse backup file from first server. Done.

(Oh and small matter of remapping IP for domain!)

3 Likes

cool, but creating backups through the admin panel not works for us))

transfer docker container

docker ps -a

docker commit c559bec6f29a local_discourse/app
docker save local_discourse/app > /tmp/local_discourse_app.tar.gz
scp /tmp/local_discourse_app.tar.gz root@my-new-server:/tmp/
---
docker load < /tmp/local_discourse_app.tar.gz
docker run local_discourse/app

problems occur at this step - NoMethodError on docker run

1 Like

You don’t have any existing backups offsite?

2 Likes

I need a copy with fresh data (no fresh backups)

1 Like

When restoring a backup on a new server, errors occur?

./launcher enter app
discourse restore discourse-2020-08-24-103334-v20200811004537.tar.gz 

Hey @Dev_Work

Sorry to hear you have having issues with this.

So, to review:

  • You did a full manual backup of your site and that backup is:
discourse-2020-08-24-103334-v20200811004537.tar.gz 
  • Then you installed Discourse from scratch using the standard method:
https://github.com/discourse/discourse/blob/master/docs/INSTALL-cloud.md
  • You did this on a shiny new ubuntu 20.04.1 LTS server with docker installed and tested.

  • Then you manually restored with the backup above on the new server digs?

Is this correct, and this is where you are now in the caper?

2 Likes

Thanks for the answer.
Yes, that’s right, a problem arises on the new ubuntu 20.04.1 LTS

Did you create an admin with username “pavel_BLANKEDOUT” before attempting the manual discourse restore process?

I am guessing this is not the issue, just asking to cover all bases since user “pavel_BLANKEDOUT”" is mentioned in your duplicate key error.

No, did not create.

User admin who is not in the backup database.

Hey @Dev_Work

And your original site where the backup you are using is down completely?

Or can you enter the container of your old site and check the discourse DB for a corrupted index?

Also, your old broken instance was running what version of PostgreSQL? 10 or 12?

2 Likes

Sounds like my problem, Search results for 'could not create unique index category:6' - Discourse Meta

cry :frowning:

ERROR:  could not create unique index "index_user_emails_on_email"
DETAIL:  Key (lower(email::text))=(andrii_test@local.com) is duplicated.
EXCEPTION: psql failed: DETAIL:  Key (lower(email::text))=(andrii_test@local.com) is duplicated.
/var/www/discourse/lib/backup_restore/database_restorer.rb:95:in `restore_dump'
/var/www/discourse/lib/backup_restore/database_restorer.rb:26:in `restore'
/var/www/discourse/lib/backup_restore/restorer.rb:49:in `run'
script/discourse:143:in `restore'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/thor-1.0.1/lib/thor/base.rb:485:in `start'
script/discourse:284:in `<top (required)>'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `load'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `kernel_load'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:28:in `run'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:476:in `exec'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:30:in `dispatch'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:24:in `start'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'

Old server used PostgreSQL 12

Old site and server working

i can enter the container of old site (how to check corrupted index?)

Looks like you have two users with the same email in the backup? Delete one of those users or change one email and create a new backup?

1 Like

You’ve got a corrupt index. I had a similar issue (with a differnent table and index, I think), and described what I did to fix it at Can't restore due to corrupt indexes (with some clues on how to deal with corrupt indexes). Your solution will be different, but what I describe there might be what you need to figure out how to solve your problem.

3 Likes

Hey @Dev_Work

That is good news.

You have a number of options. Off the top of my head, here are only two:

Analyze and reindex the user table:

VACUUM ANALYZE users;

and then you can:

REINDEX VERBOSE TABLE users

This will analyze and reindex on a “per table” basis (“users” in the example above).

You can also do the same thing for the entire database

VACUUM;
REINDEX DATABASE discourse;

Either way, after you REINDEX your “old DB”, you can then create a new backup and see how that goes when you restore to your “new DB”.

Hope this helps.

See Also:

https://medium.com/@andreigridnev/db-maintenance-commands-in-postgresql-analyze-reindex-vacuum-4a14d75bee6c

4 Likes

Why is this error happening?

[STARTED]
'system' has started the restore!
Marking restore as running...
Making sure /var/www/discourse/tmp/restores/default/2020-08-26-075829 exists...
Copying archive to tmp directory...
Unzipping archive, this may take a while...
Extracting dump file...
EXCEPTION: No space left on device @ io_write - /shared/tmp/restores/default/2020-08-26-075829/dump.sql
/var/www/discourse/lib/compression/gzip.rb:50:in `write'
/var/www/discourse/lib/compression/gzip.rb:50:in `<<'
/var/www/discourse/lib/compression/gzip.rb:50:in `block in extract_file'
/var/www/discourse/lib/compression/gzip.rb:45:in `open'
/var/www/discourse/lib/compression/gzip.rb:45:in `extract_file'
/var/www/discourse/lib/compression/strategy.rb:26:in `block (2 levels) in decompress'
/var/www/discourse/lib/compression/strategy.rb:18:in `each'
/var/www/discourse/lib/compression/strategy.rb:18:in `block in decompress'
/var/www/discourse/lib/compression/gzip.rb:30:in `get_compressed_file_stream'
/var/www/discourse/lib/compression/strategy.rb:15:in `decompress'
/var/www/discourse/lib/backup_restore/backup_file_handler.rb:86:in `extract_db_dump'
/var/www/discourse/lib/backup_restore/backup_file_handler.rb:23:in `decompress'
/var/www/discourse/lib/backup_restore/restorer.rb:42:in `run'
script/discourse:143:in `restore'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/thor-1.0.1/lib/thor/base.rb:485:in `start'
script/discourse:284:in `<top (required)>'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `load'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `kernel_load'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:28:in `run'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:476:in `exec'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:30:in `dispatch'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:24:in `start'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Trying to rollback...
There was no need to rollback
Cleaning stuff up...
Removing tmp '/var/www/discourse/tmp/restores/default/2020-08-26-075829' directory...
Unpausing sidekiq...
Marking restore as finished...
Notifying 'system' of the end of the restore...
Finished!
[FAILED]
Restore done.