Hangs discourse and discourse restore FAILED

I run the command ./launcher rebuild app


redis is now ready to exit, bye bye ... hangs discourse - 40min, stop ctrl + c

It is loading all time. As result we did not create new container, could you explain what kind of logs in discourse can help us find to the reason?

Could this be a resource allocation issue?

Are there any other websites or services hosted on this server? How much RAM/CPU is available on your server?

server resources
2020-08-18_143023

That’s normal.

That is not. It’s generally much less than a minute after that message that the new container gets started up.

And you’ve tried this more than once with the same result?

yes, several times, with the same result

Was on a branch 3d050bdaa31633a954758894629c0eb9fea537d0

want to upgrade to fe71c43c57c0248a8264245cb6ff0dc114da335a

and hangs discourse !!

production.log

Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH) subscribe failed, reconnecting in 1 second. Call stack ["/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:363:in `rescue in establish_connection'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:344:in `establish_connection'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:106:in `block in connect'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:307:in `with_reconnect'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:105:in `connect'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:382:in `ensure_connected'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:231:in `block in process'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:320:in `logging'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:230:in `process'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:139:in `block in call_loop'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:293:in `with_socket_timeout'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:138:in `call_loop'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/subscribe.rb:44:in `subscription'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/subscribe.rb:13:in `subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:3468:in `_subscription'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:2301:in `block in subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:51:in `block in synchronize'", "/usr/local/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:51:in `synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:2300:in `subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus/backends/redis.rb:287:in `global_subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus.rb:741:in `global_subscribe_thread'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus.rb:689:in `block in new_subscriber_thread'"]
Job exception: Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH)

Job exception: Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH)

Job exception: Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH)

Job exception: Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH)

Job exception: Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH)

Job exception: Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH)

Job exception: Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH)

Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH) subscribe failed, reconnecting in 1 second. Call stack ["/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:363:in `rescue in establish_connection'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:344:in `establish_connection'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:106:in `block in connect'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:307:in `with_reconnect'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:105:in `connect'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:382:in `ensure_connected'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:231:in `block in process'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:320:in `logging'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:230:in `process'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:125:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:915:in `block in get'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:51:in `block in synchronize'", "/usr/local/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:51:in `synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:914:in `get'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus/backends/redis.rb:360:in `process_global_backlog'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus/backends/redis.rb:271:in `block in global_subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus/backends/redis.rb:284:in `global_subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus.rb:741:in `global_subscribe_thread'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus.rb:689:in `block in new_subscriber_thread'"]
Job exception: Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH)

Creating scope :visible_groups. Overwriting existing method Group.visible_groups.
Creating scope :visible. Overwriting existing method Notification.visible.
Creating scope :public_posts. Overwriting existing method Post.public_posts.
Creating scope :private_posts. Overwriting existing method Post.private_posts.
Creating scope :open. Overwriting existing method Poll.open.
Migrating to CreateDiscourseVotingCategorySettings (20200727220143)
Migrating to CreateDiscourseVotingVotes (20200728222920)
Migrating to CreateDiscourseVotingTopicVoteCount (20200729042607)
Creating scope :visible_groups. Overwriting existing method Group.visible_groups.
Creating scope :visible. Overwriting existing method Notification.visible.
Creating scope :public_posts. Overwriting existing method Post.public_posts.
Creating scope :private_posts. Overwriting existing method Post.private_posts.
Creating scope :open. Overwriting existing method Poll.open.
Creating scope :visible_groups. Overwriting existing method Group.visible_groups.
Creating scope :visible. Overwriting existing method Notification.visible.
Creating scope :public_posts. Overwriting existing method Post.public_posts.
Creating scope :private_posts. Overwriting existing method Post.private_posts.
Creating scope :open. Overwriting existing method Poll.open.

ubuntu 14.04 seems quite old, I have seen in the last weeks some posts where it has been an issue :
https://meta.discourse.org/search?q=14.04%20order%3Alatest

maybe a bump in a dependency between may and now broke something :roll_eyes:

3 Likes

Thanks the answers

I have the new server with ubuntu 20.04.1 LTS and docker on it.

How can I move my old prod-forum on this server?
Could you suggest step by step manuals for me?

Build from scratch using normal guide. Copy over app.yml. Rebuild. Import Discourse backup file from first server. Done.

(Oh and small matter of remapping IP for domain!)

3 Likes

cool, but creating backups through the admin panel not works for us))

transfer docker container

docker ps -a

docker commit c559bec6f29a local_discourse/app
docker save local_discourse/app > /tmp/local_discourse_app.tar.gz
scp /tmp/local_discourse_app.tar.gz root@my-new-server:/tmp/
---
docker load < /tmp/local_discourse_app.tar.gz
docker run local_discourse/app

problems occur at this step - NoMethodError on docker run

1 Like

You don’t have any existing backups offsite?

2 Likes

I need a copy with fresh data (no fresh backups)

1 Like

When restoring a backup on a new server, errors occur?

./launcher enter app
discourse restore discourse-2020-08-24-103334-v20200811004537.tar.gz 

Hey @Dev_Work

Sorry to hear you have having issues with this.

So, to review:

  • You did a full manual backup of your site and that backup is:
discourse-2020-08-24-103334-v20200811004537.tar.gz 
  • Then you installed Discourse from scratch using the standard method:
https://github.com/discourse/discourse/blob/master/docs/INSTALL-cloud.md
  • You did this on a shiny new ubuntu 20.04.1 LTS server with docker installed and tested.

  • Then you manually restored with the backup above on the new server digs?

Is this correct, and this is where you are now in the caper?

2 Likes

Thanks for the answer.
Yes, that’s right, a problem arises on the new ubuntu 20.04.1 LTS

Did you create an admin with username “pavel_BLANKEDOUT” before attempting the manual discourse restore process?

I am guessing this is not the issue, just asking to cover all bases since user “pavel_BLANKEDOUT”" is mentioned in your duplicate key error.

No, did not create.

User admin who is not in the backup database.

Hey @Dev_Work

And your original site where the backup you are using is down completely?

Or can you enter the container of your old site and check the discourse DB for a corrupted index?

Also, your old broken instance was running what version of PostgreSQL? 10 or 12?

2 Likes

Sounds like my problem, Search results for 'could not create unique index category:6' - Discourse Meta

cry :frowning:

ERROR:  could not create unique index "index_user_emails_on_email"
DETAIL:  Key (lower(email::text))=(andrii_test@local.com) is duplicated.
EXCEPTION: psql failed: DETAIL:  Key (lower(email::text))=(andrii_test@local.com) is duplicated.
/var/www/discourse/lib/backup_restore/database_restorer.rb:95:in `restore_dump'
/var/www/discourse/lib/backup_restore/database_restorer.rb:26:in `restore'
/var/www/discourse/lib/backup_restore/restorer.rb:49:in `run'
script/discourse:143:in `restore'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/thor-1.0.1/lib/thor/base.rb:485:in `start'
script/discourse:284:in `<top (required)>'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `load'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `kernel_load'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:28:in `run'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:476:in `exec'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:30:in `dispatch'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:24:in `start'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'

Old server used PostgreSQL 12

Old site and server working

i can enter the container of old site (how to check corrupted index?)