# Restore fails, hangs, timeouts on ALTER TABLE

**URL:** https://meta.discourse.org/t/restore-fails-hangs-timeouts-on-alter-table/185309
**Category:** Support
**Created:** [April 2, 2021, 11:43pm UTC](https://meta.discourse.org/t/restore-fails-hangs-timeouts-on-alter-table/185309 "2021-04-02T23:43:10Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![deargle](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/deargle/32/120295_2.png) [@deargle](https://meta.discourse.org/u/deargle)
#### Post date: [April 2, 2021, 11:43pm UTC](https://meta.discourse.org/t/restore-fails-hangs-timeouts-on-alter-table/185309/1 "2021-04-02T23:43:11Z")

</div>

I had a disk failure on a discourse server, and I’m trying to restore a backup from S3 onto an AWS EC2 t2.small instance. Consistently getting the below error.

Using this flow:

```plaintext
git clone https://github.com/discourse/discourse_docker /var/discourse

```

I’m using [this app.yml](https://github.com/deargle/island-vNext/blob/master/discourse-containers/app.yml)

`./launcher bootstrap app`  
`./launcher start app`

Then I scp the backup, `./launcher enter app`, `discourse enable_restore` `discourse restore <the file>`.

Then this happens, beginning truncated:

```plaintext
14
(1 row)
setval
--------
1
(1 row)
setval
--------
1
(1 row)
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
 #<Thread:0x0000562604925448 /var/www/discourse/lib/sidekiq/pausable.rb:79 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
        33: from /var/www/discourse/lib/sidekiq/pausable.rb:83:in `block (2 levels) in extend_lease_thread'
        32: from /var/www/discourse/lib/sidekiq/pausable.rb:83:in `synchronize'
        31: from /var/www/discourse/lib/sidekiq/pausable.rb:84:in `block (3 levels) in extend_lease_thread'
        30: from /usr/local/lib/ruby/2.7.0/set.rb:328:in `each'
        29: from /usr/local/lib/ruby/2.7.0/set.rb:328:in `each_key'
        28: from /var/www/discourse/lib/sidekiq/pausable.rb:85:in `block (4 levels) in extend_lease_thread'
        27: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-2.5.0/lib/rails_multisite/connection_management.rb:76:in `with_connection'
        26: from /var/www/discourse/lib/sidekiq/pausable.rb:86:in `block (5 levels) in extend_lease_thread'
        25: from /var/www/discourse/lib/discourse_redis.rb:59:in `block (2 levels) in <class:DiscourseRedis>'
        24: from /var/www/discourse/lib/discourse_redis.rb:29:in `ignore_readonly'
        23: from /var/www/discourse/lib/discourse_redis.rb:59:in `block (3 levels) in <class:DiscourseRedis>'
        22: from /var/www/discourse/lib/discourse_redis.rb:59:in `public_send'
        21: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis.rb:425:in `expire'
        20: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis.rb:69:in `synchronize'
        19: from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `mon_synchronize'
        18: from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `synchronize'
        17: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis.rb:69:in `block in synchronize'
        16: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis.rb:426:in `block in expire'
        15: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rack-mini-profiler-2.3.1/lib/mini_profiler/profiling_methods.rb:85:in `block in profile_method'
        14: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:131:in `call'
        13: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:237:in `process'
        12: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:325:in `logging'
        11: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:238:in `block in process'
        10: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:389:in `ensure_connected'
         9: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:248:in `block (2 levels) in process'
         8: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:131:in `block in call'
         7: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:279:in `read'
         6: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:268:in `io'
         5: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:280:in `block in read'
         4: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:378:in `read'
         3: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:45:in `gets'
         2: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:52:in `_read_from_socket'
         1: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:52:in `loop'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:56:in `block in _read_from_socket': Redis::TimeoutError (Redis::TimeoutError)
        33: from /var/www/discourse/lib/sidekiq/pausable.rb:83:in `block (2 levels) in extend_lease_thread'
        32: from /var/www/discourse/lib/sidekiq/pausable.rb:83:in `synchronize'
        31: from /var/www/discourse/lib/sidekiq/pausable.rb:84:in `block (3 levels) in extend_lease_thread'
        30: from /usr/local/lib/ruby/2.7.0/set.rb:328:in `each'
        29: from /usr/local/lib/ruby/2.7.0/set.rb:328:in `each_key'
        28: from /var/www/discourse/lib/sidekiq/pausable.rb:85:in `block (4 levels) in extend_lease_thread'
        27: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-2.5.0/lib/rails_multisite/connection_management.rb:76:in `with_connection'
        26: from /var/www/discourse/lib/sidekiq/pausable.rb:86:in `block (5 levels) in extend_lease_thread'
        25: from /var/www/discourse/lib/discourse_redis.rb:59:in `block (2 levels) in <class:DiscourseRedis>'
        24: from /var/www/discourse/lib/discourse_redis.rb:29:in `ignore_readonly'
        23: from /var/www/discourse/lib/discourse_redis.rb:59:in `block (3 levels) in <class:DiscourseRedis>'
        22: from /var/www/discourse/lib/discourse_redis.rb:59:in `public_send'
        21: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis.rb:425:in `expire'
        20: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis.rb:69:in `synchronize'
        19: from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `mon_synchronize'
        18: from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `synchronize'
        17: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis.rb:69:in `block in synchronize'
        16: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis.rb:426:in `block in expire'
        15: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rack-mini-profiler-2.3.1/lib/mini_profiler/profiling_methods.rb:85:in `block in profile_method'
        14: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:131:in `call'
        13: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:237:in `process'
        12: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:325:in `logging'
        11: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:238:in `block in process'
        10: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:389:in `ensure_connected'
         9: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:248:in `block (2 levels) in process'
         8: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:131:in `block in call'
         7: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:279:in `read'
         6: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:268:in `io'
         5: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:280:in `block in read'
         4: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:378:in `read'
         3: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:45:in `gets'
         2: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:52:in `_read_from_socket'
         1: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:52:in `loop'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:56:in `block in _read_from_socket': Connection timed out (Redis::TimeoutError)
^[[A ^[[A^[[B server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
connection to server was lost
EXCEPTION: psql failed: connection to server was lost

```

Consistently, the process will reach this far into the restore process, then hang on that last (85th) `ALTER TABLE`. I extracted my backup’s dump.sql, and the 85th is, if I’m counting right, this “public.reviewables” one:

 ![image](https://global.discourse-cdn.com/meta/original/3X/5/b/5b4fd6c9469383a9a55b2a46fbfd0292879713df.png)

Any idea from the above what might be going wrong?

---

<div class="post-metadata">

### Author: ![deargle](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/deargle/32/120295_2.png) [@deargle](https://meta.discourse.org/u/deargle)
#### Post date: [April 3, 2021, 12:57am UTC](https://meta.discourse.org/t/restore-fails-hangs-timeouts-on-alter-table/185309/2 "2021-04-03T00:57:15Z")

</div>

I think I figured it out. I should have run `./discourse-setup` once on the new host to (1) get a swap file created, and (2) to get a sane `db_shared_buffers` set. One of those two being wrong, maybe both, was causing the consistent hang.

---

<div class="post-metadata">

### Author: ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [April 3, 2021, 1:03am UTC](https://meta.discourse.org/t/restore-fails-hangs-timeouts-on-alter-table/185309/3 "2021-04-03T01:03:19Z")

</div>

Good catch! Welcome back and welcome to the “answer your own question” club! 😉 I’ve got a bunch of those topics, though I’m sure many have been deleted.

The other thing would be to copy the app.yml, but you’d still have had to remember the swap. Except for the swap snag, the yml file is all you need if the S3 config is in the yml.

Glad you’re back up!

---

<div class="post-metadata">

### Author: ![system](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/system/32/443519_2.png) [@system](https://meta.discourse.org/u/system)
#### Post date: [May 3, 2021, 1:03am UTC](https://meta.discourse.org/t/restore-fails-hangs-timeouts-on-alter-table/185309/4 "2021-05-03T01:03:33Z")

</div>

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.