فشل الاستعادة، وتعليق، وانتهاء مهلة في ALTER TABLE

الدعم
1

I had a disk failure on a discourse server, and I’m trying to restore a backup from S3 onto an AWS EC2 t2.small instance. Consistently getting the below error.

Using this flow:

git clone https://github.com/discourse/discourse_docker /var/discourse

I’m using this app.yml

./launcher bootstrap app
./launcher start app

Then I scp the backup, ./launcher enter app, discourse enable_restore discourse restore <the file>.

Then this happens, beginning truncated:

14
(1 row)
setval
--------
1
(1 row)
setval
--------
1
(1 row)
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
 #<Thread:0x0000562604925448 /var/www/discourse/lib/sidekiq/pausable.rb:79 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
        33: from /var/www/discourse/lib/sidekiq/pausable.rb:83:in `block (2 levels) in extend_lease_thread'
        32: from /var/www/discourse/lib/sidekiq/pausable.rb:83:in `synchronize'
        31: from /var/www/discourse/lib/sidekiq/pausable.rb:84:in `block (3 levels) in extend_lease_thread'
        30: from /usr/local/lib/ruby/2.7.0/set.rb:328:in `each'
        29: from /usr/local/lib/ruby/2.7.0/set.rb:328:in `each_key'
        28: from /var/www/discourse/lib/sidekiq/pausable.rb:85:in `block (4 levels) in extend_lease_thread'
        27: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-2.5.0/lib/rails_multisite/connection_management.rb:76:in `with_connection'
        26: from /var/www/discourse/lib/sidekiq/pausable.rb:86:in `block (5 levels) in extend_lease_thread'
        25: from /var/www/discourse/lib/discourse_redis.rb:59:in `block (2 levels) in <class:DiscourseRedis>'
        24: from /var/www/discourse/lib/discourse_redis.rb:29:in `ignore_readonly'
        23: from /var/www/discourse/lib/discourse_redis.rb:59:in `block (3 levels) in <class:DiscourseRedis>'
        22: from /var/www/discourse/lib/discourse_redis.rb:59:in `public_send'
        21: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis.rb:425:in `expire'
        20: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis.rb:69:in `synchronize'
        19: from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `mon_synchronize'
        18: from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `synchronize'
        17: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis.rb:69:in `block in synchronize'
        16: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis.rb:426:in `block in expire'
        15: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rack-mini-profiler-2.3.1/lib/mini_profiler/profiling_methods.rb:85:in `block in profile_method'
        14: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:131:in `call'
        13: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:237:in `process'
        12: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:325:in `logging'
        11: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:238:in `block in process'
        10: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:389:in `ensure_connected'
         9: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:248:in `block (2 levels) in process'
         8: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:131:in `block in call'
         7: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:279:in `read'
         6: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:268:in `io'
         5: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:280:in `block in read'
         4: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:378:in `read'
         3: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:45:in `gets'
         2: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:52:in `_read_from_socket'
         1: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:52:in `loop'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:56:in `block in _read_from_socket': Redis::TimeoutError (Redis::TimeoutError)
        33: from /var/www/discourse/lib/sidekiq/pausable.rb:83:in `block (2 levels) in extend_lease_thread'
        32: from /var/www/discourse/lib/sidekiq/pausable.rb:83:in `synchronize'
        31: from /var/www/discourse/lib/sidekiq/pausable.rb:84:in `block (3 levels) in extend_lease_thread'
        30: from /usr/local/lib/ruby/2.7.0/set.rb:328:in `each'
        29: from /usr/local/lib/ruby/2.7.0/set.rb:328:in `each_key'
        28: from /var/www/discourse/lib/sidekiq/pausable.rb:85:in `block (4 levels) in extend_lease_thread'
        27: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-2.5.0/lib/rails_multisite/connection_management.rb:76:in `with_connection'
        26: from /var/www/discourse/lib/sidekiq/pausable.rb:86:in `block (5 levels) in extend_lease_thread'
        25: from /var/www/discourse/lib/discourse_redis.rb:59:in `block (2 levels) in <class:DiscourseRedis>'
        24: from /var/www/discourse/lib/discourse_redis.rb:29:in `ignore_readonly'
        23: from /var/www/discourse/lib/discourse_redis.rb:59:in `block (3 levels) in <class:DiscourseRedis>'
        22: from /var/www/discourse/lib/discourse_redis.rb:59:in `public_send'
        21: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis.rb:425:in `expire'
        20: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis.rb:69:in `synchronize'
        19: from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `mon_synchronize'
        18: from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `synchronize'
        17: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis.rb:69:in `block in synchronize'
        16: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis.rb:426:in `block in expire'
        15: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rack-mini-profiler-2.3.1/lib/mini_profiler/profiling_methods.rb:85:in `block in profile_method'
        14: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:131:in `call'
        13: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:237:in `process'
        12: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:325:in `logging'
        11: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:238:in `block in process'
        10: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:389:in `ensure_connected'
         9: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:248:in `block (2 levels) in process'
         8: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:131:in `block in call'
         7: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:279:in `read'
         6: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:268:in `io'
         5: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/client.rb:280:in `block in read'
         4: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:378:in `read'
         3: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:45:in `gets'
         2: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:52:in `_read_from_socket'
         1: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:52:in `loop'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:56:in `block in _read_from_socket': Connection timed out (Redis::TimeoutError)
^[[A  ^[[A^[[B       server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
connection to server was lost
EXCEPTION: psql failed: connection to server was lost

Consistently, the process will reach this far into the restore process, then hang on that last (85th) ALTER TABLE. I extracted my backup’s dump.sql, and the 85th is, if I’m counting right, this “public.reviewables” one:

image
image1216×700 11 KB

Any idea from the above what might be going wrong?

2

أعتقد أنني وجدت الحل. كان يجب أن أشغّل ./discourse-setup مرة واحدة على المضيف الجديد لـ (1) إنشاء ملف تبديل، و (2) ضبط db_shared_buffers بشكل صحيح. ربما كان أحد هذين الخطأين، أو كلاهما، هو ما تسبب في التعليد المستمر.

3

أحسنت! أهلاً بك مرة أخرى، وأهلاً بك في نادي “أجب على سؤالك الخاص”! :wink: لديّ مجموعة من هذه المواضيع، على الرغم من أنني متأكد من أن العديد منها قد حُذِف.

الأمر الآخر هو نسخ ملف app.yml، لكنك كنت ستظل بحاجة إلى تذكر عملية التبديل. باستثناء مشكلة التبديل، فإن ملف yml هو كل ما تحتاجه إذا كانت إعدادات S3 موجودة في ملف yml.

سعيد بعودتك للعمل!