POSTGRES 업그레이드 실패 - 모든 방법을 시도해 보았지만

[jameshahnii]

사이트가 갑자기 ‘Welcome to nginx!’ 화면에서 멈춰서 디버깅을 하고 있는데, 아래 에러들이 계속 발생합니다.

Success. You can now start the database server using:

    pg_ctlcluster 10 main start

Warning: The selected stats_temp_directory /var/run/postgresql/10-main.pg_stat_tmp
is not writable for the cluster owner. Not adding this setting in
postgresql.conf.
Ver Cluster Port Status Owner    Data directory              Log file
10  main    5433 down   postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
update-alternatives: warning: forcing reinstallation of alternative /usr/share/postgresql/12/man/man1/postmaster.1.gz because link group postmaster.1.gz is broken
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Processing triggers for postgresql-common (215.pgdg100+1) ...
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
Stopping PostgreSQL 10 database server: main.
Stopping PostgreSQL 12 database server: main.
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok

The source cluster was not shut down cleanly.
Failure, exiting
-------------------------------------------------------------------------------------
UPGRADE OF POSTGRES FAILED

Please visit https://meta.discourse.org/t/postgresql-12-update/151236 for support

You can run ./launcher start app to restart your app in the meanwhile



FAILED
--------------------
Pups::ExecError: /root/upgrade_postgres failed with return #<Process::Status: pid 45 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params "/root/upgrade_postgres"
bfe8265213ad992fa3245d252d192977f05d902d7213b361f53d3bc2b0d16b3a
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
root@bitkcor:/var/discourse# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
tmpfs           395M  648K  394M   1% /run
/dev/vda1        78G   23G   55G  30% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vda15      105M  3.4M  101M   4% /boot/efi
tmpfs           395M     0  395M   0% /run/user/0
root@bitkcor:/var/discourse# ls
bin   containers        discourse-setup  launcher  README.md  scripts  templates
cids  discourse-doctor  image            LICENSE   samples    shared
root@bitkcor:/var/discourse# cd shared
root@bitkcor:/var/discourse/shared# chown -R postgres postgres_data
chown: invalid user: ‘postgres’
root@bitkcor:/var/discourse/shared# ./launcher start app
-bash: ./launcher: No such file or directory
root@bitkcor:/var/discourse/shared# cd ..
root@bitkcor:/var/discourse# ./launcher start app

starting up existing container
+ /usr/bin/docker start app
Error response from daemon: driver failed programming external connectivity on endpoint app (10f3d3b7938496e743c73affc9ddb2d821e04419985bf8e3ebde2ec9ec704a0b): Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use
Error: failed to start containers: app
root@bitkcor:/var/discourse# cd containers
root@bitkcor:/var/discourse/containers# ls
app.yml  app.yml.2018-08-29-034711.bak
root@bitkcor:/var/discourse/containers# cd ..
root@bitkcor:/var/discourse# ./launcher stop app
+ /usr/bin/docker stop -t 10 app
app
root@bitkcor:/var/discourse# ./launcher stop app
+ /usr/bin/docker stop -t 10 app
app
root@bitkcor:/var/discourse# git pull
Already up to date.
root@bitkcor:/var/discourse# ./launcher start app

starting up existing container
+ /usr/bin/docker start app
Error response from daemon: driver failed programming external connectivity on endpoint app (f149fe57c125ae0c276c339a1367c83262f866227ab9317ff06bf026e8776f65): Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use
Error: failed to start containers: app

https://meta.discourse.org/t/postgresql-12-update/151236 링크에서 지원을 요청했고, @Falco 님의 여기 제안 사항과 메타 포럼의 다른 제안들도 시도해 보았습니다. 하지만 아무것도 해결되지 않습니다.

제가 아는 한, 이 작업을 수행할 충분한 디스크 공간은 확보되어 있습니다.

제가 무엇을 놓치고 있는지 알려주실 수 있을까요?

[Falco]

@jameshahnii

The source cluster was not shut down cleanly.

이와 관련된 FAQ 항목이 PostgreSQL 12 업데이트에 있습니다.

[jameshahnii]

빠른 답변 감사합니다! 네, 저도 그 방법을 시도해 보았습니다.

root@bitkcor:/var/discourse# ./launcher start app

starting up existing container
+ /usr/bin/docker start app
Error response from daemon: driver failed programming external connectivity on endpoint app (49f1fdf896618efc824e50f782c1fba91bf81320e49ccadb5e5e80b342552e3e): Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use
Error: failed to start containers: app
root@bitkcor:/var/discourse# ./launcher stop app
+ /usr/bin/docker stop -t 10 app
app
root@bitkcor:/var/discourse# tail -f shared/data/log/var-log/postgres/current
tail: cannot open 'shared/data/log/var-log/postgres/current' for reading: No such file or directory
tail: no files remaining

[Falco]

@jameshahnii

listen tcp 0.0.0.0:80: bind: address already in use

어떤 방식으로든 누군가 호스트에 nginx를 설치한 것 같습니다. VPS에서 불필요한 웹 서버를 제거한 후 설명을 따라 진행해 주세요.

[jameshahnii]

nginx가 설치되어 있는 이유는 이 포럼이 몇 년 동안 정상적으로 작동해 왔기 때문입니다. Digital Ocean에서 작업을 늦게 처리하여 Droplet이 꺼진 상태였습니다. 다시 켠 후 해당 주소로 접속하면 521 오류가 발생합니다. IP 주소로 직접 접속하면 다시 Welcome 화면이 표시됩니다.

Cloudflare 설정은 변경된 적이 없습니다.

또한 NameCheap에서 네임서버 설정도 정확합니다.

몇 달 동안 아무것도 변경한 적이 없어서 어디서부터 확인해야 할지 막막합니다.

[Falco]

@jameshahnii

Droplet이 꺼져 있네요

nginx는 설치되어 있었지만 실행되지 않고 있었던 것이죠. Droplet을 끄고 다시 켜면서 서비스가 재시작되었고, 이로 인해 웹 포트가 차단되고 있습니다.

서버가 재시작될 때마다 이런 일이 반복되지 않도록 nginx를 완전히 제거하는 것이 좋습니다…

[jameshahnii]

무언가 바뀌었나요? 이걸 설치했을 때 nginx의 환영 화면이 설치 과정의 일부였습니다. nginx가 Discourse를 실행하는 데 필수인 줄 알았거든요. 어떻게 제거하나요?

[itsbhanusharma (Bhanu Sharma)]

@jameshahnii

shared/data/log/var-log/postgres/current

경로를
tail -f shared/standalone/log/var-log/postgres/current

으로 변경해 주세요.

가이드에 언급된 경로는 두 개의 컨테이너(데이터 컨테이너) 설치 환경을 가리키지만, 사용자의 설치는 단일 컨테이너 방식인 것으로 보입니다.

[jameshahnii]

완료. 이 결과가 나옵니다.

root@bitkcor:/var/discourse# tail -f shared/standalone/log/var-log/postgres/current
2020-07-19 03:33:56.864 UTC [19933] discourse@discourse LOG:  duration: 279.207 ms  statement: COPY public.scheduler_stats (id, name, hostname, pid, duration_ms, live_slots_start, live_slots_finish, started_at, success, error) TO stdout;
2020-07-19 03:34:09.436 UTC [19933] discourse@discourse LOG:  duration: 12555.420 ms  statement: COPY public.stylesheet_cache (id, target, digest, content, created_at, updated_at, theme_id, source_map) TO stdout;
2020-07-19 03:34:10.211 UTC [19933] discourse@discourse LOG:  duration: 727.297 ms  statement: COPY public.unsubscribe_keys (key, user_id, created_at, updated_at, unsubscribe_key_type, topic_id, post_id) TO stdout;
2020-07-21 01:56:22.105 UTC [6388] discourse@discourse LOG:  duration: 167.853 ms  execute <unnamed>: INSERT INTO "unsubscribe_keys" ("key", "user_id", "created_at", "updated_at", "unsubscribe_key_type") VALUES ('352fc5679876a1a700dfe7b45f8fa67612592421a3659e08ec5c2ccbf8f0e2d2', 2, '2020-07-21 01:56:21.932109', '2020-07-21 01:56:21.932109', 'digest') RETURNING "key"
2020-07-26 03:34:50.570 UTC [27570] discourse@discourse LOG:  duration: 147.456 ms  statement: COPY public.post_revisions (id, user_id, post_id, modifications, number, created_at, updated_at, hidden) TO stdout;
2020-07-26 03:34:50.925 UTC [27570] discourse@discourse LOG:  duration: 349.648 ms  statement: COPY public.post_search_data (post_id, search_data, raw_data, locale, version) TO stdout;
2020-07-26 03:34:51.236 UTC [27570] discourse@discourse LOG:  duration: 292.799 ms  statement: COPY public.posts (id, user_id, topic_id, post_number, raw, cooked, created_at, updated_at, reply_to_post_number, reply_count, quote_count, deleted_at, off_topic_count, like_count, incoming_link_count, bookmark_count, avg_time, score, reads, post_type, sort_order, last_editor_id, hidden, hidden_reason_id, notify_moderators_count, spam_count, illegal_count, inappropriate_count, last_version_at, user_deleted, reply_to_user_id, percent_rank, notify_user_count, like_score, deleted_by_id, edit_reason, word_count, version, cook_method, wiki, baked_at, baked_version, hidden_at, self_edits, reply_quoted, via_email, raw_email, public_version, action_code, image_url, locked_by_id) TO stdout;
2020-07-26 03:34:51.547 UTC [27570] discourse@discourse LOG:  duration: 296.400 ms  statement: COPY public.scheduler_stats (id, name, hostname, pid, duration_ms, live_slots_start, live_slots_finish, started_at, success, error) TO stdout;
2020-07-26 03:35:04.123 UTC [27570] discourse@discourse LOG:  duration: 12549.364 ms  statement: COPY public.stylesheet_cache (id, target, digest, content, created_at, updated_at, theme_id, source_map) TO stdout;
2020-07-26 03:35:04.760 UTC [27570] discourse@discourse LOG:  duration: 588.788 ms  statement: COPY public.unsubscribe_keys (key, user_id, created_at, updated_at, unsubscribe_key_type, topic_id, post_id) TO stdout;

[itsbhanusharma (Bhanu Sharma)]

Postgres가 실행 중인 것 같습니다. 로그를 확인하기 전에 Discourse 컨테이너를 종료했나요?

./launcher stop app

[jameshahnii]

그것을 멈추는 법을 모르겠습니다.

root@bitkcor:/var/discourse# ./launcher stop app
+ /usr/bin/docker stop -t 10 app
app
root@bitkcor:/var/discourse# tail -f shared/standalone/log/var-log/postgres/current
2020-07-19 03:33:56.864 UTC [19933] discourse@discourse LOG:  duration: 279.207 ms  statement: COPY public.scheduler_stats (id, name, hostname, pid, duration_ms, live_slots_start, live_slots_finish, started_at, success, error) TO stdout;
2020-07-19 03:34:09.436 UTC [19933] discourse@discourse LOG:  duration: 12555.420 ms  statement: COPY public.stylesheet_cache (id, target, digest, content, created_at, updated_at, theme_id, source_map) TO stdout;
2020-07-19 03:34:10.211 UTC [19933] discourse@discourse LOG:  duration: 727.297 ms  statement: COPY public.unsubscribe_keys (key, user_id, created_at, updated_at, unsubscribe_key_type, topic_id, post_id) TO stdout;
2020-07-21 01:56:22.105 UTC [6388] discourse@discourse LOG:  duration: 167.853 ms  execute <unnamed>: INSERT INTO "unsubscribe_keys" ("key", "user_id", "created_at", "updated_at", "unsubscribe_key_type") VALUES ('352fc5679876a1a700dfe7b45f8fa67612592421a3659e08ec5c2ccbf8f0e2d2', 2, '2020-07-21 01:56:21.932109', '2020-07-21 01:56:21.932109', 'digest') RETURNING "key"
2020-07-26 03:34:50.570 UTC [27570] discourse@discourse LOG:  duration: 147.456 ms  statement: COPY public.post_revisions (id, user_id, post_id, modifications, number, created_at, updated_at, hidden) TO stdout;
2020-07-26 03:34:50.925 UTC [27570] discourse@discourse LOG:  duration: 349.648 ms  statement: COPY public.post_search_data (post_id, search_data, raw_data, locale, version) TO stdout;
2020-07-26 03:34:51.236 UTC [27570] discourse@discourse LOG:  duration: 292.799 ms  statement: COPY public.posts (id, user_id, topic_id, post_number, raw, cooked, created_at, updated_at, reply_to_post_number, reply_count, quote_count, deleted_at, off_topic_count, like_count, incoming_link_count, bookmark_count, avg_time, score, reads, post_type, sort_order, last_editor_id, hidden, hidden_reason_id, notify_moderators_count, spam_count, illegal_count, inappropriate_count, last_version_at, user_deleted, reply_to_user_id, percent_rank, notify_user_count, like_score, deleted_by_id, edit_reason, word_count, version, cook_method, wiki, baked_at, baked_version, hidden_at, self_edits, reply_quoted, via_email, raw_email, public_version, action_code, image_url, locked_by_id) TO stdout;
2020-07-26 03:34:51.547 UTC [27570] discourse@discourse LOG:  duration: 296.400 ms  statement: COPY public.scheduler_stats (id, name, hostname, pid, duration_ms, live_slots_start, live_slots_finish, started_at, success, error) TO stdout;
2020-07-26 03:35:04.123 UTC [27570] discourse@discourse LOG:  duration: 12549.364 ms  statement: COPY public.stylesheet_cache (id, target, digest, content, created_at, updated_at, theme_id, source_map) TO stdout;
2020-07-26 03:35:04.760 UTC [27570] discourse@discourse LOG:  duration: 588.788 ms  statement: COPY public.unsubscribe_keys (key, user_id, created_at, updated_at, unsubscribe_key_type, topic_id, post_id) TO stdout;c

[Falco]

nginx의 환영 화면은 프로세스의 일부가 아니었습니다. 잘못 기억하고 계십니다.

[Stephen]

@jameshahnii

nginx가 설치되어 있는 이유는 이 포럼이 몇 년 동안 정상적으로 작동해 왔기 때문입니다.

Docker 컨테이너 내부에도 nginx가 설치되어 있으며, 외부에서는 필요하지 않았습니다.

현재 동일한 포트에서 두 개의 nginx 인스턴스가 리스닝하고 있습니다. 이 서버에 Discourse만 실행 중이라면, 컨테이너 외부의 중복된 nginx를 비교적 안전하게 제거할 수 있습니다.

[jameshahnii]

음, 죄송합니다.

[jameshahnii]

그곳이 어디에 있는지 아세요?

[Stephen]

서버가 Ubuntu라고 가정하고, 컨테이너 외부에서 다음을 실행하세요:

sudo apt-get remove nginx nginx-common

[jameshahnii]

아니, 이제 사이트 접속이 안 되네. 백업 복원을 시도해 보고, 잘 되길 바랄 뿐이다.

[Stephen]

사실 좋은 소식입니다. 컨테이너 외부의 nginx 인스턴스가 더 이상 포트를 가로채지 않는다는 뜻이니까요.

nginx를 제거한 후에는 서버를 재시작해야 모든 서비스가 정상적으로 시작됩니다.