Forum down, troubleshooting help needed

I went to navigate through the server hosted on Digital Ocean, and when I exited the ssh session the forum now is stuck in a loading screen:

Website URL: https://www.hardwareinterviews.fyi/

I’ve tried the following troubleshooting steps like:

  1. Validating the app.yml
  2. Removing deprecated plugins
  3. Tried launching the website in safe-mode (didn’t work)

It’s been rebuilt using
cd /var/discourse
./launcher rebuild app

Any other troubleshooting steps?

When you do the launcher rebuild, you’ll get a log (a transcript) - paste that here, and someone will probably be able to see what’s wrong.

1 Like

That’s incomplete, there’s more to come …

EDIT: Uploaded the transcript file in Google Drive, removed senstive information

Link is attached here: discourse_log.txt - Google Drive

Alternatively, there was a working/stable version of the website a few days ago. Is there any way to navigate back through version control in git to a previous build?

You may want to upload one without any of your sensitive details in, like passwords, etc. :+1:

2 Likes

Thanks for the flag! What specific lines do you see this occuring in?

The very bottom section, where it lists the details of your site.

Though it looks like the rebuild was successful though, so I’m not sure what the issue is.

In the case of a failed rebuild, you can I think

./launcher start app

to restart the previous state of play. Might be worth a try but I think it might not apply in your case.

1 Like

@JammyDodger Is it reasonable to expect the website to relaunch within ~30 minutes of rebuilding? Still not seeing anything unfortunately :confused:

Going to modify the app.yml to remove all plugins and try again.

Is it possible that a Letsencrypt cert renewal is waiting? Perhaps try

ps auxfc
1 Like

EDIT: Turns out it was an A Record issue on GoDaddy!

Realized I should have opened a separate browser, this is what the screen displays:

Transcript log without cloning any plug-ins:

no luck :confused:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         2  0.0  0.0      0     0 ?        S     2020   0:10 kthreadd
root         4  0.0  0.0      0     0 ?        I<    2020   0:00  \_ kworker/0:0H
root         6  0.0  0.0      0     0 ?        I<    2020   0:00  \_ mm_percpu_wq
root         7  0.0  0.0      0     0 ?        S     2020  13:02  \_ ksoftirqd/0
root         8  0.0  0.0      0     0 ?        I     2020 893:57  \_ rcu_sched
root         9  0.0  0.0      0     0 ?        I     2020   0:00  \_ rcu_bh
root        10  0.0  0.0      0     0 ?        S     2020   0:02  \_ migration/0
root        11  0.0  0.0      0     0 ?        S     2020   3:27  \_ watchdog/0
root        12  0.0  0.0      0     0 ?        S     2020   0:00  \_ cpuhp/0
root        13  0.0  0.0      0     0 ?        S     2020   0:00  \_ cpuhp/1
root        14  0.0  0.0      0     0 ?        S     2020   3:41  \_ watchdog/1
root        15  0.0  0.0      0     0 ?        S     2020   0:02  \_ migration/1
root        16  0.0  0.0      0     0 ?        S     2020  17:08  \_ ksoftirqd/1
root        18  0.0  0.0      0     0 ?        I<    2020   0:00  \_ kworker/1:0H
root        19  0.0  0.0      0     0 ?        S     2020   0:00  \_ kdevtmpfs
root        20  0.0  0.0      0     0 ?        I<    2020   0:00  \_ netns
root        21  0.0  0.0      0     0 ?        S     2020   0:00  \_ rcu_tasks_kthre
root        22  0.0  0.0      0     0 ?        S     2020   0:00  \_ kauditd
root        24  0.0  0.0      0     0 ?        S     2020   1:16  \_ khungtaskd
root        25  0.0  0.0      0     0 ?        S     2020   0:00  \_ oom_reaper
root        26  0.0  0.0      0     0 ?        I<    2020   0:00  \_ writeback
root        27  0.0  0.0      0     0 ?        S     2020   0:01  \_ kcompactd0
root        28  0.0  0.0      0     0 ?        SN    2020   0:00  \_ ksmd
root        29  0.0  0.0      0     0 ?        SN    2020   7:55  \_ khugepaged
root        30  0.0  0.0      0     0 ?        I<    2020   0:00  \_ crypto
root        31  0.0  0.0      0     0 ?        I<    2020   0:00  \_ kintegrityd
root        32  0.0  0.0      0     0 ?        I<    2020   0:00  \_ kblockd
root        33  0.0  0.0      0     0 ?        I<    2020   0:00  \_ ata_sff
root        34  0.0  0.0      0     0 ?        I<    2020   0:00  \_ md
root        35  0.0  0.0      0     0 ?        I<    2020   0:00  \_ edac-poller
root        36  0.0  0.0      0     0 ?        I<    2020   0:00  \_ devfreq_wq
root        37  0.0  0.0      0     0 ?        I<    2020   0:00  \_ watchdogd
root        41  0.0  0.0      0     0 ?        S     2020  61:17  \_ kswapd0
root        42  0.0  0.0      0     0 ?        I<    2020   0:00  \_ kworker/u5:0
root        43  0.0  0.0      0     0 ?        S     2020   0:00  \_ ecryptfs-kthrea
root        85  0.0  0.0      0     0 ?        I<    2020   0:00  \_ kthrotld
root        86  0.0  0.0      0     0 ?        I<    2020   0:00  \_ acpi_thermal_pm
root        87  0.0  0.0      0     0 ?        S     2020   0:00  \_ scsi_eh_0
root        88  0.0  0.0      0     0 ?        I<    2020   0:00  \_ scsi_tmf_0
root        89  0.0  0.0      0     0 ?        S     2020   0:00  \_ scsi_eh_1
root        90  0.0  0.0      0     0 ?        I<    2020   0:00  \_ scsi_tmf_1
root        96  0.0  0.0      0     0 ?        I<    2020   0:00  \_ ipv6_addrconf
root       105  0.0  0.0      0     0 ?        I<    2020   0:00  \_ kstrp
root       122  0.0  0.0      0     0 ?        I<    2020   0:00  \_ charger_manager
root       172  0.0  0.0      0     0 ?        S     2020   0:00  \_ scsi_eh_2
root       173  0.0  0.0      0     0 ?        I<    2020   0:00  \_ scsi_tmf_2
root       215  0.0  0.0      0     0 ?        I<    2020   0:00  \_ ttm_swap
root       292  0.0  0.0      0     0 ?        I<    2020   0:00  \_ raid5wq
root       345  0.0  0.0      0     0 ?        I<    2020   9:26  \_ kworker/0:1H
root       346  0.0  0.0      0     0 ?        I<    2020  11:17  \_ kworker/1:1H
root       347  0.0  0.0      0     0 ?        S     2020  33:02  \_ jbd2/vda1-8
root       348  0.0  0.0      0     0 ?        I<    2020   0:00  \_ ext4-rsv-conver
root       432  0.0  0.0      0     0 ?        I<    2020   0:00  \_ iscsi_eh
root       446  0.0  0.0      0     0 ?        I<    2020   0:00  \_ ib-comp-wq
root       450  0.0  0.0      0     0 ?        I<    2020   0:00  \_ ib-comp-unb-wq
root       452  0.0  0.0      0     0 ?        I<    2020   0:00  \_ ib_mcast
root       454  0.0  0.0      0     0 ?        I<    2020   0:00  \_ ib_nl_sa_wq
root       460  0.0  0.0      0     0 ?        I<    2020   0:00  \_ rdma_cm
root      6534  0.0  0.0      0     0 ?        I<    2020   0:00  \_ xfsalloc
root      6535  0.0  0.0      0     0 ?        I<    2020   0:00  \_ xfs_mru_cache
root      6541  0.0  0.0      0     0 ?        S     2020   0:00  \_ jfsIO
root      6544  0.0  0.0      0     0 ?        S     2020   0:00  \_ jfsCommit
root      6545  0.0  0.0      0     0 ?        S     2020   0:00  \_ jfsCommit
root      6546  0.0  0.0      0     0 ?        S     2020   0:00  \_ jfsSync
root     16598  0.0  0.0      0     0 ?        I    19:39   0:02  \_ kworker/1:0
root     16613  0.0  0.0      0     0 ?        I    19:39   0:01  \_ kworker/0:0
root     20009  0.0  0.0      0     0 ?        I    19:56   0:01  \_ kworker/u4:0
root     23639  0.0  0.0      0     0 ?        I    20:13   0:00  \_ kworker/u4:1
root     25994  0.0  0.0      0     0 ?        I    20:20   0:00  \_ kworker/0:2
root     26019  0.0  0.0      0     0 ?        I    20:20   0:00  \_ kworker/1:2
root     28657  0.0  0.0      0     0 ?        I    20:25   0:00  \_ kworker/0:1
root         1  0.0  0.2 225420  4180 ?        Ss    2020  30:06 systemd
root       441  0.0  0.0 105904    32 ?        Ss    2020   0:00 lvmetad
root      1023  0.0  0.0 110484   788 ?        Ssl   2020  38:21 irqbalance
root      1028  0.0  0.0  70608   204 ?        Ss    2020   1:36 systemd-logind
daemon    1045  0.0  0.0  28332    28 ?        Ss    2020   0:00 atd
root      1053  0.0  0.0 832588     0 ?        Ssl   2020   7:12 lxcfs
message+  1057  0.0  0.0  50396  1148 ?        Ss    2020  58:31 dbus-daemon
root      1081  0.1  0.5 1055416 10836 ?       Ssl   2020 1884:04 containerd
root     27623  0.0  0.2 109108  5580 ?        Sl   20:22   0:00  \_ containerd-shim
root     27652  0.0  0.1   6828  2284 pts/0    Ss+  20:22   0:00      \_ boot
root     28232  0.0  0.0   2396   232 pts/0    S+   20:22   0:00          \_ runsvdir
root     28233  0.0  0.0   2244   364 ?        Ss   20:22   0:00              \_ runsv
root     28243  0.0  0.0   6676  1588 ?        S    20:22   0:00              |   \_ cron
root     28234  0.0  0.0   2244   244 ?        Ss   20:22   0:00              \_ runsv
root     28241  0.0  0.1 151120  3072 ?        Sl   20:22   0:00              |   \_ rsyslogd
root     28235  0.0  0.0   2244   356 ?        Ss   20:22   0:00              \_ runsv
1000     28239  0.1  0.1  15312  3272 ?        S    20:22   0:00              |   \_ unicorn_launche
1000     28275  2.6 13.4 497752 274956 ?       Sl   20:22   0:11              |       \_ ruby
1000     28367  1.4 18.2 943496 373008 ?       SNl  20:22   0:06              |       |   \_ ruby
1000     28376  0.8 16.1 837228 329136 ?       Sl   20:22   0:03              |       |   \_ ruby
1000     28384  0.8 16.0 818540 326580 ?       Sl   20:22   0:03              |       |   \_ ruby
1000     28392  0.9 15.9 818540 325748 ?       Sl   20:22   0:03              |       |   \_ ruby
1000     28400  0.9 16.1 837228 329516 ?       Sl   20:22   0:03              |       |   \_ ruby
1000     28889  0.0  0.1  13816  2212 ?        S    20:29   0:00              |       \_ sleep
root     28236  0.0  0.0   2244   252 ?        Ss   20:22   0:00              \_ runsv
root     28246  0.0  0.3  53336  6660 ?        S    20:22   0:00              |   \_ nginx
www-data 28260  0.0  0.3  55136  7108 ?        S    20:22   0:00              |       \_ nginx
www-data 28262  0.0  0.1  55136  4016 ?        S    20:22   0:00              |       \_ nginx
www-data 28263  0.0  0.1  53564  2456 ?        S    20:22   0:00              |       \_ nginx
root     28237  0.0  0.0   2244   336 ?        Ss   20:22   0:00              \_ runsv
root     28244  0.0  0.0   2392   344 ?        S    20:22   0:00              |   \_ svlogd
systemd+ 28245  0.0  1.6 352056 33996 ?        S    20:22   0:00              |   \_ postmaster
systemd+ 28269  0.0  0.4 352156  9716 ?        Ss   20:22   0:00              |       \_ postmaster
systemd+ 28270  0.0  0.2 352056  5568 ?        Ss   20:22   0:00              |       \_ postmaster
systemd+ 28271  0.0  0.6 352056 13852 ?        Ss   20:22   0:00              |       \_ postmaster
systemd+ 28272  0.0  0.4 352736  8204 ?        Ss   20:22   0:00              |       \_ postmaster
systemd+ 28273  0.0  0.2  68300  5684 ?        Ss   20:22   0:00              |       \_ postmaster
systemd+ 28274  0.0  0.3 352612  6584 ?        Ss   20:22   0:00              |       \_ postmaster
systemd+ 28298  0.3  1.7 364200 36280 ?        Ss   20:22   0:01              |       \_ postmaster
systemd+ 28408  0.0  1.6 358656 32812 ?        Ss   20:22   0:00              |       \_ postmaster
systemd+ 28764  0.0  1.2 354496 24932 ?        Ss   20:27   0:00              |       \_ postmaster
root     28238  0.0  0.0   2244   376 ?        Ss   20:22   0:00              \_ runsv
root     28240  0.0  0.0   2392   320 ?        S    20:22   0:00                  \_ svlogd
message+ 28242  0.2  0.7  65448 14408 ?        Sl   20:22   0:01                  \_ redis-server
root      1088  0.0  7.1 1343060 145676 ?      Ssl   2020 204:13 dockerd
root     27605  0.0  0.1 479272  3856 ?        Sl   20:22   0:00  \_ docker-proxy
root     27617  0.0  0.1 479272  4056 ?        Sl   20:22   0:00  \_ docker-proxy
root      1091  0.0  0.0  16412     8 ttyS0    Ss+   2020   0:00 agetty
root      1110  0.0  0.0 187688     0 ?        Ssl   2020   0:00 unattended-upgr
root      1996  0.0  0.1  76916  2304 ?        Ss    2020   0:05 systemd
root      1998  0.0  0.0 191496     0 ?        S     2020   0:00  \_ (sd-pam)
root      6664  0.0  0.0  78632    24 tty1     Ss    2020   0:00 login
root      6796  0.0  0.0  23324    16 tty1     S+    2020   0:00  \_ bash
uuidd    22263  0.0  0.0  28596     0 ?        Ss    2020   0:00 uuidd
root     13721  0.0  0.0 287988    92 ?        Ssl   2020  58:28 accounts-daemon
root     18693  0.0  0.0  72304   512 ?        Ss    2021  28:57 sshd
root     12998  0.0  0.0 105692   408 ?        Ss   18:08   0:01  \_ sshd
root     13122  0.0  0.1  23212  2468 pts/0    Ss   18:08   0:00      \_ bash
root     21472  0.0  0.0  24148  1132 pts/0    T    18:27   0:00          \_ nano
root     28890  0.0  0.1  40256  3528 pts/0    R+   20:29   0:00          \_ ps
root     27263  0.0  0.0 288884   516 ?        Ssl   2022  13:12 polkitd
root      8885  0.0  0.3 171516  7248 ?        Ssl   2022   0:01 networkd-dispat
root      1326  0.0  0.0  31756   556 ?        Ss    2022   1:01 cron
syslog   31315  0.0  0.0 263048   224 ?        Ssl   2022  11:37 rsyslogd
root      8247  0.0  0.0  45096  1348 ?        Ss    2022   1:15 systemd-udevd
systemd+ 16957  0.0  0.0  71856  1596 ?        Ss    2022   0:32 systemd-network
systemd+ 16979  0.0  0.0  70624  1940 ?        Ss    2022   0:32 systemd-resolve
systemd+ 17019  0.0  0.0 141784   496 ?        Ssl   2022   0:36 systemd-timesyn
root     17048  0.0  1.1 306728 23792 ?        S<s   2022  42:28 systemd-journal
do-agent 15930  0.0  0.4 1310900 8624 ?        Ssl  May13  15:56 do-agent
tail /var/discourse/shared/standalone/log/var-log/nginx/error.log
1 Like

Not sure what’s happening… if there’s nothing in the nginx error log, perhaps see over here:

Tried a couple of the troubleshooting steps like:

  • Makes sense why the web_only command failed, I only have one container

Just to note - you solved your problem, it was DNS, you edited an earlier reply and marked it as a solution.

4 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.