I’m trying to set up Discourse on a server. I’ve already done it a few times, as I’m testing a script to fix various aspects of the mbox archives from 18 years of mailing list activity before importing them into Discourse. It has worked the previous times.
Coming back to this after pausing the work for a while, I ran ./discourse-setup and got errors from Let’s Encrypt due to having exceeded rate limits (since I did many attempts before). To work around that, I edited containers/app.yml to remove the two Let’s encrypt templates (I don’t need HTTPS for my tests) and ran ./launcher rebuild app.
Unfortunately, I’m now getting “502 Bad gateway – nginx” when accessing the site. The output from ./launcher logs app contains a bunch of these errors:
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/configurator.rb:104:in `block in reload': directory for pid=/var/www/discourse/tmp/pids/unicorn.pid not writable (ArgumentError)
raise ArgumentError, "directory for #{var}=#{path} not writable"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/configurator.rb:100:in `each'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/configurator.rb:100:in `reload'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/configurator.rb:78:in `initialize'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:78:in `new'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:78:in `initialize'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/bin/unicorn:128:in `new'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/bin/unicorn:128:in `<top (required)>'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `load'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `<main>'
config/unicorn_launcher: line 71: kill: (90) - No such process
config/unicorn_launcher: line 15: kill: (90) - No such process
(80) exiting
Given that it worked in the past, this could have several causes:
A recent change in Discourse.
The fact that I’m not using Let’s Encrypt.
The fact that I also changed the email settings (I doubt this matters, though).
Now, the certification seems OK, but the error is still there, suggesting that it is indeed unrelated.
[Mon 14 Aug 2023 10:55:24 AM UTC] Your cert is in: /shared/letsencrypt/lilypond.community_ecc/lilypond.community.cer
[Mon 14 Aug 2023 10:55:24 AM UTC] Your cert key is in: /shared/letsencrypt/lilypond.community_ecc/lilypond.community.key
[Mon 14 Aug 2023 10:55:24 AM UTC] The intermediate CA cert is in: /shared/letsencrypt/lilypond.community_ecc/ca.cer
[Mon 14 Aug 2023 10:55:24 AM UTC] And the full chain certs is there: /shared/letsencrypt/lilypond.community_ecc/fullchain.cer
[Mon 14 Aug 2023 10:55:25 AM UTC] Installing key to: /shared/ssl/lilypond.community_ecc.key
[Mon 14 Aug 2023 10:55:25 AM UTC] Installing full chain to: /shared/ssl/lilypond.community_ecc.cer
[Mon 14 Aug 2023 10:55:25 AM UTC] Run reload cmd: sv reload nginx
warning: nginx: unable to open supervise/ok: file does not exist
[Mon 14 Aug 2023 10:55:25 AM UTC] Reload error for :
C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3
error 2 at 1 depth lookup: unable to get issuer certificate
C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Ersatz Edamame E1
error 2 at 1 depth lookup: unable to get issuer certificate
Started runsvdir, PID is 6029
ok: run: redis: (pid 6040) 0s
ok: run: postgres: (pid 6041) 0s
supervisor pid: 6039 unicorn pid: 6062
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/configurator.rb:104:in `block in reload': directory for pid=/var/www/discourse/tmp/pids/unicorn.pid not writable (ArgumentError)
raise ArgumentError, "directory for #{var}=#{path} not writable"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/configurator.rb:100:in `each'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/configurator.rb:100:in `reload'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/configurator.rb:78:in `initialize'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:78:in `new'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:78:in `initialize'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/bin/unicorn:128:in `new'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/bin/unicorn:128:in `<top (required)>'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `load'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `<main>'
config/unicorn_launcher: line 71: kill: (6062) - No such process
config/unicorn_launcher: line 15: kill: (6062) - No such process
(6039) exiting
Just to check, now that my Let’s Encrypt quota limitation has expired, I tried without applying the patch above. This is what I got:
[Mon 14 Aug 2023 07:57:34 PM UTC] Your cert is in: /shared/letsencrypt/lilypond.community_ecc/lilypond.community.cer
[Mon 14 Aug 2023 07:57:34 PM UTC] Your cert key is in: /shared/letsencrypt/lilypond.community_ecc/lilypond.community.key
[Mon 14 Aug 2023 07:57:34 PM UTC] The intermediate CA cert is in: /shared/letsencrypt/lilypond.community_ecc/ca.cer
[Mon 14 Aug 2023 07:57:34 PM UTC] And the full chain certs is there: /shared/letsencrypt/lilypond.community_ecc/fullchain.cer
[Mon 14 Aug 2023 07:57:34 PM UTC] Installing key to: /shared/ssl/lilypond.community_ecc.key
[Mon 14 Aug 2023 07:57:34 PM UTC] Installing full chain to: /shared/ssl/lilypond.community_ecc.cer
[Mon 14 Aug 2023 07:57:34 PM UTC] Run reload cmd: sv reload nginx
warning: nginx: unable to open supervise/ok: file does not exist
[Mon 14 Aug 2023 07:57:34 PM UTC] Reload error for :
C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3
error 2 at 1 depth lookup: unable to get issuer certificate
C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Ersatz Edamame E1
error 2 at 1 depth lookup: unable to get issuer certificate
Started runsvdir, PID is 6029
ok: run: redis: (pid 6038) 0s
ok: run: postgres: (pid 6041) 0s
supervisor pid: 6042 unicorn pid: 6062
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/configurator.rb:104:in `block in reload': directory for pid=/var/www/discourse/tmp/pids/unicorn.pid not writable (ArgumentError)
raise ArgumentError, "directory for #{var}=#{path} not writable"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/configurator.rb:100:in `each'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/configurator.rb:100:in `reload'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/configurator.rb:78:in `initialize'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:78:in `new'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:78:in `initialize'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/bin/unicorn:128:in `new'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/bin/unicorn:128:in `<top (required)>'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `load'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `<main>'
config/unicorn_launcher: line 71: kill: (6062) - No such process
config/unicorn_launcher: line 15: kill: (6062) - No such process
(6042) exiting
ok: run: redis: (pid 6038) 1s
timeout: down: postgres: 1s, normally up, want up
ok: run: redis: (pid 6038) 9s
timeout: down: postgres: 0s, normally up, want up
ok: run: redis: (pid 6038) 17s
timeout: down: postgres: 1s, normally up, want up
ok: run: redis: (pid 6038) 24s
timeout: down: postgres: 0s, normally up, want up
ok: run: redis: (pid 6038) 32s
timeout: down: postgres: 1s, normally up, want up
ok: run: redis: (pid 6038) 39s
timeout: down: postgres: 0s, normally up, want up
ok: run: redis: (pid 6038) 47s
ok: run: postgres: (pid 6134) 0s
supervisor pid: 6126 unicorn pid: 6136
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/configurator.rb:104:in `block in reload': directory for pid=/var/www/discourse/tmp/pids/unicorn.pid not writable (ArgumentError)
raise ArgumentError, "directory for #{var}=#{path} not writable"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/configurator.rb:100:in `each'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/configurator.rb:100:in `reload'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/configurator.rb:78:in `initialize'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:78:in `new'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:78:in `initialize'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/bin/unicorn:128:in `new'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/bin/unicorn:128:in `<top (required)>'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `load'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `<main>'
config/unicorn_launcher: line 71: kill: (6136) - No such process
config/unicorn_launcher: line 15: kill: (6136) - No such process
(6126) exiting
ok: run: redis: (pid 6038) 54s
timeout: down: postgres: 1s, normally up, want up
ok: run: redis: (pid 6038) 62s
timeout: down: postgres: 0s, normally up, want up
ok: run: redis: (pid 6038) 69s
timeout: down: postgres: 0s, normally up, want up
ok: run: redis: (pid 6038) 77s
timeout: down: postgres: 1s, normally up, want up
ok: run: redis: (pid 6038) 84s
timeout: down: postgres: 0s, normally up, want up
ok: run: redis: (pid 6038) 92s
ok: run: postgres: (pid 6193) 0s
supervisor pid: 6190 unicorn pid: 6195
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/configurator.rb:104:in `block in reload': directory for pid=/var/www/discourse/tmp/pids/unicorn.pid not writable (ArgumentError)
raise ArgumentError, "directory for #{var}=#{path} not writable"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/configurator.rb:100:in `each'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/configurator.rb:100:in `reload'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/configurator.rb:78:in `initialize'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:78:in `new'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:78:in `initialize'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/bin/unicorn:128:in `new'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/bin/unicorn:128:in `<top (required)>'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `load'
from /var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `<main>'
config/unicorn_launcher: line 71: kill: (6195) - No such process
config/unicorn_launcher: line 15: kill: (6195) - No such process
(6190) exiting
ok: run: redis: (pid 6038) 94s
This looks like a folder permission issue. This is kinda weird as that folder in inside the container filesystem, and not mounted from the host.
You can try entering the running container and tweaking the folder permissions manually. If you need help with that, can you please share the output of
docker exec -it app ls -lah /var/www/discourse/tmp/pids
I’m installing from scratch yet again and trying to pay more attention to the log messages from the app build this time.
I see these:
153:C 16 Aug 2023 20:24:11.676 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
153:C 16 Aug 2023 20:24:11.676 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=153, just started
153:C 16 Aug 2023 20:24:11.676 # Configuration loaded
153:M 16 Aug 2023 20:24:11.677 * monotonic clock: POSIX clock_gettime
153:M 16 Aug 2023 20:24:11.677 # Warning: Could not create server TCP listening socket *:6379: bind: Address already in use
153:M 16 Aug 2023 20:24:11.678 # Failed listening on port 6379 (TCP), aborting.
Also this:
I, [2023-08-16T20:24:26.172936 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'yarn install --frozen-lockfile && yarn cache clean'
warning " > @glint/environment-ember-loose@1.0.2" has unmet peer dependency "@glimmer/component@^1.1.2".
warning " > @glint/environment-ember-template-imports@1.0.2" has unmet peer dependency "ember-template-imports@^3.0.0".
warning " > @mixer/parallel-prettier@2.0.3" has unmet peer dependency "prettier@^2.0.0".
warning Resolution field "babel-plugin-ember-template-compilation@2.0.0" is incompatible with requested version "babel-plugin-ember-template-compilation@^2.0.1"
warning Resolution field "unset-value@2.0.1" is incompatible with requested version "unset-value@^1.0.0"
warning " > babel-plugin-debug-macros@0.4.0-pre1" has unmet peer dependency "@babel/core@^7.0.0".
warning "workspace-aggregator-d7aa52aa-3a92-43f5-97ca-2c6c21fe43f0 > discourse > @uppy/aws-s3@3.0.6" has incorrect peer dependency "@uppy/core@^3.1.2".
warning "workspace-aggregator-d7aa52aa-3a92-43f5-97ca-2c6c21fe43f0 > discourse > @uppy/aws-s3-multipart@3.1.3" has incorrect peer dependency "@uppy/core@^3.1.2".
warning "workspace-aggregator-d7aa52aa-3a92-43f5-97ca-2c6c21fe43f0 > discourse > @uppy/xhr-upload@3.1.1" has incorrect peer dependency "@uppy/core@^3.1.2".
warning "workspace-aggregator-d7aa52aa-3a92-43f5-97ca-2c6c21fe43f0 > discourse > @uppy/aws-s3 > @uppy/xhr-upload@3.3.0" has incorrect peer dependency "@uppy/core@^3.2.1".
(I’ll try the manual permission changes when it finishes building.)
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.7/lib/active_record/connection_adapters/postgresql_adapter.rb:87:in `rescue in new_client': connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory (ActiveRecord::ConnectionNotEstablished)
Is the server running locally and accepting connections on that socket?
In the PG log, I see:
2023-08-21 19:24:00.721 UTC [1681] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-08-21 19:24:00.728 UTC [1681] LOG: could not open configuration file "/etc/postgresql/13/main/pg_hba.conf": Permission denied
2023-08-21 19:24:00.728 UTC [1681] FATAL: could not load pg_hba.conf
2023-08-21 19:24:00.741 UTC [1681] LOG: database system is shut down
further:
# ls -l /etc/postgresql/13/main/pg_hba.conf
-rw-r----- 1 root root 4846 Aug 21 19:05 /etc/postgresql/13/main/pg_hba.conf
What user is postgres running under inside the container? With the above permissions, it has to be root or someone in group root
Ok, so I did chmod o+r /etc/postgresql/13/main/pg_hba.conf and now the container is up again.
This is all a bit concerning - why isn’t the recommended installation method not working out of the box? My plugin status currently includes the ones listed above except data explorer which I disabled since it had caused the failure the last time.
Crosslinking to
which reports similar symptoms.
Update: I changed the git command in the cmd section of the app.yml file to use sudo as described in the linked post.
I am declaring this failure to be intermittent. In 3 tries (between each I completely wiped the shared directory) it succeeded once and failed twice. When it fails, manually fixing the three permissions in question and then restarting the container resulted in what appears to be a working system. Better logging and better self tests would be nice in order to detect failing container bootstraps.
Changing the permissions on /var/www/discourse/tmp/pids and /etc/postgre/13/main/pg_hba.conf does not work for me.
I had changed my plugin list before the rebuild but even after restoring the plugin list I get the same ArgumentError. Are we sure it is the plugins list that is causing it?
After starting the container I see files written to the pids directory. My launcher log stops after this
Yes, tried to install discourse by just following the guide and thought i’ve gone mad, since everything “seemed” to work but i only get a “502” error.
I tried changing the permissions within the container without rebuilding. Now i get this:
ok: run: redis: (pid 50) 4677s
ok: run: postgres: (pid 12224) 0s
supervisor pid: 12215 unicorn pid: 12226
config/unicorn_launcher: line 71: kill: (12226) - No such process
config/unicorn_launcher: line 15: kill: (12226) - No such process
(12215) exiting
ok: run: redis: (pid 50) 4686s
ok: run: postgres: (pid 12249) 0s
supervisor pid: 12240 unicorn pid: 12251
config/unicorn_launcher: line 71: kill: (12251) - No such process
config/unicorn_launcher: line 15: kill: (12251) - No such process
(12240) exiting
ok: run: redis: (pid 50) 4695s
timeout: down: postgres: 1s, normally up, want up
ok: run: redis: (pid 50) 4703s
ok: run: postgres: (pid 12279) 0s
supervisor pid: 12275 unicorn pid: 12281
I have a second server, set up exactly like this one, running for over a year. That one just works, stopping the container, doing a git pull, rebuild and start also just worked
So I just got a brand new droplet running Ubuntu 22.04, ran the install guide and it came up just fine. Can’t reproduce this issue in either an old or a new install.
What distro were you running?
Nothing like this in a brand new install:
root@test-install:/var/discourse# cat /var/discourse/shared/standalone/log/rails/unicorn.std*
I, [2023-08-22T17:16:33.594602 #2982] INFO -- : Refreshing Gem list
I, [2023-08-22T17:16:38.624384 #2982] INFO -- : listening on addr=127.0.0.1:3000 fd=10
I, [2023-08-22T17:16:43.003213 #2982] INFO -- : starting 1 supervised sidekiqs
I, [2023-08-22T17:16:47.070059 #2982] INFO -- : master process ready
I, [2023-08-22T17:16:50.490722 #3068] INFO -- : worker=0 ready
I, [2023-08-22T17:16:52.394685 #3077] INFO -- : worker=1 ready
I, [2023-08-22T17:16:53.139229 #3085] INFO -- : worker=2 ready
I, [2023-08-22T17:16:53.518292 #3097] INFO -- : worker=3 ready
Loading Sidekiq in process id 3059
