オンライン再構築後、tests-passed Productionで502 Bad Gatewayが発生しました

[merefield (Robert)]

Restarting the server, restarting the container, console rebuilding doesn’t help.

Container is up as I can ./launcher enter app

Got a bunch of these in logs, ideas on how to investigate redis failure?

Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) subscribe failed, reconnecting in 1 second. Call stack /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:398:in `rescue in establish_connection'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:379:in `establish_connection'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:115:in `block in connect'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:344:in `with_reconnect'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:114:in `connect'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:409:in `ensure_connected'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:269:in `block in process'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:356:in `logging'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:268:in `process'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:161:in `call'first? 

Not sure if this is a good idea, but tried to manually start it:

/var/www/discourse# redis-server
3770:C 14 Dec 2023 17:20:17.171 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
3770:C 14 Dec 2023 17:20:17.171 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=3770, just started
3770:C 14 Dec 2023 17:20:17.171 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
3770:M 14 Dec 2023 17:20:17.173 * monotonic clock: POSIX clock_gettime
3770:M 14 Dec 2023 17:20:17.173 # Warning: Could not create server TCP listening socket *:6379: bind: Address already in use
3770:M 14 Dec 2023 17:20:17.173 # Failed listening on port 6379 (TCP), aborting.

[supermathie (Michael Brown)]

Is redis actually running inside the container?

Does sv think it is?

root@michael-discourse-docker-app:/# sv status redis
run: redis: (pid 548) 9843135s; run: log: (pid 547) 9843135s

Does anything show in /var/log/redis/current?

Can you sv down redis (if up) and sv up redis?

Any relevant logs show after that?

[merefield (Robert)]

/var/www/discourse/log# sv status redis
run: redis: (pid 565) 1430s; run: log: (pid 560) 1430s

/var/www/discourse/log# sv down redis
/var/www/discourse/log# sv up redis
/var/www/discourse/log# tail /var/log/redis/current
4766:M 14 Dec 2023 17:26:20.195 * monotonic clock: POSIX clock_gettime
4766:M 14 Dec 2023 17:26:20.195 * Running mode=standalone, port=6379.
4766:M 14 Dec 2023 17:26:20.195 # Server initialized
4766:M 14 Dec 2023 17:26:20.195 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
4766:M 14 Dec 2023 17:26:20.196 * Loading RDB produced by version 7.0.7
4766:M 14 Dec 2023 17:26:20.196 * RDB age 3 seconds
4766:M 14 Dec 2023 17:26:20.196 * RDB memory usage when created 24.56 Mb
4766:M 14 Dec 2023 17:26:20.317 * Done loading RDB, keys loaded: 21165, keys expired: 3.
4766:M 14 Dec 2023 17:26:20.317 * DB loaded from disk: 0.121 seconds
4766:M 14 Dec 2023 17:26:20.317 * Ready to accept connections

[merefield (Robert)]

Found the issue! A plugin! (mine!!)

[merefield (Robert)]

Thanks Michael, looks like it was a Zeitwerk naming infraction! (not picked up by Rubocop I might add, and works in Dev).

But those commands are very very useful for future reference!