For the first time in years after a rebuild my instance is not coming back up. My last successful rebuild was yesterday.
Is there maybe a fresh commit that may be the reason for this?
What are the symptoms?
What do you see if you
tail -f /var/discourse/shared/standalone/log/rails/production.log
Done in 89.60s.
Downloading MaxMindDB...
Compressing Javascript and Generating Source Maps
I, [2023-03-08T17:11:55.996529 #1] INFO -- : File > /usr/local/bin/discourse chmod: +x chown:
I, [2023-03-08T17:11:55.999205 #1] INFO -- : File > /usr/local/bin/rails chmod: +x chown:
I, [2023-03-08T17:11:56.001899 #1] INFO -- : File > /usr/local/bin/rake chmod: +x chown:
I, [2023-03-08T17:11:56.004616 #1] INFO -- : File > /usr/local/bin/rbtrace chmod: +x chown:
I, [2023-03-08T17:11:56.007306 #1] INFO -- : File > /usr/local/bin/stackprof chmod: +x chown:
I, [2023-03-08T17:11:56.010710 #1] INFO -- : File > /etc/update-motd.d/10-web chmod: +x chown:
I, [2023-03-08T17:11:56.012746 #1] INFO -- : File > /etc/logrotate.d/rails chmod: chown:
I, [2023-03-08T17:11:56.014343 #1] INFO -- : File > /etc/logrotate.d/nginx chmod: chown:
I, [2023-03-08T17:11:56.017963 #1] INFO -- : File > /etc/runit/1.d/00-ensure-links chmod: +x chown:
I, [2023-03-08T17:11:56.020609 #1] INFO -- : File > /etc/runit/1.d/01-cleanup-web-pids chmod: +x chown:
I, [2023-03-08T17:11:56.023663 #1] INFO -- : File > /root/.bash_profile chmod: 644 chown:
I, [2023-03-08T17:11:56.026021 #1] INFO -- : File > /usr/local/etc/ImageMagick-7/policy.xml chmod: chown:
I, [2023-03-08T17:11:56.026795 #1] INFO -- : Replacing (?-mix:server.+{) with limit_req_zone $binary_remote_addr zone=flood:10m rate=$reqs_per_secondr/s;
limit_req_zone $binary_remote_addr zone=bot:10m rate=$reqs_per_minuter/m;
limit_req_status 429;
limit_conn_zone $binary_remote_addr zone=connperip:10m;
limit_conn_status 429;
server {
in /etc/nginx/conf.d/discourse.conf
I, [2023-03-08T17:11:56.026984 #1] INFO -- : Replacing (?-mix:location @discourse {) with location @discourse {
limit_conn connperip $conn_per_ip;
limit_req zone=flood burst=$burst_per_second nodelay;
limit_req zone=bot burst=$burst_per_minute nodelay; in /etc/nginx/conf.d/discourse.conf
I, [2023-03-08T17:11:56.029658 #1] INFO -- : File > /etc/runit/1.d/remove-old-socket chmod: +x chown:
I, [2023-03-08T17:11:56.032272 #1] INFO -- : File > /etc/runit/3.d/remove-old-socket chmod: +x chown:
I, [2023-03-08T17:11:56.032398 #1] INFO -- : Replacing (?-mix:listen 80;) with listen unix:/shared/nginx.http.sock;
set_real_ip_from unix:;
in /etc/nginx/conf.d/discourse.conf
I, [2023-03-08T17:11:56.032577 #1] INFO -- : Replacing (?-mix:listen 443 ssl http2;) with listen unix:/shared/nginx.https.sock ssl http2;
set_real_ip_from unix:; in /etc/nginx/conf.d/discourse.conf
I, [2023-03-08T17:11:56.035350 #1] INFO -- : File > /tmp/add-cloudflare-ips chmod: +x chown:
I, [2023-03-08T17:11:56.035435 #1] INFO -- : > /tmp/add-cloudflare-ips
I, [2023-03-08T17:11:56.359453 #1] INFO -- : CloudFlare IPs:
set_real_ip_from 173.245.48.0/20; set_real_ip_from 103.21.244.0/22; set_real_ip_from 103.22.200.0/22; set_real_ip_from 103.31.4.0/22; set_real_ip_from 141.101.64.0/18; set_real_ip_from 108.162.192.0/18; set_real_ip_from 190.93.240.0/20; set_real_ip_from 188.114.96.0/20; set_real_ip_from 197.234.240.0/22; set_real_ip_from 198.41.128.0/17; set_real_ip_from 162.158.0.0/15; set_real_ip_from 104.16.0.0/13; set_real_ip_from 104.24.0.0/14; set_real_ip_from 172.64.0.0/13; set_real_ip_from 131.0.72.0/22; set_real_ip_from 2400:cb00::/32; set_real_ip_from 2606:4700::/32; set_real_ip_from 2803:f800::/32; set_real_ip_from 2405:b500::/32; set_real_ip_from 2405:8100::/32; set_real_ip_from 2a06:98c0::/29; set_real_ip_from 2c0f:f248::/32;
I, [2023-03-08T17:11:56.359655 #1] INFO -- : > rm /tmp/add-cloudflare-ips
I, [2023-03-08T17:11:56.361599 #1] INFO -- :
I, [2023-03-08T17:11:56.361818 #1] INFO -- : > echo "Beginning of custom commands"
I, [2023-03-08T17:11:56.363535 #1] INFO -- : Beginning of custom commands
I, [2023-03-08T17:11:56.367829 #1] INFO -- : File > /etc/service/monerochan_merchant_rpc/run chmod: +x chown:
I, [2023-03-08T17:11:56.368034 #1] INFO -- : > echo "End of custom commands"
I, [2023-03-08T17:11:56.369958 #1] INFO -- : End of custom commands
I, [2023-03-08T17:11:56.370117 #1] INFO -- : Terminating async processes
I, [2023-03-08T17:11:56.370225 #1] INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 42
I, [2023-03-08T17:11:56.370261 #1] INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 103
2023-03-08 17:11:56.370 UTC [42] LOG: received fast shutdown request
103:signal-handler (1678295516) Received SIGTERM scheduling shutdown...
2023-03-08 17:11:56.372 UTC [42] LOG: aborting any active transactions
2023-03-08 17:11:56.374 UTC [42] LOG: background worker "logical replication launcher" (PID 51) exited with exit code 1
2023-03-08 17:11:56.375 UTC [46] LOG: shutting down
2023-03-08 17:11:56.392 UTC [42] LOG: database system is shut down
103:M 08 Mar 2023 17:11:56.469 # User requested shutdown...
103:M 08 Mar 2023 17:11:56.469 * Saving the final RDB snapshot before exiting.
103:M 08 Mar 2023 17:11:56.570 * DB saved on disk
103:M 08 Mar 2023 17:11:56.570 # Redis is now ready to exit, bye bye...
sha256:422bd26e098f3af0623647ebce02770ac1608bfac07260aeb5469ab975696363
a0b91a9cc45e8666352e172143854705faa97b38208fcfe0650ea929989b8570
Removing old container
+ /usr/bin/docker rm app
app
âžś discourse git:(main) âś— tail -f /var/discourse/shared/standalone/log/rails/production.log
Bye!
Deprecation notice: (siwe) full_screen_login is now forced. The full_screen_login parameter can be removed from the auth_provider. (removal in Discourse 2.9.0)
At /var/www/discourse/lib/plugin/instance.rb:763:in `public_send`
Migrating to MakeChatMentionNotificationIdNullable (20230227172543)
Migrating to DropBadgeGrantedTitleColumn (20230228105851)
Migrating to AddExternalToSidebarUrls (20230303015952)
Theme setting type has changed but cannot be converted.
#<ThemeSettingsManager::Upload:0x00007f60ff714e38 @name=:background_image, @default="", @theme=#<Theme id: 31, name: "Search Banner", user_id: 2, created_at: "2021-08-03 16:38:09.042735000 +0000", updated_at: "2021-08-03 17:01:41.329058000 +0000", compiler_version: 0, user_selectable: false, hidden: false, color_scheme_id: nil, remote_theme_id: 29, component: true, enabled: true, auto_update: true>, @opts={:description=>"background image for the banner", :textarea=>false, :json_schema=>nil, :refresh=>false}, @types={:integer=>0, :float=>1, :string=>2, :bool=>3, :list=>4, :enum=>5, :upload=>6}>
Bye!
Deprecation notice: (siwe) full_screen_login is now forced. The full_screen_login parameter can be removed from the auth_provider. (removal in Discourse 2.9.0)
At /var/www/discourse/lib/plugin/instance.rb:763:in `public_send`
Migrating to MakeChatMentionNotificationIdNullable (20230227172543)
Migrating to DropBadgeGrantedTitleColumn (20230228105851)
Migrating to AddExternalToSidebarUrls (20230303015952)
Theme setting type has changed but cannot be converted.
#<ThemeSettingsManager::Upload:0x00007f60ff714e38 @name=:background_image, @default="", @theme=#<Theme id: 31, name: "Search Banner", user_id: 2, created_at: "2021-08-03 16:38:09.042735000 +0000", updated_at: "2021-08-03 17:01:41.329058000 +0000", compiler_version: 0, user_selectable: false, hidden: false, color_scheme_id: nil, remote_theme_id: 29, component: true, enabled: true, auto_update: true>, @opts={:description=>"background image for the banner", :textarea=>false, :json_schema=>nil, :refresh=>false}, @types={:integer=>0, :float=>1, :string=>2, :bool=>3, :list=>4, :enum=>5, :upload=>6}>
I’m not seeing anything out of the ordinary. Perhaps the nginx lines are troubling? (my nginx setup is from official discourse maintenance page tutorial).
What happens if you visit your site? Can you share the URL?
1 Like
Rather not in public, but I’ll shoot you a DM.
Edit: rerouted nginx to ignore 502 errors etc. Just a bad gateway error.
1 Like
You’ll need to look at the log file when something tries to load the site and see what the error is.
If cloudflare (or whatever) is stopping a browser from accessing the site then maybe you can look through the logs to look for a 500 error to see what the issue is.
You might be able to curl localhost from inside the container.
1 Like
That is some good feedback. Will try that and get back.
EDIT: Hmm, I don’t think anything happens in the logs, but I’m seeing the following.
âžś ~ tail -f /var/discourse/shared/standalone/log/rails/production.log
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/subscribe.rb:14:in `subscribe'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:288:in `_subscription'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/commands/pubsub.rb:20:in `block in subscribe'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:265:in `block in synchronize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:265:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:265:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/commands/pubsub.rb:19:in `subscribe'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.2/lib/message_bus/backends/redis.rb:302:in `global_subscribe'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.2/lib/message_bus.rb:768:in `global_subscribe_thread'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.2/lib/message_bus.rb:739:in `block in new_subscriber_thread'
In my nginx I’m seeing:
023/03/08 17:58:38 [crit] 115962#115962: *448 connect() to unix:/var/discourse/shared/standalone/nginx.http.sock failed (2: No such file or directory) while connecting to upstream, client: IP.XXX server: [domain.com](http://domain.com), request: "GET /service-worker.js HTTP/2.0", upstream: "http://unix:/var/discourse/shared/standalone/nginx.http.sock:/service-worker.js", host: "domain.com", referrer: "https://domain.com/service-worker.js"
2023/03/08 17:59:32 [notice] 318573#318573: signal process started
Does this help? Or should I completely revert the nginx setup for this?
2 Likes
If you share the full backtrace (a few more line of that file) we can help you.
1 Like
Completed 200 OK in 25ms (Views: 0.1ms | ActiveRecord: 0.0ms | Allocations: 17734)
Started GET "/categories_and_latest" for [ip removed] at 2023-03-08 16:20:41 +0000
Processing by CategoriesController#categories_and_latest as JSON
Rendered text template (Duration: 0.0ms | Allocations: 1)
Completed 200 OK in 112ms (Views: 0.2ms | ActiveRecord: 0.0ms | Allocations: 63242)
Started GET "/" for [ip removed] at 2023-03-08 16:21:00 +0000
Processing by CategoriesController#index as HTML
Rendered categories/index.html.erb within layouts/crawler (Duration: 1.4ms | Allocations: 1135)
Rendered layout layouts/crawler.html.erb (Duration: 6.7ms | Allocations: 3536)
Completed 200 OK in 75ms (Views: 7.7ms | ActiveRecord: 0.0ms | Allocations: 41712)
Started GET "/notifications?limit=30&recent=true&bump_last_seen_reviewable=true" for [ip removed] at 2023-03-08 16:21:35 +0000
Processing by NotificationsController#index as JSON
Parameters: {"limit"=>"30", "recent"=>"true", "bump_last_seen_reviewable"=>"true"}
Completed 200 OK in 60ms (Views: 0.1ms | ActiveRecord: 0.0ms | Allocations: 30822)
Started GET "/" for [ip removed] at 2023-03-08 16:22:00 +0000
Processing by CategoriesController#index as HTML
Rendered categories/index.html.erb within layouts/crawler (Duration: 1.1ms | Allocations: 1135)
Rendered layout layouts/crawler.html.erb (Duration: 5.4ms | Allocations: 3536)
Completed 200 OK in 86ms (Views: 6.1ms | ActiveRecord: 0.0ms | Allocations: 41842)
Shutting down
Terminating quiet threads
Scheduler exiting...
Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) subscribe failed, reconnecting in 1 second. Call stack /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:398:in `rescue in establish_connection'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:379:in `establish_connection'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:115:in `block in connect'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:344:in `with_reconnect'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:114:in `connect'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:409:in `ensure_connected'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:269:in `block in process'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:356:in `logging'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:268:in `process'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:175:in `block in call_loop'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:331:in `with_socket_timeout'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:174:in `call_loop'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/subscribe.rb:44:in `subscription'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/subscribe.rb:14:in `subscribe'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:288:in `_subscription'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/commands/pubsub.rb:20:in `block in subscribe'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:265:in `block in synchronize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:265:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:265:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/commands/pubsub.rb:19:in `subscribe'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.2/lib/message_bus/backends/redis.rb:302:in `global_subscribe'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.2/lib/message_bus.rb:768:in `global_subscribe_thread'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.2/lib/message_bus.rb:739:in `block in new_subscriber_thread'
Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
Job exception: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
Job exception: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
Job exception: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
Job exception: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
Job exception: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
Job exception: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
Failed to process job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) ["/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:398:in `rescue in establish_connection'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:379:in `establish_connection'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:115:in `block in connect'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:344:in `with_reconnect'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:114:in `connect'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:409:in `ensure_connected'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:269:in `block in process'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:356:in `logging'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:268:in `process'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:161:in `call'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.0.0/lib/mini_profiler/profiling_methods.rb:85:in `block in profile_method'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:270:in `block in send_command'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:269:in `synchronize'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:269:in `send_command'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/commands/scripting.rb:110:in `_eval'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/commands/scripting.rb:97:in `evalsha'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.2/lib/message_bus/backends/redis.rb:463:in `cached_eval'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.2/lib/message_bus/backends/redis.rb:150:in `publish'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.2/lib/message_bus.rb:391:in `publish'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.2/lib/message_bus.rb:751:in `block in new_subscriber_thread'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.2/lib/message_bus/timer_thread.rb:117:in `do_work'", "/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.2/lib/message_bus/timer_thread.rb:95:in `block (2 levels) in queue'"]
Pausing to allow jobs to finish...
heartbeat: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) subscribe failed, reconnecting in 1 second. Call stack /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:398:in `rescue in establish_connection'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:379:in `establish_connection'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:115:in `block in connect'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:344:in `with_reconnect'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:114:in `connect'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:409:in `ensure_connected'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:269:in `block in process'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:356:in `logging'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:268:in `process'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:161:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.0.0/lib/mini_profiler/profiling_methods.rb:85:in `block in profile_method'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:270:in `block in send_command'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:269:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:269:in `send_command'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/commands/strings.rb:191:in `get'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.2/lib/message_bus/backends/redis.rb:401:in `process_global_backlog'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.2/lib/message_bus/backends/redis.rb:286:in `block in global_subscribe'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.2/lib/message_bus/backends/redis.rb:299:in `global_subscribe'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.2/lib/message_bus.rb:768:in `global_subscribe_thread'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.2/lib/message_bus.rb:739:in `block in new_subscriber_thread'
Job exception: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
Bye!
Deprecation notice: (siwe) full_screen_login is now forced. The full_screen_login parameter can be removed from the auth_provider. (removal in Discourse 2.9.0)
At /var/www/discourse/lib/plugin/instance.rb:763:in `public_send`
Migrating to MakeChatMentionNotificationIdNullable (20230227172543)
Migrating to DropBadgeGrantedTitleColumn (20230228105851)
Migrating to AddExternalToSidebarUrls (20230303015952)
Theme setting type has changed but cannot be converted.
#<ThemeSettingsManager::Upload:0x00007f60ff714e38 @name=:background_image, @default="", @theme=#<Theme id: 31, name: "Search Banner", user_id: 2, created_at: "2021-08-03 16:38:09.042735000 +0000", updated_at: "2021-08-03 17:01:41.329058000 +0000", compiler_version: 0, user_selectable: false, hidden: false, color_scheme_id: nil, remote_theme_id: 29, component: true, enabled: true, auto_update: true>, @opts={:description=>"background image for the banner", :textarea=>false, :json_schema=>nil, :refresh=>false}, @types={:integer=>0, :float=>1, :string=>2, :bool=>3, :list=>4, :enum=>5, :upload=>6}>
Deprecation notice: (siwe) full_screen_login is now forced. The full_screen_login parameter can be removed from the auth_provider. (removal in Discourse 2.9.0)
At /var/www/discourse/lib/plugin/instance.rb:763:in `public_send`
Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) subscribe failed, reconnecting in 1 second. Call stack /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:398:in `rescue in establish_connection'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:379:in `establish_connection'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:115:in `block in connect'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:344:in `with_reconnect'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:114:in `connect'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:409:in `ensure_connected'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:269:in `block in process'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:356:in `logging'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:268:in `process'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:175:in `block in call_loop'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:331:in `with_socket_timeout'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/client.rb:174:in `call_loop'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/subscribe.rb:44:in `subscription'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/subscribe.rb:14:in `subscribe'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:288:in `_subscription'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/commands/pubsub.rb:20:in `block in subscribe'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:265:in `block in synchronize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:265:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis.rb:265:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/redis-4.8.1/lib/redis/commands/pubsub.rb:19:in `subscribe'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.2/lib/message_bus/backends/redis.rb:302:in `global_subscribe'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.2/lib/message_bus.rb:768:in `global_subscribe_thread'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/message_bus-4.3.2/lib/message_bus.rb:739:in `block in new_subscriber_thread'
This is from the moment before the “fatal” rebuild, up until after a few tries more with removed plugins etc.
I deleted the following plugins, and now my instance is booting up again.
So it’s probably related to recent commits of one of those.
1 Like
The error doesn’t get into the log because it happens rails cranks up:
root@test1-web-only:/var/www/discourse# rails c
bundler: failed to load command: pry (/var/www/discourse/vendor/bundle/ruby/3.2.0/bin/pry)
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/zeitwerk-2.6.7/lib/zeitwerk/loader/helpers.rb:135:in `const_get': uninitialized constant PluginInstance (NameError)
parent.const_get(cname, false)
^^^^^^^^^^
Did you mean? PluginStore
I have replicated the error on another install. I’m trying to figure out which plugin it is right now.
1 Like
Can you please share the plugin list and the each plugin commit?
I have figured out it’s a conflict of Automation and another plugin. It will rebuild with only automation, but not with other plugins enabled. I’m getting close to finding the other culprit…
2 Likes
Actually we had a bug in both automation and assign, and just fixed both. So rebuild now will fix it.
cc @pfaffman
4 Likes
Someone else just told me that
Here’s the list of plugins (plus one more that’s private)
- git clone https://github.com/discourse/docker_manager.git
#- git clone https://github.com/discourse/discourse-docs.git
#- git clone https://github.com/discourse/discourse-solved.git
#- git clone https://github.com/discourse/discourse-voting.git
#- git clone https://github.com/discourse/discourse-reactions.git
#- git clone https://github.com/discourse/discourse-canned-replies.git
#- git clone https://github.com/jomaxro/discourse-plugin-site-setting-override.git
#- git clone https://github.com/discourse/discourse-automation.git
I was able to get the
Aha. So apparently my test that was supposed to confirm the broken plugin confirms the fix.
1 Like
Same here haha, how’s that for timing! Thanks Falco.
1 Like
I will try to rebuild now and restore the removed plugins. Just in case, this is the rest of the plugins:
EDIT: Working with the 3 removed plugins back installed. Thanks team!
| discourse 12436d05 | Up to date |
|---|---|
| docker_manager e90c8f55 | Up to date |
| discourse-adplugin bfd4438b | Up to date |
| discourse-calendar adca3f65 | Up to date |
| discourse-category-lockdown 2cf5f064 | Up to date |
| discourse-chat-integration 75cf4136 | Up to date |
| discourse-chatbot eb9c50ae | Up to date |
| discourse-data-explorer 389b8e15 | Up to date |
| discourse-docs 63bb4629 | Up to date |
| discourse-encrypt 0f3c612b | Up to date |
| discourse-formatting-toolbar d99f3c6d | Up to date |
| discourse-gamification a842e183 | Up to date |
| discourse-patreon 778829aa | Up to date |
| discourse-policy b86d520c | Up to date |
| discourse-pushover-notifications 30711ac7 | Up to date |
| discourse-siwe 752687c8 | Up to date |
| discourse-solved 2c1c64af | Up to date |
| discourse-staff-alias 10ae5329 | Up to date |
| discourse-telegram-notifications d9886998 | Up to date |
| discourse-whos-online aeee51e4 | Up to date |
1 Like
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.