Forum doesn't work after latest upgrade

I’ve just pressed button to update the Discourse engine and got 500 error now:

The software powering this discussion forum encountered an unexpected problem. We apologize for the inconvenience.

The safe mode also doesn’t work. This is the forum URL — https://forum.dudin.tv

How to restore the forum? Which logs I should check out?

Welcome to Meta!
You’re not the only one with that problem

2 Likes

Please can you visit /logs on your forum and see if there are any obvious errors?

2 Likes

.launcher rebuild app worked but I don’t see any difference on the frontend (still have that 500 error)… Should I run any other commands after the “rebuilding”?

Thanks!

I’ve took a look root@forum:/var/discourse/shared/standalone/log/rails# tail -n 60 production.log
and there are some errors, but I’m not a developer to understand what to do with it :confused:

Started GET "/.env" for 78.153.140.179 at 2024-05-17 15:16:40 +0000
ActionController::RoutingError (No route matches [GET] "/.env")
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
  Rendered exceptions/not_found.html.erb within layouts/no_ember (Duration: 181.9ms | Allocations: 3554)
  Rendered layout layouts/no_ember.html.erb (Duration: 2208.3ms | Allocations: 42766)
Failed to handle exception in exception app middleware : ActionView::Template::Error : No such file or directory @ rb_sysopen - /var/www/discourse/node_modules/ember-source/package.json
Started GET "/" for 143.178.246.7 at 2024-05-17 15:16:44 +0000
NoMethodError (undefined method `custom_homepage' for #<ThemeModifierHelper:0x00007fc09c95c1d0 @theme_ids=[]>)
lib/homepage_helper.rb:5:in `resolve'
lib/homepage_constraint.rb:12:in `matches?'
lib/middleware/omniauth_bypass_middleware.rb:64:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:391:in `call'
lib/middleware/csp_script_nonce_injector.rb:12:in `call'
config/initializers/008-rack-cors.rb:14:in `call'
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
Started GET "/" for 5.164.29.116 at 2024-05-17 15:17:43 +0000
NoMethodError (undefined method `custom_homepage' for #<ThemeModifierHelper:0x00007fc09c94eb20 @theme_ids=[]>)
lib/homepage_helper.rb:5:in `resolve'
lib/homepage_constraint.rb:12:in `matches?'
lib/middleware/omniauth_bypass_middleware.rb:64:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:391:in `call'
lib/middleware/csp_script_nonce_injector.rb:12:in `call'
config/initializers/008-rack-cors.rb:14:in `call'
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
Job exception: PG::UndefinedTable: ERROR:  relation "chat_mention_notifications" does not exist
LINE 1: ...ntions.chat_message_id = c_msg.id LEFT OUTER JOIN chat_menti...
                                                             ^


start
done
Job exception: PG::UndefinedTable: ERROR:  relation "problem_check_trackers" does not exist
LINE 1: SELECT "problem_check_trackers".* FROM "problem_check_tracke...
                                               ^


Job exception: PG::UndefinedTable: ERROR:  relation "discourse_automation_automations" does not exist
LINE 9:  WHERE a.attrelid = '"discourse_automation_automations"'::re...

Are these log timestamps before or after you ran the command line rebuild?

The errors all suggest that database migrations have not been run. Which is strange, because they absolutely should be run as part of a successful command-line rebuild.

I run rebuild after these timestamps.

1 Like

Now that the CLI rebuild is done, are you seeing any more errors?

There are fresh ones:

Started GET "/" for 143.178.246.7 at 2024-05-17 15:31:37 +0000
NoMethodError (undefined method `custom_homepage' for #<ThemeModifierHelper:0x00007fc09e86a838 @theme_ids=[]>)
lib/homepage_helper.rb:5:in `resolve'
lib/homepage_constraint.rb:12:in `matches?'
lib/middleware/omniauth_bypass_middleware.rb:64:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:391:in `call'
lib/middleware/csp_script_nonce_injector.rb:12:in `call'
config/initializers/008-rack-cors.rb:14:in `call'
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
Started GET "/" for 143.178.246.7 at 2024-05-17 15:31:38 +0000
NoMethodError (undefined method `custom_homepage' for #<ThemeModifierHelper:0x00007fc09e636698 @theme_ids=[]>)
lib/homepage_helper.rb:5:in `resolve'
lib/homepage_constraint.rb:12:in `matches?'
lib/middleware/omniauth_bypass_middleware.rb:64:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:391:in `call'
lib/middleware/csp_script_nonce_injector.rb:12:in `call'
config/initializers/008-rack-cors.rb:14:in `call'
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
Job exception: PG::UndefinedTable: ERROR:  relation "chat_mention_notifications" does not exist
LINE 1: ...ntions.chat_message_id = c_msg.id LEFT OUTER JOIN chat_menti...
                                                             ^


Started GET "/" for 143.178.246.7 at 2024-05-17 15:33:36 +0000
NoMethodError (undefined method `custom_homepage' for #<ThemeModifierHelper:0x00007fc0ad283528 @theme_ids=[]>)
lib/homepage_helper.rb:5:in `resolve'
lib/homepage_constraint.rb:12:in `matches?'
lib/middleware/omniauth_bypass_middleware.rb:64:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:391:in `call'
lib/middleware/csp_script_nonce_injector.rb:12:in `call'
config/initializers/008-rack-cors.rb:14:in `call'
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
start
done
Started GET "/" for 90.240.110.180 at 2024-05-17 15:33:47 +0000
NoMethodError (undefined method `custom_homepage' for #<ThemeModifierHelper:0x00007fc0ad17c2d8 @theme_ids=[]>)
lib/homepage_helper.rb:5:in `resolve'
lib/homepage_constraint.rb:12:in `matches?'
lib/middleware/omniauth_bypass_middleware.rb:64:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:391:in `call'
lib/middleware/csp_script_nonce_injector.rb:12:in `call'
config/initializers/008-rack-cors.rb:14:in `call'
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'

It looks like producing different errors. There are more errors:

root@forum:/var/discourse/shared/standalone/log/rails# tail -n 60 production.log
lib/middleware/csp_script_nonce_injector.rb:12:in `call'
config/initializers/008-rack-cors.rb:14:in `call'
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
Started GET "/l" for 13.57.1.58 at 2024-05-17 15:34:45 +0000
ActionController::RoutingError (No route matches [GET] "/l")
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
Started GET "/lo" for 13.57.1.58 at 2024-05-17 15:34:45 +0000
  Rendered exceptions/not_found.html.erb within layouts/no_ember (Duration: 2.3ms | Allocations: 185)
  Rendered layout layouts/no_ember.html.erb (Duration: 13.9ms | Allocations: 1405)
Failed to handle exception in exception app middleware : ActionView::Template::Error : No such file or directory @ rb_sysopen - /var/www/discourse/node_modules/ember-source/package.json
ActionController::RoutingError (No route matches [GET] "/lo")
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
ActionController::RoutingError (No route matches [GET] "/logs")
config/initializers/100-silence_logger.rb:27:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
Failed to handle exception in exception app middleware : ActionView::Template::Error : No such file or directory @ rb_sysopen - /var/www/discourse/node_modules/ember-source/package.json
  Rendered exceptions/not_found.html.erb within layouts/no_ember (Duration: 239.6ms | Allocations: 4049)
Started GET "/lo" for 13.57.1.58 at 2024-05-17 15:34:47 +0000
ActionController::RoutingError (No route matches [GET] "/lo")
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
  Rendered exceptions/not_found.html.erb within layouts/no_ember (Duration: 3.9ms | Allocations: 246)
  Rendered layout layouts/no_ember.html.erb (Duration: 16.0ms | Allocations: 1464)
Failed to handle exception in exception app middleware : ActionView::Template::Error : No such file or directory @ rb_sysopen - /var/www/discourse/node_modules/ember-source/package.json
  Rendered layout layouts/no_ember.html.erb (Duration: 1179.6ms | Allocations: 29667)
Failed to handle exception in exception app middleware : ActionView::Template::Error : No such file or directory @ rb_sysopen - /var/www/discourse/node_modules/ember-source/package.json
Started GET "/log" for 13.57.1.58 at 2024-05-17 15:34:47 +0000
ActionController::RoutingError (No route matches [GET] "/log")
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
  Rendered exceptions/not_found.html.erb within layouts/no_ember (Duration: 1.4ms | Allocations: 185)
  Rendered layout layouts/no_ember.html.erb (Duration: 9.5ms | Allocations: 1403)
Failed to handle exception in exception app middleware : ActionView::Template::Error : No such file or directory @ rb_sysopen - /var/www/discourse/node_modules/ember-source/package.json
Started GET "/" for 143.178.246.7 at 2024-05-17 15:35:17 +0000
NoMethodError (undefined method `custom_homepage' for #<ThemeModifierHelper:0x00007fc0a0b75800 @theme_ids=[]>)
lib/homepage_helper.rb:5:in `resolve'
lib/homepage_constraint.rb:12:in `matches?'
lib/middleware/omniauth_bypass_middleware.rb:64:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:391:in `call'
lib/middleware/csp_script_nonce_injector.rb:12:in `call'
config/initializers/008-rack-cors.rb:14:in `call'
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
1 Like

Please can you try:

cd /var/discourse
./launcher enter app
ls -l node_modules | grep ember-source

It should produce something like

# ls -l node_modules | grep ember-source
drwxr-xr-x   9 discourse discourse  4096 May  2 00:23 ember-source
drwxr-xr-x   4 discourse discourse  4096 May  2 00:23 ember-source-channel-url

seems nothing :confused:

root@forum:/var/discourse# ls -l node_modules | grep ember-source
ls: cannot access 'node_modules': No such file or directory

When you ran the CLI rebuild, did it definitely complete successfully? It seems like it might’ve got halfway through, and then failed?

1 Like

It argues that

You have less than 5GB of free space on the disk where /var/lib/docker is located. You will need more space to continue

Maybe this is the reason?.. I’ll try to increase volume.

I thought 25 Gb is enough.

1 Like

I resized the disk to 35 Gb, but it looks like removed /var/discourse

root@forum-app:/var/www/discourse# cd /var/discourse
bash: cd: /var/discourse: No such file or directory

How can I restore the folder?

This files are kept:

root@forum-app:/var/www/discourse# ls
app         config.ru        discourse.sublime-project  jsconfig.json  migrations    public     test
bin         CONTRIBUTING.md  docs                       lefthook.yml   node_modules  Rakefile   tmp
Brewfile    COPYRIGHT.md     Gemfile                    lib            package.json  README.md  translator.yml
CODEOWNERS  d                Gemfile.lock               LICENSE.txt    patches       script     vendor
config      db               images                     log            plugins       spec       yarn.lock

It looks like you are running this inside the docker container. Run exit to get back to your server.

Then run ./launcher rebuild app to run a rebuild.

2 Likes

Now it worked much longer and looks well!
Now, I have 502 Bad Gateway :slight_smile:

The forum is working!

Thank you so much! Appreciate your quick response :heart:

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.