dudintv reported a 500 error after updating their Discourse forum, with safe mode also failing. Moin noted others faced similar issues and linked to a related thread. David suggested checking /logs, which dudintv did, revealing errors like missing database tables and Ember.js files. After running .launcher rebuild app, errors persisted, prompting David to suspect incomplete database migrations. dudintv confirmed logs were from before the rebuild. Fresh logs showed continued NoMethodError and missing ember-source files. David asked to verify ember-source in node_modules, but dudintv found the directory missing. David suspected a failed rebuild, which dudintv confirmed was due to low disk space (under 5GB). After resizing to 35GB, dudintv accidentally deleted /var/discourse while inside the container. David advised exiting the container and rebuilding. The rebuild succeeded, though a 502 error appeared briefly dudintv. Finally, the forum was restored and working dudintv, who thanked David for the help.
I’ve just pressed button to update the Discourse engine and got 500 error now:
The software powering this discussion forum encountered an unexpected problem. We apologize for the inconvenience.
The safe mode also doesn’t work. This is the forum URL — https://forum.dudin.tv
How to restore the forum? Which logs I should check out?
Welcome to Meta!
You’re not the only one with that problem
Please can you visit /logs on your forum and see if there are any obvious errors?
.launcher rebuild app worked but I don’t see any difference on the frontend (still have that 500 error)… Should I run any other commands after the “rebuilding”?
Thanks!
I’ve took a look root@forum:/var/discourse/shared/standalone/log/rails# tail -n 60 production.log
and there are some errors, but I’m not a developer to understand what to do with it 
Started GET "/.env" for 78.153.140.179 at 2024-05-17 15:16:40 +0000
ActionController::RoutingError (No route matches [GET] "/.env")
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
Rendered exceptions/not_found.html.erb within layouts/no_ember (Duration: 181.9ms | Allocations: 3554)
Rendered layout layouts/no_ember.html.erb (Duration: 2208.3ms | Allocations: 42766)
Failed to handle exception in exception app middleware : ActionView::Template::Error : No such file or directory @ rb_sysopen - /var/www/discourse/node_modules/ember-source/package.json
Started GET "/" for 143.178.246.7 at 2024-05-17 15:16:44 +0000
NoMethodError (undefined method `custom_homepage' for #<ThemeModifierHelper:0x00007fc09c95c1d0 @theme_ids=[]>)
lib/homepage_helper.rb:5:in `resolve'
lib/homepage_constraint.rb:12:in `matches?'
lib/middleware/omniauth_bypass_middleware.rb:64:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:391:in `call'
lib/middleware/csp_script_nonce_injector.rb:12:in `call'
config/initializers/008-rack-cors.rb:14:in `call'
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
Started GET "/" for 5.164.29.116 at 2024-05-17 15:17:43 +0000
NoMethodError (undefined method `custom_homepage' for #<ThemeModifierHelper:0x00007fc09c94eb20 @theme_ids=[]>)
lib/homepage_helper.rb:5:in `resolve'
lib/homepage_constraint.rb:12:in `matches?'
lib/middleware/omniauth_bypass_middleware.rb:64:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:391:in `call'
lib/middleware/csp_script_nonce_injector.rb:12:in `call'
config/initializers/008-rack-cors.rb:14:in `call'
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
Job exception: PG::UndefinedTable: ERROR: relation "chat_mention_notifications" does not exist
LINE 1: ...ntions.chat_message_id = c_msg.id LEFT OUTER JOIN chat_menti...
^
start
done
Job exception: PG::UndefinedTable: ERROR: relation "problem_check_trackers" does not exist
LINE 1: SELECT "problem_check_trackers".* FROM "problem_check_tracke...
^
Job exception: PG::UndefinedTable: ERROR: relation "discourse_automation_automations" does not exist
LINE 9: WHERE a.attrelid = '"discourse_automation_automations"'::re...
Are these log timestamps before or after you ran the command line rebuild?
The errors all suggest that database migrations have not been run. Which is strange, because they absolutely should be run as part of a successful command-line rebuild.
I run rebuild after these timestamps.
Now that the CLI rebuild is done, are you seeing any more errors?
There are fresh ones:
Started GET "/" for 143.178.246.7 at 2024-05-17 15:31:37 +0000
NoMethodError (undefined method `custom_homepage' for #<ThemeModifierHelper:0x00007fc09e86a838 @theme_ids=[]>)
lib/homepage_helper.rb:5:in `resolve'
lib/homepage_constraint.rb:12:in `matches?'
lib/middleware/omniauth_bypass_middleware.rb:64:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:391:in `call'
lib/middleware/csp_script_nonce_injector.rb:12:in `call'
config/initializers/008-rack-cors.rb:14:in `call'
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
Started GET "/" for 143.178.246.7 at 2024-05-17 15:31:38 +0000
NoMethodError (undefined method `custom_homepage' for #<ThemeModifierHelper:0x00007fc09e636698 @theme_ids=[]>)
lib/homepage_helper.rb:5:in `resolve'
lib/homepage_constraint.rb:12:in `matches?'
lib/middleware/omniauth_bypass_middleware.rb:64:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:391:in `call'
lib/middleware/csp_script_nonce_injector.rb:12:in `call'
config/initializers/008-rack-cors.rb:14:in `call'
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
Job exception: PG::UndefinedTable: ERROR: relation "chat_mention_notifications" does not exist
LINE 1: ...ntions.chat_message_id = c_msg.id LEFT OUTER JOIN chat_menti...
^
Started GET "/" for 143.178.246.7 at 2024-05-17 15:33:36 +0000
NoMethodError (undefined method `custom_homepage' for #<ThemeModifierHelper:0x00007fc0ad283528 @theme_ids=[]>)
lib/homepage_helper.rb:5:in `resolve'
lib/homepage_constraint.rb:12:in `matches?'
lib/middleware/omniauth_bypass_middleware.rb:64:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:391:in `call'
lib/middleware/csp_script_nonce_injector.rb:12:in `call'
config/initializers/008-rack-cors.rb:14:in `call'
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
start
done
Started GET "/" for 90.240.110.180 at 2024-05-17 15:33:47 +0000
NoMethodError (undefined method `custom_homepage' for #<ThemeModifierHelper:0x00007fc0ad17c2d8 @theme_ids=[]>)
lib/homepage_helper.rb:5:in `resolve'
lib/homepage_constraint.rb:12:in `matches?'
lib/middleware/omniauth_bypass_middleware.rb:64:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:391:in `call'
lib/middleware/csp_script_nonce_injector.rb:12:in `call'
config/initializers/008-rack-cors.rb:14:in `call'
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
It looks like producing different errors. There are more errors:
root@forum:/var/discourse/shared/standalone/log/rails# tail -n 60 production.log
lib/middleware/csp_script_nonce_injector.rb:12:in `call'
config/initializers/008-rack-cors.rb:14:in `call'
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
Started GET "/l" for 13.57.1.58 at 2024-05-17 15:34:45 +0000
ActionController::RoutingError (No route matches [GET] "/l")
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
Started GET "/lo" for 13.57.1.58 at 2024-05-17 15:34:45 +0000
Rendered exceptions/not_found.html.erb within layouts/no_ember (Duration: 2.3ms | Allocations: 185)
Rendered layout layouts/no_ember.html.erb (Duration: 13.9ms | Allocations: 1405)
Failed to handle exception in exception app middleware : ActionView::Template::Error : No such file or directory @ rb_sysopen - /var/www/discourse/node_modules/ember-source/package.json
ActionController::RoutingError (No route matches [GET] "/lo")
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
ActionController::RoutingError (No route matches [GET] "/logs")
config/initializers/100-silence_logger.rb:27:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
Failed to handle exception in exception app middleware : ActionView::Template::Error : No such file or directory @ rb_sysopen - /var/www/discourse/node_modules/ember-source/package.json
Rendered exceptions/not_found.html.erb within layouts/no_ember (Duration: 239.6ms | Allocations: 4049)
Started GET "/lo" for 13.57.1.58 at 2024-05-17 15:34:47 +0000
ActionController::RoutingError (No route matches [GET] "/lo")
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
Rendered exceptions/not_found.html.erb within layouts/no_ember (Duration: 3.9ms | Allocations: 246)
Rendered layout layouts/no_ember.html.erb (Duration: 16.0ms | Allocations: 1464)
Failed to handle exception in exception app middleware : ActionView::Template::Error : No such file or directory @ rb_sysopen - /var/www/discourse/node_modules/ember-source/package.json
Rendered layout layouts/no_ember.html.erb (Duration: 1179.6ms | Allocations: 29667)
Failed to handle exception in exception app middleware : ActionView::Template::Error : No such file or directory @ rb_sysopen - /var/www/discourse/node_modules/ember-source/package.json
Started GET "/log" for 13.57.1.58 at 2024-05-17 15:34:47 +0000
ActionController::RoutingError (No route matches [GET] "/log")
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
Rendered exceptions/not_found.html.erb within layouts/no_ember (Duration: 1.4ms | Allocations: 185)
Rendered layout layouts/no_ember.html.erb (Duration: 9.5ms | Allocations: 1403)
Failed to handle exception in exception app middleware : ActionView::Template::Error : No such file or directory @ rb_sysopen - /var/www/discourse/node_modules/ember-source/package.json
Started GET "/" for 143.178.246.7 at 2024-05-17 15:35:17 +0000
NoMethodError (undefined method `custom_homepage' for #<ThemeModifierHelper:0x00007fc0a0b75800 @theme_ids=[]>)
lib/homepage_helper.rb:5:in `resolve'
lib/homepage_constraint.rb:12:in `matches?'
lib/middleware/omniauth_bypass_middleware.rb:64:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:391:in `call'
lib/middleware/csp_script_nonce_injector.rb:12:in `call'
config/initializers/008-rack-cors.rb:14:in `call'
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:277:in `call'
Please can you try:
cd /var/discourse
./launcher enter app
ls -l node_modules | grep ember-source
It should produce something like
# ls -l node_modules | grep ember-source
drwxr-xr-x 9 discourse discourse 4096 May 2 00:23 ember-source
drwxr-xr-x 4 discourse discourse 4096 May 2 00:23 ember-source-channel-url
seems nothing
root@forum:/var/discourse# ls -l node_modules | grep ember-source
ls: cannot access 'node_modules': No such file or directory
When you ran the CLI rebuild, did it definitely complete successfully? It seems like it might’ve got halfway through, and then failed?
It argues that
You have less than 5GB of free space on the disk where /var/lib/docker is located. You will need more space to continue
Maybe this is the reason?.. I’ll try to increase volume.
I thought 25 Gb is enough.
I resized the disk to 35 Gb, but it looks like removed /var/discourse
root@forum-app:/var/www/discourse# cd /var/discourse
bash: cd: /var/discourse: No such file or directory
How can I restore the folder?
This files are kept:
root@forum-app:/var/www/discourse# ls
app config.ru discourse.sublime-project jsconfig.json migrations public test
bin CONTRIBUTING.md docs lefthook.yml node_modules Rakefile tmp
Brewfile COPYRIGHT.md Gemfile lib package.json README.md translator.yml
CODEOWNERS d Gemfile.lock LICENSE.txt patches script vendor
config db images log plugins spec yarn.lock
It looks like you are running this inside the docker container. Run exit to get back to your server.
Then run ./launcher rebuild app to run a rebuild.
Now it worked much longer and looks well!
Now, I have 502 Bad Gateway 
The forum is working!
Thank you so much! Appreciate your quick response 