Hey, we’ve been stuck at a very nasty bug as of late that broke our entire website with no seemingly feasible way out of it. We’ve tried everything from rebuilding to removing/adding plugins to no avail. We have a working backup from a couple months ago, but it seems that any attempt to rebuild this with a newer version just straight up breaks the site. The main message that gets displayed reads: Oops The software powering this discussion forum encountered an unexpected problem. We apologize for the inconvenience. Detailed information about the error was logged, and an automatic notification generated. We'll take a look at it. No further action is necessary. However, if the error condition persists, you can provide additional detail, including steps to reproduce the error, by posting a discussion topic in the site's feedback category. This is a generic Discourse error message that yields no actual results when troubleshooting. Checking the logs of our docker, I was able to find this: Completed 500 Internal Server Error in 152ms (ActiveRecord: 0.0ms | Allocations: 17980) ActionView::Template::Error (undefined method `[]' for nil:NilClass) lib/svg_sprite/svg_sprite.rb:502:in `block in custom_icons' lib/svg_sprite/svg_sprite.rb:500:in `each' lib/svg_sprite/svg_sprite.rb:500:in `custom_icons' lib/svg_sprite/svg_sprite.rb:275:in `block in all_icons' lib/distributed_cache.rb:25:in `defer_get_set' lib/svg_sprite/svg_sprite.rb:517:in `get_set_cache' lib/svg_sprite/svg_sprite.rb:268:in `all_icons' lib/svg_sprite/svg_sprite.rb:337:in `bundle' lib/svg_sprite/svg_sprite.rb:285:in `block in version' lib/distributed_cache.rb:25:in `defer_get_set' lib/svg_sprite/svg_sprite.rb:517:in `get_set_cache' lib/svg_sprite/svg_sprite.rb:284:in `version' lib/svg_sprite/svg_sprite.rb:290:in `path' app/helpers/application_helper.rb:586:in `client_side_setup_data' app/views/layouts/application.html.erb:61 lib/topic_list_responder.rb:13:in `block (2 levels) in respond_with_list' lib/topic_list_responder.rb:9:in `respond_with_list' app/controllers/list_controller.rb:103:in `block (2 levels) in ' app/controllers/application_controller.rb:387:in `block in with_resolved_locale' app/controllers/application_controller.rb:387:in `with_resolved_locale' lib/middleware/omniauth_bypass_middleware.rb:71:in `call' lib/content_security_policy/middleware.rb:12:in `call' lib/middleware/anonymous_cache.rb:356:in `call' config/initializers/100-quiet_logger.rb:23:in `call' config/initializers/100-silence_logger.rb:31:in `call' lib/middleware/enforce_hostname.rb:23:in `call' lib/middleware/request_tracker.rb:198:in `call' So, naturally, I thought I could fix the issue by simply implementing a try/catch at line 501 where it errors. This is the method in question: def self.custom_icons(theme_id) # Automatically register icons in sprites added via themes or plugins icons = [] custom_svg_sprites(theme_id).each do |item| begin svg_file = Nokogiri::XML(item[:sprite]) svg_file.css('symbol').each do |sym| icons << sym.attributes['id'].value if sym.attributes['id'].present? end rescue Exception: puts Exception end end icons end While this does restore functionality to the site, it unfortunately does so by not loading any icons or images on the entire website, so ultimately this doesn’t fix anything. We’re using the following plugins: - git clone https://github.com/discourse/docker_manager.git - git clone https://github.com/discourse/discourse-bbcode.git - git clone https://github.com/discourse/discourse-follow.git - git clone https://github.com/discourse/discourse-user-notes.git From what I recall, we’ve used some plugins such as discourse-follow before they were officially supported, so perhaps sidegrading them broke something in the process. The original version before the update was 2.8.0.beta2, we are now on 2.9.0.beta1 I’ve tried just about everything I can think of, I wasn’t sure if I should post here because I feel as if this is too generic of a bug for anyone to be able to help, but if someone has any ideas as to what’s causing this, I’d appreciate the assistance.

The could be a Postgres issue where the postgres is trying to upgrade to version 13 from 10. Check the logs to see if you see anything related to postgres

[image] IAmGav: The could be a Postgres issue where the postgres is trying to upgrade to version 13 from 10. Check the logs to see if you see anything related to postgres Are you referring to the logs contained in /shared/standalone/log/rails? Because they mention nothing relating to postgres. As for the logs contained in /shared/standalone/log/var-log/postgres, at around the time of the error, 19:51:08, there’s a request does not seem out of the ordinary. No errors or anything that seems to fail. And around the time of the update, I don’t think that gets logged here. Do you have something in particular I can look for?

make sure your apt-get update | apt-get upgrade is up to date and then run ./launcher rebuild app check for errors or look in /var/discourse/shared/standalone/log/rails

[image] IAmGav: make sure your apt-get update | apt-get upgrade is up to date and then run ./launcher rebuild app check for errors or look in /var/discourse/shared/standalone/log/rails I’ve updated both apt-get update and apt-get upgrade, did a full rebuild, and unfortunately am met with the very same errors. From what I can see, nothing changed.

Some questions: Does the rebuild complete without errors? Have you tried loading the site via safe mode? Can you share the URL of the site so we can take a look?

[image] jomaxro: Some questions: Does the rebuild complete without errors? Have you tried loading the site via safe mode? Can you share the URL of the site so we can take a look? Yes, the rebuild completes without errors. Yes, and much like when I fix the error mentioned above, it boots the website in a sort of “no icon / image” mode where nothing loads. Yes and no, the site contains some explicit content and I’m going to have to ask around if people are comfortable with me sharing it. In the meantime, I’ve found out something else, when browsing the website in safe mode, I noticed that the DiscoTOC throws an error. Upon disabling it, the website works normally again without the need for safe mode, however the icons/images still do not load properly. [image] All images are throwing 404s in the console, most icons like the favicon are throwing 500. Naturally, they were all working fine before.

To add to this, could it be that the rebuild has somehow cleared the database of all image references?

To keep everyone updated, I’ve since tried out several suggestions from different threads, but just to name two: [image] Images lost in backup/restore, change of domain, can't upload new Support My solution was FTP move them (crude but it worked). Not 100% sure why. At first I moved only the optimized images and it didn’t fix the problem, but when I moved all the images it resolved my issue. Thanks for the more detail instruction on how to FTP into my two Discourses. The uploads file was about 125 MB on the old Discourse, 60 MB on the new one. So I copied the old one to my laptop’s Desktop. Then I copied its contents, folder-by-folder, to the new Discourse, skipping any duplications. … [image] Avatars lost after restore. How to get them back? Support @Iceman This is mostly FYI, and maybe it is TMI, but it might help you in some small way gain additional insight into some of the interesting things we experience running three containers (two web application and one data container) at the same time (and how it also effects user avatars). It’s is very interesting (and very cool, in my view), how the Redis / Sidekiq job scheduler works when they are running in parallel, but only one “active on the user web side”: Hope you find this short disc… None of the suggestions helped,rake posts:rebake, rake posts:missing_uploads and rake uploads:recover_from_tombstone all did not work, unfortunately. I’ve also checked out the sidekiq dashboard, but that’s not restoring anything either. To anyone having any idea how to restore the now missing images, I’d be grateful to hear them.

After a lot of trial and error, I was able to successfully restore the images after all, so I’ll consider this issue resolved.

Was there something that you wish you’d know at the start?

Updating / Rebuilding Discourse to latest version irreversibly breaks website

Product Support Self-hosting

pfaffman (Jay Pfaffman) February 4, 2022, 12:36pm 13

Hard for anyone to guess that! Glad you figured it out.

Topic		Replies	Views
Attempted to update Discourse, frontpage now errors Product Support	2	1064	July 23, 2016
Problem upgrading Discourse Self-hosting	4	1663	October 11, 2019
Error when updating Discourse to the new version (need urgent support)? Product Support	14	625	October 1, 2024
After rebuild, site won't load Self-hosting	13	2335	April 23, 2018
An upgrade knocked my site offline; how long until it's restored? Self-hosting	15	726	July 10, 2025

Updating / Rebuilding Discourse to latest version irreversibly breaks website

Related topics