### Author:  [@RyouTakeshi](https://meta.discourse.org/u/RyouTakeshi)
#### Post date: [January 31, 2022, 9:23pm UTC](https://meta.discourse.org/t/updating-rebuilding-discourse-to-latest-version-irreversibly-breaks-website/216635/1 "2022-01-31T21:23:06Z")
Hey, we’ve been stuck at a very nasty bug as of late that broke our entire website with no seemingly feasible way out of it. We’ve tried everything from rebuilding to removing/adding plugins to no avail. We have a working backup from a couple months ago, but it seems that any attempt to rebuild this with a newer version just straight up breaks the site.
The main message that gets displayed reads:
```plaintext
Oops
The software powering this discussion forum encountered an unexpected problem. We apologize for the inconvenience.
Detailed information about the error was logged, and an automatic notification generated. We'll take a look at it.
No further action is necessary. However, if the error condition persists, you can provide additional detail, including steps to reproduce the error, by posting a discussion topic in the site's feedback category.
```
This is a generic Discourse error message that yields no actual results when troubleshooting. Checking the logs of our docker, I was able to find this:
```plaintext
Completed 500 Internal Server Error in 152ms (ActiveRecord: 0.0ms | Allocations: 17980)
ActionView::Template::Error (undefined method `[]' for nil:NilClass)
lib/svg_sprite/svg_sprite.rb:502:in `block in custom_icons'
lib/svg_sprite/svg_sprite.rb:500:in `each'
lib/svg_sprite/svg_sprite.rb:500:in `custom_icons'
lib/svg_sprite/svg_sprite.rb:275:in `block in all_icons'
lib/distributed_cache.rb:25:in `defer_get_set'
lib/svg_sprite/svg_sprite.rb:517:in `get_set_cache'
lib/svg_sprite/svg_sprite.rb:268:in `all_icons'
lib/svg_sprite/svg_sprite.rb:337:in `bundle'
lib/svg_sprite/svg_sprite.rb:285:in `block in version'
lib/distributed_cache.rb:25:in `defer_get_set'
lib/svg_sprite/svg_sprite.rb:517:in `get_set_cache'
lib/svg_sprite/svg_sprite.rb:284:in `version'
lib/svg_sprite/svg_sprite.rb:290:in `path'
app/helpers/application_helper.rb:586:in `client_side_setup_data'
app/views/layouts/application.html.erb:61
lib/topic_list_responder.rb:13:in `block (2 levels) in respond_with_list'
lib/topic_list_responder.rb:9:in `respond_with_list'
app/controllers/list_controller.rb:103:in `block (2 levels) in '
app/controllers/application_controller.rb:387:in `block in with_resolved_locale'
app/controllers/application_controller.rb:387:in `with_resolved_locale'
lib/middleware/omniauth_bypass_middleware.rb:71:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:356:in `call'
config/initializers/100-quiet_logger.rb:23:in `call'
config/initializers/100-silence_logger.rb:31:in `call'
lib/middleware/enforce_hostname.rb:23:in `call'
lib/middleware/request_tracker.rb:198:in `call'
```
So, naturally, I thought I could fix the issue by simply implementing a try/catch at line 501 where it errors. This is the method in question:
```plaintext
def self.custom_icons(theme_id)
# Automatically register icons in sprites added via themes or plugins
icons = []
custom_svg_sprites(theme_id).each do |item|
begin
svg_file = Nokogiri::XML(item[:sprite])
svg_file.css('symbol').each do |sym|
icons << sym.attributes['id'].value if sym.attributes['id'].present?
end
rescue Exception:
puts Exception
end
end
icons
end
```
While this _does_ restore functionality to the site, it unfortunately does so by not loading any icons or images on the entire website, so ultimately this doesn’t fix anything.
We’re using the following plugins:
```plaintext
- git clone https://github.com/discourse/docker_manager.git
- git clone https://github.com/discourse/discourse-bbcode.git
- git clone https://github.com/discourse/discourse-follow.git
- git clone https://github.com/discourse/discourse-user-notes.git
```
From what I recall, we’ve used some plugins such as discourse-follow before they were officially supported, so perhaps sidegrading them broke something in the process.
The original version before the update was 2.8.0.beta2, we are now on 2.9.0.beta1
I’ve tried just about everything I can think of, I wasn’t sure if I should post here because I feel as if this is too generic of a bug for anyone to be able to help, but if someone has any ideas as to what’s causing this, I’d appreciate the assistance.
---
### Author:  [@IAmGav](https://meta.discourse.org/u/IAmGav)
#### Post date: [January 31, 2022, 11:04pm UTC](https://meta.discourse.org/t/updating-rebuilding-discourse-to-latest-version-irreversibly-breaks-website/216635/2 "2022-01-31T23:04:42Z")
The could be a Postgres issue where the postgres is trying to upgrade to version 13 from 10.
Check the logs to see if you see anything related to postgres
---
### Author:  [@RyouTakeshi](https://meta.discourse.org/u/RyouTakeshi)
#### Post date: [January 31, 2022, 11:22pm UTC](https://meta.discourse.org/t/updating-rebuilding-discourse-to-latest-version-irreversibly-breaks-website/216635/3 "2022-01-31T23:22:35Z")
> [@IAmGav](#):
>
> The could be a Postgres issue where the postgres is trying to upgrade to version 13 from 10.
>
> Check the logs to see if you see anything related to postgres
Are you referring to the logs contained in /shared/standalone/log/rails? Because they mention nothing relating to postgres. As for the logs contained in /shared/standalone/log/var-log/postgres, at around the time of the error, 19:51:08, there’s a request does not seem out of the ordinary. No errors or anything that seems to fail. And around the time of the update, I don’t think that gets logged here.
Do you have something in particular I can look for?
---
### Author:  [@IAmGav](https://meta.discourse.org/u/IAmGav)
#### Post date: [January 31, 2022, 11:38pm UTC](https://meta.discourse.org/t/updating-rebuilding-discourse-to-latest-version-irreversibly-breaks-website/216635/4 "2022-01-31T23:38:06Z")
make sure your `apt-get update | apt-get upgrade` is up to date
and then run `./launcher rebuild app`
check for errors or look in `/var/discourse/shared/standalone/log/rails`
---
### Author:  [@RyouTakeshi](https://meta.discourse.org/u/RyouTakeshi)
#### Post date: [February 1, 2022, 12:34am UTC](https://meta.discourse.org/t/updating-rebuilding-discourse-to-latest-version-irreversibly-breaks-website/216635/5 "2022-02-01T00:34:55Z")
> [@IAmGav](#):
>
> make sure your `apt-get update | apt-get upgrade` is up to date
> and then run `./launcher rebuild app`
>
> check for errors or look in `/var/discourse/shared/standalone/log/rails`
I’ve updated both `apt-get update` and `apt-get upgrade`, did a full rebuild, and unfortunately am met with the very same errors. From what I can see, nothing changed.
---
### Author:  [@jomaxro](https://meta.discourse.org/u/jomaxro)
#### Post date: [February 1, 2022, 1:04am UTC](https://meta.discourse.org/t/updating-rebuilding-discourse-to-latest-version-irreversibly-breaks-website/216635/6 "2022-02-01T01:04:44Z")
Some questions:
1. Does the rebuild complete without errors?
2. Have you tried loading the site via [safe mode](https://meta.discourse.org/t/53504?silent=true)?
3. Can you share the URL of the site so we can take a look?
---
### Author:  [@RyouTakeshi](https://meta.discourse.org/u/RyouTakeshi)
#### Post date: [February 1, 2022, 1:53am UTC](https://meta.discourse.org/t/updating-rebuilding-discourse-to-latest-version-irreversibly-breaks-website/216635/7 "2022-02-01T01:53:55Z")
> [@jomaxro](#):
>
> Some questions:
>
> 1. Does the rebuild complete without errors?
> 2. Have you tried loading the site via [safe mode](https://meta.discourse.org/t/how-to-use-discourse-safe-mode/53504)?
> 3. Can you share the URL of the site so we can take a look?
1. Yes, the rebuild completes without errors.
2. Yes, and much like when I fix the error mentioned above, it boots the website in a sort of “no icon / image” mode where nothing loads.
3. Yes and no, the site contains some explicit content and I’m going to have to ask around if people are comfortable with me sharing it.
In the meantime, I’ve found out something else, when browsing the website in [safe mode](https://meta.discourse.org/t/53504?silent=true), I noticed that the DiscoTOC throws an error. Upon disabling it, the website works normally again without the need for [safe mode](https://meta.discourse.org/t/53504?silent=true), however the icons/images still do not load properly.
All images are throwing 404s in the console, most icons like the favicon are throwing 500.
Naturally, they were all working fine before.
---
### Author:  [@RyouTakeshi](https://meta.discourse.org/u/RyouTakeshi)
#### Post date: [February 1, 2022, 1:59am UTC](https://meta.discourse.org/t/updating-rebuilding-discourse-to-latest-version-irreversibly-breaks-website/216635/8 "2022-02-01T01:59:55Z")
To add to this, could it be that the rebuild has somehow cleared the database of all image references?
---
### Author:  [@RyouTakeshi](https://meta.discourse.org/u/RyouTakeshi)
#### Post date: [February 1, 2022, 8:54pm UTC](https://meta.discourse.org/t/updating-rebuilding-discourse-to-latest-version-irreversibly-breaks-website/216635/9 "2022-02-01T20:54:15Z")
To keep everyone updated, I’ve since tried out several suggestions from different threads, but just to name two:
> [@Images lost in backup/restore, change of domain, can't upload new](https://meta.discourse.org/t/images-lost-in-backup-restore-change-of-domain-cant-upload-new/168208/8):
>
> My solution was FTP move them (crude but it worked). Not 100% sure why. At first I moved only the optimized images and it didn’t fix the problem, but when I moved all the images it resolved my issue. Thanks for the more detail instruction on how to FTP into my two Discourses. The uploads file was about 125 MB on the old Discourse, 60 MB on the new one. So I copied the old one to my laptop’s Desktop. Then I copied its contents, folder-by-folder, to the new Discourse, skipping any duplications. …
> [@Avatars lost after restore. How to get them back?](https://meta.discourse.org/t/avatars-lost-after-restore-how-to-get-them-back/148223/59):
>
> @Iceman This is mostly FYI, and maybe it is TMI, but it might help you in some small way gain additional insight into some of the interesting things we experience running three containers (two web application and one data container) at the same time (and how it also effects user avatars). It’s is very interesting (and very cool, in my view), how the Redis / Sidekiq job scheduler works when they are running in parallel, but only one “active on the user web side”: Hope you find this short disc…
None of the suggestions helped,`rake posts:rebake`, `rake posts:missing_uploads` and `rake uploads:recover_from_tombstone` all did not work, unfortunately. I’ve also checked out the `sidekiq` dashboard, but that’s not restoring anything either.
To anyone having any idea how to restore the now missing images, I’d be grateful to hear them.
---
### Author:  [@RyouTakeshi](https://meta.discourse.org/u/RyouTakeshi)
#### Post date: [February 2, 2022, 4:42am UTC](https://meta.discourse.org/t/updating-rebuilding-discourse-to-latest-version-irreversibly-breaks-website/216635/10 "2022-02-02T04:42:57Z")
After a lot of trial and error, I was able to successfully restore the images after all, so I’ll consider this issue resolved.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [February 2, 2022, 10:14am UTC](https://meta.discourse.org/t/updating-rebuilding-discourse-to-latest-version-irreversibly-breaks-website/216635/11 "2022-02-02T10:14:17Z")
Was there something that you wish you’d know at the start?
---
### Author:  [@RyouTakeshi](https://meta.discourse.org/u/RyouTakeshi)
#### Post date: [February 4, 2022, 11:45am UTC](https://meta.discourse.org/t/updating-rebuilding-discourse-to-latest-version-irreversibly-breaks-website/216635/12 "2022-02-04T11:45:55Z")
Perhaps having a more thorough look at the website in [safe mode](https://meta.discourse.org/t/53504?silent=true) from the start would’ve helped. I also have to admit, I was only involved in the restoration of the site after some work had already been done on it, it turned out that the images where in a separate backed up folder and only had to be moved over.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [February 4, 2022, 12:36pm UTC](https://meta.discourse.org/t/updating-rebuilding-discourse-to-latest-version-irreversibly-breaks-website/216635/13 "2022-02-04T12:36:05Z")
> [@RyouTakeshi](#):
>
> it turned out that the images where in a separate backed up folder and only had to be moved over.
Hard for anyone to guess that! Glad you figured it out.
---
### Author:  [@jomaxro](https://meta.discourse.org/u/jomaxro)
#### Post date: [March 6, 2022, 12:36pm UTC](https://meta.discourse.org/t/updating-rebuilding-discourse-to-latest-version-irreversibly-breaks-website/216635/14 "2022-03-06T12:36:30Z")
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.