Our Discourse installation, a standard install based on the official repo but using SSO integration as its sole login system, has been running for several years serving a larger userbase, running on an Ubuntu-based dedicated server operating some other applications, without significant issue I am delighted to say.
Unfortunately, it recently suffered a bit of a spontaneous failure and no longer seems to be working at all.
Initial reports were that the site was exhibiting a lot of seemingly random 500 errors and notifications were not being delivered, but otherwise it still functioned. The context was that the install was last updated a month or two ago and no changes had been made to it or the server environment, so this was a surprising development.
We confirmed 500 errors were occurring, and first tried restarting the server and the container to no effect. An “apt-get update” was run to get all packages up to date as well. We also tried to update the Discourse via the backend, but whilst logging in with the admin link no login links were being sent out by email, and after the container restart the admin login page only generated a 500 error.
I also checked the server environment for any abnormalities with hardware, OS or otherwise, and nothing seemed to be awry.
Next we attempted full rebuild and “git pull” update via CLI. This failed during the upgrade’s database migrations with the following error:
I, [2022-08-09T23:15:21.229857 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'LOAD_PLUGINS=0 bundle exec rake plugin:pull_compatible_all'
I, [2022-08-09T23:15:23.938876 #1] INFO -- : docker_manager is already at latest compatible version
I, [2022-08-09T23:15:23.939220 #1] INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate'
2022-08-09 23:15:28.956 UTC [835] discourse@discourse ERROR: duplicate key value violates unique constraint "idx_bookmarks_user_polymorphic_unique"
2022-08-09 23:15:28.956 UTC [835] discourse@discourse DETAIL: Key (user_id, bookmarkable_type, bookmarkable_id)=(4115, Post, 103355) already exists.
2022-08-09 23:15:28.956 UTC [835] discourse@discourse STATEMENT: UPDATE bookmarks
SET bookmarkable_id = post_id, bookmarkable_type = 'Post'
WHERE NOT bookmarks.for_topic AND bookmarkable_id IS NULL
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
ERROR: duplicate key value violates unique constraint "idx_bookmarks_user_polymorphic_unique"
DETAIL: Key (user_id, bookmarkable_type, bookmarkable_id)=(4115, Post, 103355) already exists.
If there is a database inconsistency, I have no idea how it occurred… I can provide the rest of the console history if needed.
Running “./discourse-doctor” completed without apparent issue, except for a warning about mail server configuration - “certificate verify failed (self signed certificate)” - which might explain why the emails were not being sent out. Previously they were being sent out with configuration as is, and the mail server seems to be otherwise functioning without issue so I am not sure about this.
Currently the site is completely down (just returning 500 errors after the failed update). We don’t really have any idea how to proceed at this point, beyond trying to seek external support. Any guidance would be much appeciated!