Stuck with 500 error after weird bugs and a rebuild

Currently my site is down. No clue why, but I will do my best to recount what happened and what I’ve tried.


I haven’t visited my site in a month at this point, but I didn’t hear complaints of anything broken from any users or mods. Today I visited, opened a few pages in new tabs, and went to the admin dashboard.

On the admin dashboard, I saw that 3 theme components were out of date. One was discotoc, I don’t remember the other two. I used the UI to update them. I also saw that it recommended that I remove the checklist plugin, but I’m like 90% certain I didn’t have it in my plugin list.

I was visiting one of the category pages, and it seemed really buggy. At first, I was getting a browser error that the page was unresponsive. Another visit and it was showing an empty category list, and another visit it was showing the top of the category page UI repeated like 100 times.

  • POTENTIALLY this has something to do with the theme component updates???

Anyway, I thought it was weird, and the dashboard told me I should update, so I went into the command line and ran ./launcher rebuild app. I don’t recall if that failed or not. After it finished running, the site wouldn’t load and shows the 500 error above.

We tried a whole bunch of things and nothing seems to fix it

  • I can enter the app via command line and create a backup and it also uploaded to our backup bucket
  • running discourse doctor works. It successfully sends a test email, and I could successfully view the output .txt file in the browser being served from the website url. (it seems to be down right now, maybe since I ran rebuild again?)
    • It did show that discourse version was NOT FOUND for both the site and localhost
  • pinging the site from within the droplet cmd works just fine
  • tried restarting the DO droplet
  • tried commenting out all plugins and rebuilding
  • tried running git pull, it said everything was up to date already
  • tried wiping the static assets cdn cache
  • tried setting the discourse version to v3.2.0.beta1, which definitely worked in the past
  • tried visiting the /safe-mode and /?safe_mode=no_themes,no_plugins, but get the same 500 error

We’ve seen a variety of errors, some fatal, some not. But it doesn’t seem like we’ve figured anything out.

This one we’ve worked around, I don’t remember what we did. Maybe disable plugins?

1 gem installed
gem install prizm -v 0.0.3 -i /var/www/discourse/plugins/discourse-topic-previews-sidecar/gems/3.2.2 --no-document --ignore-dependencies --no-user-install
Successfully installed prizm-0.0.3
1 gem installed
 
I, [2024-01-06T03:44:20.128194 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile'
Plugin name is 'discourse-topic-voting', but plugin directory is named 'discourse-voting'
^Cnode:events:492
      throw er; // Unhandled 'error' event
      ^
 
Error: write EPIPE
    at afterWriteDispatched (node:internal/stream_base_commons:160:15)
    at writeGeneric (node:internal/stream_base_commons:151:3)
    at Socket._writeGeneric (node:net:931:11)
    at Socket._write (node:net:943:8)
    at writeOrBuffer (node:internal/streams/writable:392:12)
    at _write (node:internal/streams/writable:333:10)
    at Writable.write (node:internal/streams/writable:337:10)
    at DestroyableTransform.ondata (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_readable.js:629:20)
    at DestroyableTransform.emit (node:events:514:28)
    at addChunk (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_readable.js:279:12)
    at readableAddChunk (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_readable.js:262:11)
    at Readable.push (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_readable.js:228:10)
    at Transform.push (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_transform.js:132:32)
    at DestroyableTransform.afterTransform (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_transform.js:85:10)
    at DestroyableTransform._transform (/var/www/discourse/app/assets/javascripts/node_modules/console-ui/lib/index.js:46:7)
    at Transform._read (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_transform.js:166:10)
    at Transform._write (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_transform.js:155:83)
    at doWrite (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_writable.js:390:139)
    at writeOrBuffer (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_writable.js:381:5)
    at Writable.write (/var/www/discourse/app/assets/javascripts/node_modules/readable-stream/lib/_stream_writable.js:302:11)
    at UI.write (/var/www/discourse/app/assets/javascripts/node_modules/console-ui/lib/index.js:73:25)
    at UI.writeLine (/var/www/discourse/app/assets/javascripts/node_modules/console-ui/lib/index.js:85:10)
    at UI.writeInfoLine (/var/www/discourse/app/assets/javascripts/node_modules/console-ui/lib/index.js:111:10)
    at Class._handleProgress (/var/www/discourse/app/assets/javascripts/ember-cli-progress-ci/index.js:45:23)
    at listOnTimeout (node:internal/timers:569:17)
    at process.processTimers (node:internal/timers:512:7)
Emitted 'error' event on Socket instance at:
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at processTicksAndRejections (node:internal/process/task_queues:82:21)
    at runNextTicks (node:internal/process/task_queues:64:3)
    at process.processTimers (node:internal/timers:509:9) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}
 
Node.js v18.17.1
error Command failed with exit code 1.
I, [2024-01-06T03:45:42.152091 #1]  INFO -- : Terminating async processes
I, [2024-01-06T03:45:42.152209 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 41
I, [2024-01-06T03:45:42.152250 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 110
110:signal-handler (1704512742) Received SIGTERM scheduling shutdown...
2024-01-06 03:45:42.152 UTC [41] LOG:  received fast shutdown request
2024-01-06 03:45:42.179 UTC [41] LOG:  aborting any active transactions
2024-01-06 03:45:42.181 UTC [41] LOG:  background worker "logical replication launcher" (PID 50) exited with exit code 1
2024-01-06 03:45:42.183 UTC [45] LOG:  shutting down
2024-01-06 03:45:42.222 UTC [41] LOG:  database system is shut down
110:M 06 Jan 2024 03:45:42.240 # User requested shutdown...
110:M 06 Jan 2024 03:45:42.240 * Saving the final RDB snapshot before exiting.
110:M 06 Jan 2024 03:45:45.463 * DB saved on disk
110:M 06 Jan 2024 03:45:45.463 # Redis is now ready to exit, bye bye...
/usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/exec_command.rb:122:in `readlines': Interrupt
        from /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/exec_command.rb:122:in `block in spawn'
        from /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/exec_command.rb:115:in `popen'
        from /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/exec_command.rb:115:in `spawn'
        from /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/exec_command.rb:88:in `block in run'
        from /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/exec_command.rb:86:in `each'
        from /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/exec_command.rb:86:in `run'
        from /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/command.rb:10:in `run'
        from /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/config.rb:227:in `block (2 levels) in run_commands'
        from /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/config.rb:211:in `each'
        from /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/config.rb:211:in `block in run_commands'
        from /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/config.rb:210:in `each'
        from /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/config.rb:210:in `run_commands'
        from /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/config.rb:191:in `run'
        from /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/cli.rb:89:in `run'
        from /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/bin/pups:9:in `<top (required)>'
        from /usr/local/bin/pups:25:in `load'
        from /usr/local/bin/pups:25:in `<main>'
sha256:da4950febbafda88432e357e0bd7cc8e0ba528b750569e60cce2a05dcdbed2a8
18c1a3ab556d2c4ddca2cf35a934fba94d748e48c1e09907ca16e17b91b64564
Removing old container
+ /usr/bin/docker rm app
app

The below error was found in the unicorn logs, but idk if this is fatal (only have a screenshot of some of the text)

app error: Missing 'secret_key_base' for 'production' environment, set this stri....

And another weird one where I entered the app to make a backup. But then a 2nd person entered the app and then it starts downloading a discourse image. There were no changes or restarts between these two events, though we were accessing on two separate computers.

./launcher enter app
WARNING: We are about to start downloading the Discourse base image
This process may take anywhere between a few minutes to an hour, depending on your network speed

Please be patient
...

Running out of ideas on what to do.

  • Haven’t tried anything regarding disabling theme-components, not sure how to do that if I can’t visit the admin panel.
  • I guess we could wipe the droplet and restore from either the latest or 2nd latest backup
  • not sure what else

500 is a back-end error, so this is unlikely to be down to a Theme Component.

I suggest rebuilding without any unofficial plugins, see if that works, then add any additional plugins back one by one, rebuild at a time and see if you can isolate the issue.

Make sure you are using the original sources of plugins and not any unsupported forks.

I deleted all the plugin lines just to be extra safe. Rebuild failed, so changed the version to the commit hash of v3.2.0.beta1. It still doesn’t work and I’m seeing a couple errors in the log. The below logs are from running discourse-doctor, which failed and then tried to rebuild

I, [2024-01-06T16:44:38.708297 #1]  INFO -- : > cd /var/www/discourse && git reset --hard
205:C 06 Jan 2024 16:44:38.710 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
205:C 06 Jan 2024 16:44:38.710 # Redis version=6.2.3, bits=64, commit=00000000, modified=0, pid=205, just started
205:C 06 Jan 2024 16:44:38.710 # Configuration loaded
205:M 06 Jan 2024 16:44:38.711 * monotonic clock: POSIX clock_gettime
205:M 06 Jan 2024 16:44:38.711 * Running mode=standalone, port=6379.
205:M 06 Jan 2024 16:44:38.711 # Server initialized
205:M 06 Jan 2024 16:44:38.711 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
205:M 06 Jan 2024 16:44:38.711 # Can't handle RDB format version 10
205:M 06 Jan 2024 16:44:38.711 # Fatal error loading the DB: Invalid argument. Exiting.
I, [2024-01-06T16:44:49.503562 #1]  INFO -- : > cd /var/www/discourse && find /var/www/discourse ! -user discours
e -exec chown discourse {} \+
I, [2024-01-06T16:44:52.109667 #1]  INFO -- : 
I, [2024-01-06T16:44:52.109952 #1]  INFO -- : > cd /var/www/discourse && su discourse -c 'bundle install --deploy
ment --retry 3 --jobs 4 --verbose --without test development'
[DEPRECATED] The `--deployment` flag is deprecated because it relies on being remembered across bundler invocatio
ns, which bundler will no longer do in future versions. Instead please use `bundle config set --local deployment 
'true'`, and stop using this flag
[DEPRECATED] The `--without` flag is deprecated because it relies on being remembered across bundler invocations,
 which bundler will no longer do in future versions. Instead please use `bundle config set --local without 'test 
development'`, and stop using this flag
Bundler::PathError: The path `/var/www/discourse/vendor/bundle/ruby/2.7.0/bundler/gems/sprockets-f4d3dae71ef2` do
es not exist.
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/source/path.rb:209:in `load_spec_files'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/source/git.rb:206:in `load_spec_files'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/source/path.rb:107:in `local_specs'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/source/git.rb:172:in `specs'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/lazy_specification.rb:82:in `materialize_for_ins
tallation'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/spec_set.rb:205:in `map!'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/spec_set.rb:205:in `specs_for_dependency'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/spec_set.rb:34:in `block in for'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/spec_set.rb:22:in `loop'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/spec_set.rb:22:in `for'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/spec_set.rb:85:in `materialize'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/definition.rb:209:in `missing_specs'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/definition.rb:213:in `missing_specs?'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/installer.rb:255:in `resolve_if_needed'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/installer.rb:82:in `block in run'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/process_lock.rb:12:in `block in lock'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/process_lock.rb:9:in `open'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/process_lock.rb:9:in `lock'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/installer.rb:71:in `run'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/installer.rb:23:in `install'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/cli/install.rb:62:in `run'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/cli.rb:261:in `block in install'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/settings.rb:131:in `temporary'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/cli.rb:260:in `install'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
 /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/cli.rb:34:in `dispatch'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/cli.rb:28:in `start'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/exe/bundle:45:in `block in <top (required)>'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/exe/bundle:33:in `<top (required)>'
  /usr/local/bin/bundle:23:in `load'
  /usr/local/bin/bundle:23:in `<main>'
Bundler::InstallError: colored2-4.0.0 requires ruby version >= 3.1, which is incompatible with the current version, 2.7.2
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/installer.rb:233:in `block in ensure_specs_are_compatible!'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/spec_set.rb:165:in `each'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/spec_set.rb:165:in `each'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/installer.rb:231:in `ensure_specs_are_compatible!'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/installer.rb:83:in `block in run'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/process_lock.rb:12:in `block in lock'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/process_lock.rb:9:in `open'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/process_lock.rb:9:in `lock'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/installer.rb:71:in `run'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/installer.rb:23:in `install'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/cli/install.rb:62:in `run'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/cli.rb:261:in `block in install'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/settings.rb:131:in `temporary'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/cli.rb:260:in `install'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/cli.rb:34:in `dispatch'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/cli.rb:28:in `start'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/exe/bundle:45:in `block in <top (required)>'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.4.13/exe/bundle:33:in `<top (required)>'
  /usr/local/bin/bundle:23:in `load'
  /usr/local/bin/bundle:23:in `<main>'
I, [2024-01-06T16:45:02.658038 #1]  INFO -- : Running `bundle install --deployment --jobs 4 --retry 3 --verbose --without "test" "development"` with bundler 2.4.13
Frozen, using resolution from the lockfile
The definition is missing dependencies, failed to resolve & materialize locally (https://github.com/rails/sprockets (at 3.x@f4d3dae) is not yet checked out. Run `bundle install` first.)
Frozen, using resolution from the lockfile
HTTP GET https://index.rubygems.org/versions
HTTP 206 Partial Content https://index.rubygems.org/versions
HTTP GET https://index.rubygems.org/versions
HTTP 200 OK https://index.rubygems.org/versions
Fetching gem metadata from https://rubygems.org/
HTTP 206 Partial Content https://index.rubygems.org/info/domain_name
HTTP 200 OK https://index.rubygems.org/info/cucumber-tag_expressions
Looking up gems ["powerbar", "coffee-script-source", "needle", "jruby-pageant", "bcrypt_pbkdf", "rbnacl", "rbnacl-libsodium", "pattern-match", "middleware", "virtus"]
HTTP GET https://index.rubygems.org/info/middleware
HTTP GET https://index.rubygems.org/info/virtus
HTTP 206 Partial Content https://index.rubygems.org/info/virtus
HTTP 200 OK https://index.rubygems.org/info/middleware
Looking up gems ["ansi", "descendants_tracker", "axiom-types", "coercible", "equalizer"]
Looking up gems ["adamantium", "ice_nine"]
Looking up gems ["memoizable"]
Fetching https://github.com/rails/sprockets

I, [2024-01-06T16:45:02.658395 #1]  INFO -- : Terminating async processes
I, [2024-01-06T16:45:02.658420 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 59
2024-01-06 16:45:02.658 UTC [59] LOG:  received fast shutdown request
2024-01-06 16:45:02.662 UTC [59] LOG:  aborting any active transactions
2024-01-06 16:45:02.664 UTC [59] LOG:  background worker "logical replication launcher" (PID 68) exited with exit code 1
2024-01-06 16:45:02.665 UTC [63] LOG:  shutting down
2024-01-06 16:45:02.701 UTC [59] LOG:  database system is shut down


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle install --deployment --retry 3 --jobs 4 --verbose --without test development' failed with return #<Process::Status: pid 339 exit 5>
Location of failure: /pups/lib/pups/exec_command.rb:112:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"bundle_exec", "cmd"=>["su discourse -c 'bundle install --deployment --retry 3 --jobs 4 --verbose --without test development'"]}
2c272af0e6076b670160aa14969f8864f1178ccce394e13b8dcfad9ae5ab3fa1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
==================== END REBUILD LOG ====================
Failed to rebuild app.

Checking your domain name . . .

Connection to discuss.eroscripts.com succeeded.
You should probably remove any non-standard plugins and rebuild.
Attempting to restart existing container. . . 

starting up existing container
+ /usr/bin/docker start app
app
Restarted the container.

It says I have non-official plugins, but I definitely deleted every line in the plugins section.

==================== PLUGINS ====================


WARNING:
You have what appear to be non-official plugins.
If you are having trouble, you should disable them and try rebuilding again.

See https://github.com/discourse/discourse/blob/master/lib/plugin/metadata.rb for the official list.

========================================
Discourse version at discuss.eroscripts.com: NOT FOUND
Discourse version at localhost: NOT FOUND


==================== MEMORY INFORMATION ====================
OS: Linux
RAM (MB): 16773

              total        used        free      shared  buff/cache   available
Mem:          15996         347        8980           0        6668       15310
Swap:          2047           2        2045

==================== DISK SPACE CHECK ====================
---------- OS Disk Space ----------
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       310G  117G  194G  38% /

==================== DISK INFORMATION ====================
Disk /dev/vda: 320 GiB, 343597383680 bytes, 671088640 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 3D5B5FC8-E964-4310-AC68-89DD2E51D6D4

Device      Start       End   Sectors   Size Type
/dev/vda1  227328 671088606 670861279 319.9G Linux filesystem
/dev/vda14   2048     10239      8192     4M BIOS boot
/dev/vda15  10240    227327    217088   106M Microsoft basic data

Partition table entries are not in disk order.

==================== END DISK INFORMATION ====================

It also sent an email deliverability test, which seems to have failed. The odd part is that the email test was at least working yesterday. Maybe because the rebuild wasn’t failing back them even though the site was still broken.

This is very suspicious - what kind of install is this? Have you followed the standard install?

The current version of Ruby on which a whole host of code relies is 3.2.2

Are you using the official discourse repo on branch tests-passed?

Something is very wrong with your installation here.

1 Like

This droplet’s been running for 2 years, and I think it was the standard install instructions. I was just running on the 2nd latest release prior to everything breaking. I didn’t edit the version in app.yml prior, so it should have defaulted to the tests-passed branch.

I’m currently attempting to set up a new droplet and restore my backup there. But I’m struggling to get anything to show up, maybe due to my app.yml settings for cdn on static assets or trying to use the ip instead of a url.

I’m going to try restoring in a new droplet. But do you have any advice on how to do so without immediately pointing the hostname to the new ip address? Ideally want to set up/restore the site without anyone being able to access the new admin creation page on first load of a new discourse instance. But when I try things like setting the hostname to the ip in app.yml, things aren’t working. (I think I’d done this before and it worked, but I understand that the instructions say you can’t)

My advice would be to bite the bullet and point it to the new location - should not take more than 30 mins which is nothing in the grand scheme of things and you are in a bit of a pickle here :slight_smile:

Do not try to pin Discourse to any specific tag, just rely on the normal updates.

Pick stable or tests-passed and stick with it …

this time i created a new droplet, did the minimum it takes to get ready for a restore, then followed these instructions to restore the backup I managed to create after my site went down even though it was still accessible via command line: Restore a backup from the command line

Now I’m running into the below error. Perhaps my database got corrupted somehow? :thinking:

I can try restoring an earlier backup

./launcher rebuild app
x86_64 arch detected.
WARNING: containers/app.yml file is world-readable. You can secure this file by running: chmod o-rwx containers/app.yml
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
Stopping old container
+ /usr/bin/docker stop -t 600 app
app
2.0.20231218-0429: Pulling from discourse/base
Digest: sha256:468f70b9bb4c6d0c6c2bbb3efc1a5e12d145eae57bdb6946b7fe5558beb52dc1
Status: Image is up to date for discourse/base:2.0.20231218-0429
docker.io/discourse/base:2.0.20231218-0429
/usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2024-01-06T21:09:54.354218 #1]  INFO -- : Reading from stdin
I, [2024-01-06T21:09:54.365267 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2024-01-06T21:09:54.421443 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2024-01-06T21:09:54.421795 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2024-01-06T21:09:54.427121 #1]  INFO -- : 
I, [2024-01-06T21:09:54.427537 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2024-01-06T21:09:54.434259 #1]  INFO -- : 
I, [2024-01-06T21:09:54.434795 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2024-01-06T21:09:54.441347 #1]  INFO -- : 
I, [2024-01-06T21:09:54.441696 #1]  INFO -- : > rm -fr /var/run/postgresql
I, [2024-01-06T21:09:54.448226 #1]  INFO -- : 
I, [2024-01-06T21:09:54.448601 #1]  INFO -- : > ln -s /shared/postgres_run /var/run/postgresql
I, [2024-01-06T21:09:54.454601 #1]  INFO -- : 
I, [2024-01-06T21:09:54.454965 #1]  INFO -- : > socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 && echo postgres already running stop container ; exit 1
2024/01/06 21:09:54 socat[19] E connect(6, AF=1 "/shared/postgres_run/.s.PGSQL.5432", 36): Connection refused
I, [2024-01-06T21:09:54.470043 #1]  INFO -- : 
I, [2024-01-06T21:09:54.470647 #1]  INFO -- : > rm -fr /shared/postgres_run/.s*
I, [2024-01-06T21:09:54.478233 #1]  INFO -- : 
I, [2024-01-06T21:09:54.478786 #1]  INFO -- : > rm -fr /shared/postgres_run/*.pid
I, [2024-01-06T21:09:54.486059 #1]  INFO -- : 
I, [2024-01-06T21:09:54.486533 #1]  INFO -- : > mkdir -p /shared/postgres_run/13-main.pg_stat_tmp
I, [2024-01-06T21:09:54.491565 #1]  INFO -- : 
I, [2024-01-06T21:09:54.491920 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run/13-main.pg_stat_tmp
I, [2024-01-06T21:09:54.496463 #1]  INFO -- : 
I, [2024-01-06T21:09:54.508217 #1]  INFO -- : File > /etc/service/postgres/run  chmod: +x  chown: 
I, [2024-01-06T21:09:54.519782 #1]  INFO -- : File > /etc/service/postgres/log/run  chmod: +x  chown: 
I, [2024-01-06T21:09:54.531602 #1]  INFO -- : File > /etc/runit/3.d/99-postgres  chmod: +x  chown: 
I, [2024-01-06T21:09:54.545224 #1]  INFO -- : File > /root/upgrade_postgres  chmod: +x  chown: 
I, [2024-01-06T21:09:54.545698 #1]  INFO -- : > chown -R root /var/lib/postgresql/13/main
I, [2024-01-06T21:09:57.594747 #1]  INFO -- : 
I, [2024-01-06T21:09:57.596324 #1]  INFO -- : > [ ! -e /shared/postgres_data ] && install -d -m 0755 -o postgres -g postgres /shared/postgres_data && sudo -E -u postgres /usr/lib/postgresql/13/bin/initdb -D /shared/postgres_data || exit 0
I, [2024-01-06T21:09:57.601398 #1]  INFO -- : 
I, [2024-01-06T21:09:57.601559 #1]  INFO -- : > chown -R postgres:postgres /shared/postgres_data
I, [2024-01-06T21:09:57.647669 #1]  INFO -- : 
I, [2024-01-06T21:09:57.648126 #1]  INFO -- : > chown -R postgres:postgres /var/run/postgresql
I, [2024-01-06T21:09:57.654847 #1]  INFO -- : 
I, [2024-01-06T21:09:57.655228 #1]  INFO -- : > /root/upgrade_postgres
I, [2024-01-06T21:09:57.671346 #1]  INFO -- : 
I, [2024-01-06T21:09:57.671840 #1]  INFO -- : > rm /root/upgrade_postgres
I, [2024-01-06T21:09:57.677190 #1]  INFO -- : 
I, [2024-01-06T21:09:57.678815 #1]  INFO -- : Replacing data_directory = '/var/lib/postgresql/13/main' with data_directory = '/shared/postgres_data' in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-06T21:09:57.680570 #1]  INFO -- : Replacing (?-mix:#?listen_addresses *=.*) with listen_addresses = '*' in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-06T21:09:57.681591 #1]  INFO -- : Replacing (?-mix:#?synchronous_commit *=.*) with synchronous_commit = $db_synchronous_commit in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-06T21:09:57.693950 #1]  INFO -- : Replacing (?-mix:#?shared_buffers *=.*) with shared_buffers = $db_shared_buffers in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-06T21:09:57.696644 #1]  INFO -- : Replacing (?-mix:#?work_mem *=.*) with work_mem = $db_work_mem in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-06T21:09:57.699678 #1]  INFO -- : Replacing (?-mix:#?default_text_search_config *=.*) with default_text_search_config = '$db_default_text_search_config' in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-06T21:09:57.706129 #1]  INFO -- : > install -d -m 0755 -o postgres -g postgres /shared/postgres_backup
I, [2024-01-06T21:09:57.716052 #1]  INFO -- : 
I, [2024-01-06T21:09:57.716636 #1]  INFO -- : Replacing (?-mix:#?checkpoint_segments *=.*) with checkpoint_segments = $db_checkpoint_segments in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-06T21:09:57.717255 #1]  INFO -- : Replacing (?-mix:#?logging_collector *=.*) with logging_collector = $db_logging_collector in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-06T21:09:57.721602 #1]  INFO -- : Replacing (?-mix:#?log_min_duration_statement *=.*) with log_min_duration_statement = $db_log_min_duration_statement in /etc/postgresql/13/main/postgresql.conf
I, [2024-01-06T21:09:57.727937 #1]  INFO -- : Replacing (?-mix:^#local +replication +postgres +peer$) with local replication postgres  peer in /etc/postgresql/13/main/pg_hba.conf
I, [2024-01-06T21:09:57.728569 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*127.*$) with host all all 0.0.0.0/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2024-01-06T21:09:57.732277 #1]  INFO -- : Replacing (?-mix:^host.*all.*all.*::1\/128.*$) with host all all ::/0 md5 in /etc/postgresql/13/main/pg_hba.conf
I, [2024-01-06T21:09:57.733290 #1]  INFO -- : > HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main
I, [2024-01-06T21:09:57.738665 #1]  INFO -- : > sleep 5
2024-01-06 21:09:57.897 UTC [42] LOG:  starting PostgreSQL 13.13 (Debian 13.13-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2024-01-06 21:09:57.897 UTC [42] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-01-06 21:09:57.897 UTC [42] LOG:  listening on IPv6 address "::", port 5432
2024-01-06 21:09:57.904 UTC [42] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-01-06 21:09:57.914 UTC [45] LOG:  database system was interrupted; last known up at 2024-01-06 21:06:38 UTC
2024-01-06 21:09:58.287 UTC [45] LOG:  database system was not properly shut down; automatic recovery in progress
2024-01-06 21:09:58.300 UTC [45] LOG:  redo starts at 6/3F42F228
I, [2024-01-06T21:10:02.744704 #1]  INFO -- : 
I, [2024-01-06T21:10:02.745102 #1]  INFO -- : > su postgres -c 'createdb discourse' || true
2024-01-06 21:10:02.852 UTC [49] postgres@postgres FATAL:  the database system is starting up
2024-01-06 21:10:02.857 UTC [50] postgres@template1 FATAL:  the database system is starting up
createdb: error: could not connect to database template1: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2024-01-06T21:10:02.865413 #1]  INFO -- : 
I, [2024-01-06T21:10:02.867743 #1]  INFO -- : > su postgres -c 'psql discourse -c "create user discourse;"' || true
2024-01-06 21:10:02.984 UTC [54] postgres@discourse FATAL:  the database system is starting up
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2024-01-06T21:10:02.989529 #1]  INFO -- : 
I, [2024-01-06T21:10:02.989918 #1]  INFO -- : > su postgres -c 'psql discourse -c "grant all privileges on database discourse to discourse;"' || true
2024-01-06 21:10:03.100 UTC [58] postgres@discourse FATAL:  the database system is starting up
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2024-01-06T21:10:03.107339 #1]  INFO -- : 
I, [2024-01-06T21:10:03.107772 #1]  INFO -- : > su postgres -c 'psql discourse -c "alter schema public owner to discourse;"'
2024-01-06 21:10:03.200 UTC [62] postgres@discourse FATAL:  the database system is starting up
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
I, [2024-01-06T21:10:03.222053 #1]  INFO -- : 
I, [2024-01-06T21:10:03.222926 #1]  INFO -- : Terminating async processes
I, [2024-01-06T21:10:03.223018 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 42
2024-01-06 21:10:03.238 UTC [42] LOG:  received fast shutdown request
2024-01-06 21:10:03.362 UTC [65] LOG:  shutting down
2024-01-06 21:10:03.602 UTC [42] LOG:  database system is shut down


FAILED
--------------------
Pups::ExecError: su postgres -c 'psql discourse -c "alter schema public owner to discourse;"' failed with return #<Process::Status: pid 59 exit 2>
Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params "su postgres -c 'psql $db_name -c \"alter schema public owner to $db_user;\"'"
bootstrap failed with exit code 2
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
a74fe31949946fbbd088418657abc63cca8f2d8ecda9d2d200979f275df55711

Your can do Move a Discourse site to another VPS with rsync. That’ll copy the certs as well as the database.

Weird, this is where it starts to go wrong, everything before this appears normal.

It should I believe look like:

2024-01-06 11:20:06.423 UTC [44] LOG:  database system was shut down at 2024-01-06 10:47:08 UTC
2024-01-06 11:20:06.467 UTC [41] LOG:  database system is ready to accept connections

How about simply creating a basic Discourse without the backup, then restoring the backup online?

Back to:

database system was not properly shut down; automatic recovery in progress

A quick google does indeed suggest this is a symptom of the database having been corrupted.

Yes, try an earlier backup!

okay maybe making progress.

I’m getting a 502 bad gateway page now. If I run ./launcher logs app I see this


nginx: [emerg] cannot load certificate "/shared/ssl/discuss.eroscripts.com.cer": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/discourse.conf:60
nginx: [emerg] cannot load certificate "/shared/ssl/discuss.eroscripts.com.cer": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
Shutting Down
run-parts: executing /etc/runit/3.d/01-nginx
ok: down: nginx: 0s, normally up, want up
run-parts: executing /etc/runit/3.d/02-unicorn
(5248) exiting
ok: down: unicorn: 1s, normally up
run-parts: executing /etc/runit/3.d/10-redis
ok: down: redis: 0s, normally up
run-parts: executing /etc/runit/3.d/99-postgres
timeout: run: postgres: (pid 5242) 2470s, want down, got TERM
run-parts: /etc/runit/3.d/99-postgres exited with return code 1
ok: down: nginx: 9s, normally up
ok: down: redis: 8s, normally up
ok: down: unicorn: 9s, normally up
ok: down: cron: 1s, normally up
ok: down: rsyslog: 1s, normally up
kill: run: postgres: (pid 5242) 2478s, want down, got TERM
run-parts: executing /etc/runit/1.d/00-ensure-links
run-parts: executing /etc/runit/1.d/00-fix-var-logs
run-parts: executing /etc/runit/1.d/01-cleanup-web-pids
run-parts: executing /etc/runit/1.d/anacron
run-parts: executing /etc/runit/1.d/cleanup-pids
Cleaning stale PID files
run-parts: executing /etc/runit/1.d/copy-env
run-parts: executing /etc/runit/1.d/letsencrypt
[Sat 06 Jan 2024 10:44:40 PM UTC] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Sat 06 Jan 2024 10:44:40 PM UTC] Single domain='discuss.eroscripts.com'
[Sat 06 Jan 2024 10:44:41 PM UTC] Getting domain auth token for each domain
[Sat 06 Jan 2024 10:44:43 PM UTC] Getting webroot for domain='discuss.eroscripts.com'
[Sat 06 Jan 2024 10:44:43 PM UTC] Verifying: discuss.eroscripts.com
[Sat 06 Jan 2024 10:44:44 PM UTC] Pending, The CA is processing your order, please just wait. (1/30)
[Sat 06 Jan 2024 10:44:48 PM UTC] discuss.eroscripts.com:Verify error:159.223.125.223: Invalid response from https://discuss.eroscripts.com/.well-known/acme-challenge/ygqIckpuhRNDWk1xFI_LdPhAHKd3WIncOluj_7oQusc: 502
[Sat 06 Jan 2024 10:44:48 PM UTC] Please check log file for more details: /shared/letsencrypt/acme.sh.log
Can't open ca.cer for reading, No such file or directory
139736654603584:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:69:fopen('ca.cer','r')
139736654603584:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:76:
unable to load certificate
Error loading file /dev/fd/63
[Sat 06 Jan 2024 10:44:49 PM UTC] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Sat 06 Jan 2024 10:44:50 PM UTC] Single domain='discuss.eroscripts.com'
[Sat 06 Jan 2024 10:44:50 PM UTC] Getting domain auth token for each domain
[Sat 06 Jan 2024 10:44:51 PM UTC] Create new order error. Le_OrderFinalize not found. {
  "type": "urn:ietf:params:acme:error:rateLimited",
  "detail": "Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/",
  "status": 429
}
[Sat 06 Jan 2024 10:44:51 PM UTC] Please check log file for more details: /shared/letsencrypt/acme.sh.log
[Sat 06 Jan 2024 10:44:52 PM UTC] Installing key to: /shared/ssl/discuss.eroscripts.com.key
[Sat 06 Jan 2024 10:44:52 PM UTC] Installing full chain to: /shared/ssl/discuss.eroscripts.com.cer
cat: /shared/letsencrypt/discuss.eroscripts.com/fullchain.cer: No such file or directory
[Sat 06 Jan 2024 10:44:53 PM UTC] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Sat 06 Jan 2024 10:44:53 PM UTC] Single domain='discuss.eroscripts.com'
[Sat 06 Jan 2024 10:44:54 PM UTC] Getting domain auth token for each domain
[Sat 06 Jan 2024 10:44:55 PM UTC] Create new order error. Le_OrderFinalize not found. {
  "type": "urn:ietf:params:acme:error:rateLimited",
  "detail": "Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/",
  "status": 429
}
[Sat 06 Jan 2024 10:44:55 PM UTC] Please check log file for more details: /shared/letsencrypt/acme.sh.log
Can't open ca.cer for reading, No such file or directory
140139135829312:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:69:fopen('ca.cer','r')
140139135829312:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:76:
unable to load certificate
Error loading file /dev/fd/63
[Sat 06 Jan 2024 10:44:56 PM UTC] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Sat 06 Jan 2024 10:44:56 PM UTC] Single domain='discuss.eroscripts.com'
[Sat 06 Jan 2024 10:44:57 PM UTC] Getting domain auth token for each domain
[Sat 06 Jan 2024 10:44:58 PM UTC] Create new order error. Le_OrderFinalize not found. {
  "type": "urn:ietf:params:acme:error:rateLimited",
  "detail": "Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/",
  "status": 429
}
[Sat 06 Jan 2024 10:44:58 PM UTC] Please check log file for more details: /shared/letsencrypt/acme.sh.log
[Sat 06 Jan 2024 10:44:58 PM UTC] Installing key to: /shared/ssl/discuss.eroscripts.com_ecc.key
[Sat 06 Jan 2024 10:44:58 PM UTC] Installing full chain to: /shared/ssl/discuss.eroscripts.com_ecc.cer
cat: /shared/letsencrypt/discuss.eroscripts.com_ecc/fullchain.cer: No such file or directory
Can't open ca.cer for reading, No such file or directory
140163653252416:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:69:fopen('ca.cer','r')

and copying the ones that are red

[Sat 06 Jan 2024 10:44:48 PM UTC] discuss.eroscripts.com:Verify error:159.223.125.223: Invalid response from https://discuss.eroscripts.com/.well-known/acme-challenge/ygqIckpuhRNDWk1xFI_LdPhAHKd3WIncOluj_7oQusc: 502
[Sat 06 Jan 2024 10:44:48 PM UTC] Please check log file for more details: /shared/letsencrypt/acme.sh.log

[Sat 06 Jan 2024 10:44:51 PM UTC] Create new order error. Le_OrderFinalize not found. {
  "type": "urn:ietf:params:acme:error:rateLimited",
  "detail": "Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/",
  "status": 429
}

HI,
I also admin the same server and here’s what I’ve done so far.
I have upgraded the host from focal to jammy, removed all the images and re-downloaded new ones.
I discovered we were on the master branch so I changed to main branch.
made sure I was on latest commit.

It now builds but we still have an error 500 issue.
I looked at the unicorn logs and found it spamming:

E, [2024-01-07T04:58:49.140147 #682] ERROR -- : app error: Missing `secret_key_base` for 'production' environment, set this string with `bin/rails credentials:edit` (ArgumentError)
E, [2024-01-07T04:58:49.140266 #682] ERROR -- : /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/application.rb:576:in `validate_secret_key_base'
E, [2024-01-07T04:58:49.140278 #682] ERROR -- : /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/application.rb:419:in `secret_key_base'
E, [2024-01-07T04:58:49.140287 #682] ERROR -- : /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/application.rb:254:in `env_config'
E, [2024-01-07T04:58:49.140295 #682] ERROR -- : /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/engine.rb:716:in `build_request'
E, [2024-01-07T04:58:49.140304 #682] ERROR -- : /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/application.rb:598:in `build_request'
E, [2024-01-07T04:58:49.140311 #682] ERROR -- : /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/engine.rb:529:in `call'
E, [2024-01-07T04:58:49.140319 #682] ERROR -- : /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/railtie.rb:226:in `public_send'
E, [2024-01-07T04:58:49.140327 #682] ERROR -- : /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8/lib/rails/railtie.rb:226:in `method_missing'
E, [2024-01-07T04:58:49.140336 #682] ERROR -- : /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-2.2.8/lib/rack/urlmap.rb:74:in `block in call'
E, [2024-01-07T04:58:49.140344 #682] ERROR -- : /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-2.2.8/lib/rack/urlmap.rb:58:in `each'
E, [2024-01-07T04:58:49.140353 #682] ERROR -- : /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-2.2.8/lib/rack/urlmap.rb:58:in `call'
E, [2024-01-07T04:58:49.140368 #682] ERROR -- : /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:634:in `process_client'
E, [2024-01-07T04:58:49.140378 #682] ERROR -- : /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:739:in `worker_loop'
E, [2024-01-07T04:58:49.140386 #682] ERROR -- : /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:547:in `spawn_missing_workers'
E, [2024-01-07T04:58:49.140409 #682] ERROR -- : /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:143:in `start'
E, [2024-01-07T04:58:49.140417 #682] ERROR -- : /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/bin/unicorn:128:in `<top (required)>'
E, [2024-01-07T04:58:49.140426 #682] ERROR -- : /var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `load'
E, [2024-01-07T04:58:49.140434 #682] ERROR -- : /var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `<main>'

This is currently my only indication of what is wrong.

We have fixed this.
Please close or lock the thread. we ultimately moved to a new server and restored from backup.

the short story is: we ran into a large variety of seemingly random issues that popped up in the original server. Migrating to a new droplet and restoring seems to have fixed it all. We still have no clue why the original issue occurred in the first place.

Some more details on the issues we ran into and tried on the original server

  • The site could not communicate to postgres and redis because the owner of the folder/files didn’t exist. They had mismatched IDs from a previous docker image.
  • Needed to update the host OS to have the correct version of Ruby and master branch to the main branch. Changed the package repos from focal to jammy and ran the update, this would ultimately not fix the dependancy issues. but it did fix a failing build of the image.
  • Unicorn webserver was throwing an error about a secret_base_key that was missing. We don’t handle any sensitive tokens in the webserver so no idea why this was being thrown

We had some struggle with DNS TTL being too long, making it hard to test out fixes, but that’s on us.

However we did run into some weird errors with our CDN that eventually got resolved

  • We didn’t change any settings. we purged the cache to re-populate the cache to the backup version. For a while, it was throwing a ton of 500 errors, but seemed to resolve over time. We asked their tech support but they didn’t find any issue on their side. Discourse seemed to be serving files with content-length: 0, or at least that’s the only error we looked into. But this seemed to resolve itself over time for some reason.

At the end of the day, we truly have no clue what happened in the original server that caused all those issues.

1 Like

Our emails seem to be down (we use amazon SES). Have not investigated what’s wrong with it yet.

After a restore the disable emails setting is automatically toggled on. Could that be the issue? If so, you can set it back to ‘no’ again.

1 Like

I shouldn’t need to make any changes in AWS, DNS, app.yml, or settings after a restore right?

in settings, emails were not toggled on after restoring. It still says ‘disable emails: no’.

aws SES shows 0 email sends since the restore.

/sidekiq shows 44,934 Processed 1,470 Failed, but not sure how to find more info about the failed jobs.

/logs fatal tab (please excuse the nsfw titles)


ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "posts" does not exist LINE 1: SELECT COUNT(*) FROM "posts" WHERE "posts"."deleted_at" IS N... ^ ) lib
Su 10:15 am
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "topics" does not exist LINE 1: ...cs"."bannered_until", "topics"."external_id" FROM "topics" W...
Su 10:15 am
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "topics" does not exist LINE 1: ...cs"."bannered_until", "topics"."external_id" FROM "topics" W...
Su 10:16 am
4
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "users" does not exist LINE 1: SELECT "users".* FROM "users" WHERE "users"."username_lower"... ^ ) ap
Su 10:17 am
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "categories" does not exist LINE 1: ..., "categories"."uploaded_background_dark_id" FROM "categorie...
Su 10:17 am
2
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "categories" does not exist LINE 1: ..., "categories"."uploaded_background_dark_id" FROM "categorie...
Su 10:17 am
2
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "categories" does not exist LINE 1: SELECT "categories"."id" FROM "categories" WHERE "categories...
Su 10:18 am
4
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "user_auth_tokens" does not exist LINE 1: SELECT "user_auth_tokens".* FROM "user_auth_tokens" WHERE ((...
Su 10:18 am
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "user_auth_tokens" does not exist LINE 1: SELECT "user_auth_tokens".* FROM "user_auth_tokens" WHERE ((...
Su 10:29 am
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "posts" does not exist LINE 1: ...ge_upload_id", "posts"."outbound_message_id" FROM "posts" WH...
Su 10:29 am
2
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "sitemaps" does not exist LINE 1: SELECT "sitemaps".* FROM "sitemaps" WHERE "sitemaps"."name" ...
Su 10:32 am
74
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "user_auth_tokens" does not exist LINE 1: SELECT "user_auth_tokens".* FROM "user_auth_tokens" WHERE ((...
Su 10:33 am
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "themes" does not exist LINE 1: SELECT "themes"."color_scheme_id" FROM "themes" WHERE "theme...
Su 10:33 am
3
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "tags" does not exist LINE 1: SELECT "tags"."name" FROM "tags" WHERE "tags"."id" NOT IN (S... ^ )
Su 10:33 am
8
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "categories" does not exist LINE 1: SELECT "categories"."id" FROM "categories" LEFT OUTER JOIN "...
Su 10:33 am
15
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "user_auth_tokens" does not exist LINE 1: SELECT "user_auth_tokens".* FROM "user_auth_tokens" WHERE ((...
Su 10:34 am
29
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "categories" does not exist LINE 1: SELECT "categories"."id" FROM "categories" WHERE "categories...
Su 10:34 am
3
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "users" does not exist LINE 1: SELECT "users".* FROM "users" INNER JOIN "user_emails" ON "u... ^ ) ap
Su 10:34 am
9
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "stylesheet_cache" does not exist LINE 1: SELECT "stylesheet_cache"."created_at" FROM "stylesheet_cach...
Su 10:35 am
5
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "categories" does not exist LINE 1: SELECT "categories"."id" FROM "categories" WHERE "categories...
Su 10:35 am
39
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "permalinks" does not exist LINE 1: SELECT 1 AS one FROM "permalinks" WHERE "permalinks"."url" =... ^
Su 10:35 am
50
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "user_auth_tokens" does not exist LINE 1: SELECT "user_auth_tokens".* FROM "user_auth_tokens" WHERE ((...
Su 10:35 am
457
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "categories" does not exist LINE 1: SELECT "categories"."id" FROM "categories" WHERE "categories...
Su 10:35 am
30
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "permalinks" does not exist LINE 1: SELECT 1 AS one FROM "permalinks" WHERE "permalinks"."url" =... ^
Su 10:35 am
1189
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "user_auth_tokens" does not exist LINE 1: SELECT "user_auth_tokens".* FROM "user_auth_tokens" WHERE ((...
Su 10:35 am
553
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "categories" does not exist LINE 1: SELECT "categories"."id" FROM "categories" WHERE "categories...
Su 10:35 am
36
ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "user_auth_tokens" does not exist LINE 1: SELECT "user_auth_tokens".* FROM "user_auth_tokens" WHERE ((... 

/logs errors tab

Failed to warm up pretty text: PG::UndefinedTable: ERROR: relation "custom_emojis" does not exist LINE 1: SELECT "custom_emojis".* FROM "custom_emojis" ORDER BY "cust...
Su 10:27 am
2
Job exception: could not obtain a connection from the pool within 5.000 seconds (waited 5.036 seconds); all pooled connections were in use
Su 10:53 am
default: Failed to pull hotlinked image (https://discourse-s3-cdn.eroscripts.com/uploads/optimized/3X/3/3/33050afb6576555544a856d5c6f19583bfc854be_2_690x362.png) post: 325176 PG::UniqueViolation: ERRO
Su 5:33 pm
default: Failed to pull hotlinked image (https://rule34video.com/contents/videos_screenshots/3125000/3125485/preview.jpg) post: 338794 PG::UniqueViolation: ERROR: duplicate key value violates unique
1:04 am
2
default: Failed to pull hotlinked image (https://rule34video.com/contents/videos_screenshots/3125000/3125482/preview_1080p.mp4.jpg) post: 338794 PG::UniqueViolation: ERROR: duplicate key value violat 

/logs warning tab

 Load more
553
Failed to handle exception in exception app middleware : ActiveRecord::StatementInvalid : PG::UndefinedTable: ERROR: relation "categories" does not exist LINE 1: SELECT "categories"."id" FROM "catego
Su 10:35 am
36
Failed to handle exception in exception app middleware : ActiveRecord::StatementInvalid : PG::UndefinedTable: ERROR: relation "user_auth_tokens" does not exist LINE 1: SELECT "user_auth_tokens".* FRO
Su 10:36 am
645
Unexpected error in Message Bus : ActiveRecord::StatementInvalid : PG::UndefinedTable: ERROR: relation "user_auth_tokens" does not exist LINE 1: SELECT "user_auth_tokens".* FROM "user_auth_tokens" WH
Su 10:36 am
2
Topic List Previews theme/component is throwing errors: TypeError: e.capitalize is not a function. (In 'e.capitalize()', 'e.capitalize' is undefined)
Su 10:36 am
Patreon token refresh failed with status: 401. {"error": "invalid_grant"}
Su 10:41 am
4
Topic List Previews theme/component is throwing errors: TypeError: e.capitalize is not a function
Su 10:41 am
10
Topic List Previews theme/component is throwing errors: TypeError: e.capitalize is not a function
Su 10:41 am
[DiscourseJsProcessor] DEPRECATION: Invoking the `<LinkTo>` component with positional arguments is deprecated.Please use the equivalent named arguments (`@route`). ('discourse/theme-10/discourse/templ
Su 10:46 am
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
Su 10:51 am
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
Su 10:53 am
Deprecation notice: `SiteSetting.min_trust_to_edit_post` has been deprecated. Please use `SiteSetting.edit_post_allowed_groups` instead. (removal in Discourse 3.3) At /var/www/discourse/lib/guardian/
Su 10:58 am
2
Deprecation notice: `SiteSetting.min_trust_level_to_allow_ignore` has been deprecated. Please use `SiteSetting.ignore_allowed_groups` instead. (removal in Discourse 3.3) At /var/www/discourse/lib/gua
Su 1:33 pm
Deprecation notice: `SiteSetting.min_trust_to_create_tag` has been deprecated. Please use `SiteSetting.create_tag_allowed_groups` instead. (removal in Discourse 3.3) At /var/www/discourse/lib/guardia
Su 3:01 pm
Deprecation notice: `SiteSetting.min_trust_level_to_allow_ignore` has been deprecated. Please use `SiteSetting.ignore_allowed_groups` instead. (removal in Discourse 3.3) At /var/www/discourse/lib/gua
Su 3:16 pm
10
MaxMindDB (/var/www/discourse/vendor/data/GeoLite2-City.mmdb) could not be found: No such file or directory @ rb_sysopen - /var/www/discourse/vendor/data/GeoLite2-City.mmdb
Su 4:39 pm
10
MaxMindDB (/var/www/discourse/vendor/data/GeoLite2-ASN.mmdb) could not be found: No such file or directory @ rb_sysopen - /var/www/discourse/vendor/data/GeoLite2-ASN.mmdb
Su 4:39 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
Su 4:40 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;CreativeWork&quot;, &quot;name&quot;: &quot;Abella Anderson shows her perfect T--- while ridin
Su 4:42 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
Su 4:50 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
Su 5:04 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
Su 5:06 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
Su 5:06 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
Su 5:06 pm
MaxMindDB (/var/www/discourse/vendor/data/GeoLite2-City.mmdb) could not be found: No such file or directory @ rb_sysopen - /var/www/discourse/vendor/data/GeoLite2-City.mmdb
Su 5:06 pm
MaxMindDB (/var/www/discourse/vendor/data/GeoLite2-ASN.mmdb) could not be found: No such file or directory @ rb_sysopen - /var/www/discourse/vendor/data/GeoLite2-ASN.mmdb
Su 5:06 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
Su 5:10 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
Su 5:11 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
Su 5:33 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
Su 5:35 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
Su 6:05 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
Su 6:15 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
Su 6:20 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
Su 6:20 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
Su 6:20 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
Su 6:20 pm
3
MaxMindDB (/var/www/discourse/vendor/data/GeoLite2-City.mmdb) could not be found: No such file or directory @ rb_sysopen - /var/www/discourse/vendor/data/GeoLite2-City.mmdb
Su 6:32 pm
3
MaxMindDB (/var/www/discourse/vendor/data/GeoLite2-ASN.mmdb) could not be found: No such file or directory @ rb_sysopen - /var/www/discourse/vendor/data/GeoLite2-ASN.mmdb
Su 6:32 pm
2
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
Su 6:43 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
Su 6:50 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
Su 6:50 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
Su 7:05 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
Su 7:05 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
Su 7:19 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
Su 7:20 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
Su 7:20 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
Su 7:35 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
Su 7:35 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
Su 7:35 pm
Failed to process hijacked response correctly : NoMethodError : undefined method `days_visited' for nil:NilClass
Su 8:27 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
Su 8:51 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
Su 9:36 pm
2
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;CreativeWork&quot;, &quot;name&quot;: &quot;PHONKY HERO&quot;, &quot;description&quot;: &quot
Su 10:21 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
Su 10:41 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
Su 10:51 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
Su 10:51 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
Su 10:51 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
Su 11:07 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;CreativeWork&quot;, &quot;name&quot;: &quot;【HMV】mizumizuni&quot;, &quot;description&quot;: &
Su 11:07 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;CreativeWork&quot;, &quot;name&quot;: &quot;(TIER 1 EARLY-ACCESS) SK3L3T0K. DARK ELF LINK&quot
Su 11:22 pm
2
Deprecation notice: `SiteSetting.min_trust_level_to_allow_ignore` has been deprecated. Please use `SiteSetting.ignore_allowed_groups` instead. (removal in Discourse 3.3) At /var/www/discourse/lib/gua
Su 11:32 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;CreativeWork&quot;, &quot;name&quot;: &quot;Dezyred - Sensual Blowjob With Sybil (Bonus Scene)
Su 11:36 pm
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;CreativeWork&quot;, &quot;name&quot;: &quot;Dezyred - Sensual Blowjob With Sybil (Bonus Scene)
Su 11:36 pm
2
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
12:52 am
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
1:00 am
4
Deprecation notice: `SiteSetting.min_trust_to_create_tag` has been deprecated. Please use `SiteSetting.create_tag_allowed_groups` instead. (removal in Discourse 3.3) At /var/www/discourse/lib/guardia
1:02 am
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
1:05 am
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
1:21 am
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
1:31 am
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
1:51 am
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
2:20 am
4
Deprecation notice: `SiteSetting.min_trust_level_to_allow_ignore` has been deprecated. Please use `SiteSetting.ignore_allowed_groups` instead. (removal in Discourse 3.3) At /var/www/discourse/lib/gua
2:33 am
5
Deprecation notice: `SiteSetting.min_trust_to_edit_post` has been deprecated. Please use `SiteSetting.edit_post_allowed_groups` instead. (removal in Discourse 3.3) At /var/www/discourse/lib/guardian/
3:26 am
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
3:34 am
4
Deprecation notice: `SiteSetting.min_trust_to_edit_post` has been deprecated. Please use `SiteSetting.edit_post_allowed_groups` instead. (removal in Discourse 3.3) At /var/www/discourse/app/models/co
3:53 am
6
Deprecation notice: `SiteSetting.min_trust_level_to_allow_ignore` has been deprecated. Please use `SiteSetting.ignore_allowed_groups` instead. (removal in Discourse 3.3) At /var/www/discourse/lib/gua
4:33 am
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
4:34 am
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
5:18 am
2
Deprecation notice: `SiteSetting.min_trust_to_edit_post` has been deprecated. Please use `SiteSetting.edit_post_allowed_groups` instead. (removal in Discourse 3.3) At /var/www/discourse/app/models/co
5:53 am
9
Sidekiq is consuming too much memory (using: 523.61M) for 'discuss.eroscripts.com', restarting
5:59 am
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ImageObject&quot;, &quot;contentUrl&quot;: &quot;https://c10.patreonusercontent.com/4/patreon-
6:24 am
14
Deprecation notice: `SiteSetting.min_trust_to_edit_post` has been deprecated. Please use `SiteSetting.edit_post_allowed_groups` instead. (removal in Discourse 3.3) At /var/www/discourse/lib/guardian/
6:26 am
4
Deprecation notice: `SiteSetting.min_trust_level_to_allow_ignore` has been deprecated. Please use `SiteSetting.ignore_allowed_groups` instead. (removal in Discourse 3.3) At /var/www/discourse/lib/gua
6:33 am
2
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;CreativeWork&quot;, &quot;name&quot;: &quot;Recovery - Golden Hands&quot;, &quot;description&
6:34 am
10
Deprecation notice: `SiteSetting.min_trust_to_edit_post` has been deprecated. Please use `SiteSetting.edit_post_allowed_groups` instead. (removal in Discourse 3.3) At /var/www/discourse/app/models/co
6:56 am
5
Deprecation notice: `SiteSetting.min_trust_to_flag_posts` has been deprecated. Please use `SiteSetting.flag_post_allowed_groups` instead. (removal in Discourse 3.3) At /var/www/discourse/lib/site_set
7:01 am
5
Deprecation notice: `SiteSetting.min_trust_level_to_allow_user_card_background` has been deprecated. Please use `SiteSetting.user_card_background_allowed_groups` instead. (removal in Discourse 3.3) A
7:01 am
5
Deprecation notice: `SiteSetting.min_trust_level_to_allow_ignore` has been deprecated. Please use `SiteSetting.ignore_allowed_groups` instead. (removal in Discourse 3.3) At /var/www/discourse/lib/sit
7:01 am
5
Deprecation notice: `SiteSetting.min_trust_to_create_tag` has been deprecated. Please use `SiteSetting.create_tag_allowed_groups` instead. (removal in Discourse 3.3) At /var/www/discourse/lib/site_se
7:01 am
16
Deprecation notice: `SiteSetting.min_trust_to_create_tag` has been deprecated. Please use `SiteSetting.create_tag_allowed_groups` instead. (removal in Discourse 3.3) At /var/www/discourse/lib/guardia
7:02 am
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;CreativeWork&quot;, &quot;name&quot;: &quot;Dezyred - Sensual Blowjob With Sybil (Bonus Scene)
7:03 am
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;CreativeWork&quot;, &quot;name&quot;: &quot;Dezyred - Sensual Blowjob With Sybil (Bonus Scene)
7:03 am
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;CreativeWork&quot;, &quot;name&quot;: &quot;Dezyred - A Day With Bella Rolland - Netflix and C
7:03 am
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;ProfilePage&quot;, &quot;author&quot;: { &quot;@type&quot;: &quot;Person&quot;, &quot;name
7:03 am
2
Error parsing JSON-LD: { &quot;@context&quot;: &quot;http://schema.org&quot;, &quot;@type&quot;: &quot;CreativeWork&quot;, &quot;name&quot;: &quot;VRConk - Jessica Rabbit - Vanna Bardot&quot;, &qu 

Ah okay the problem was that the app.yml was updated due to discourse-setup or something. The lines for

  • DISCOURSE_SMTP_DOMAIN:
  • DISCOURSE_NOTIFICATION_EMAIL:

were uncommented, and had the default placeholder of discourse.example.com. I guess we set up email before these were added. Filling these fields makes email work again.

1 Like

When migrating you don’t usually use discourse-setup, you would normally copy over your app.yml and run rebuild directly.

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.