At first, the upgrade screen had everything disabled. I rebooted and then I was able to select the docker_manager upgrade. That failed with:
rake aborted!
NoMethodError: undefined method `path' for nil:NilClass
So, then I opted to ssh into the server and tried a rebuild and here is what I got:
root@Discourse-1:/var/discourse# ./launcher rebuild app
x86_64 arch detected.
WARNING: containers/app.yml file is world-readable. You can secure this file by running: chmod o-rwx containers/app.yml
Ensuring launcher is up to date
Fetching origin
Launcher is up-to-date
............
Downloading MaxMindDB...
Compressing Javascript and Generating Source Maps
I, [2023-03-10T03:08:58.958320 #1] INFO -- : Terminating async processes
I, [2023-03-10T03:08:58.958649 #1] INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/13/bin/postmaster -D /etc/postgresql/13/main pid: 42
I, [2023-03-10T03:08:58.958745 #1] INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 103
103:signal-handler (1678417738) Received SIGTERM scheduling shutdown...
2023-03-10 03:08:58.959 UTC [42] LOG: received fast shutdown request
2023-03-10 03:08:58.966 UTC [42] LOG: aborting any active transactions
2023-03-10 03:08:58.969 UTC [42] LOG: background worker "logical replication launcher" (PID 51) exited with exit code 1
2023-03-10 03:08:58.971 UTC [46] LOG: shutting down
103:M 10 Mar 2023 03:08:59.047 # User requested shutdown...
103:M 10 Mar 2023 03:08:59.047 * Saving the final RDB snapshot before exiting.
2023-03-10 03:08:59.060 UTC [42] LOG: database system is shut down
103:M 10 Mar 2023 03:08:59.073 * DB saved on disk
103:M 10 Mar 2023 03:08:59.073 # Redis is now ready to exit, bye bye...
FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake themes:update assets:precompile' failed with return #<Process::Status: pid 1004 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"assets_precompile", "cmd"=>["su discourse -c 'bundle exec rake themes:update assets:precompile'"]}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
e956a73644feb880e4b524bcbd4fa3c626fcfcc4edb3752f72e523194f663b4e
root@Discourse-1:/var/discourse# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
root@Discourse-1:/var/discourse#
Do you have any plugins in your app.yml ?
When this kind of thing happens for me it’s usually a broken plugin that has to be commented out before the rebuild works.
Personally, if a backup failed, I would not try an update. A backup - including having a copy downloaded - is for me a prerequisite. This may not be a helpful comment at this time, but I think it has to be said.
If the problem is a failure to spawn, then not only do we care about RAM and swap usage, but we care about the kernel configuration. If this appears in the logs
# WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
My installation of Discourse is very generic. After a reboot I got a clean backup. So, ultimately I decided to do a from scratch installation. The installation failed with the same errors I posted above. I have been running Discourse with 6GB of memory in a LXD container the past four years with no issues. The memory utilization is steady at 1.9GB. During an upgrade, the memory peaks out at 5.1GB. My database backup is only 30GB of data and so my installation is small as compared with others. My configuration is running on Ubuntu 22.04. Should I be on another distro?
I have maybe 40 users and 150 posts. The system uses 1.8GB of memory and I gave it 6GB of memory. How much memory is required to have a rebuild work? That’s always been adequate in the past.
## this is the all-in-one, standalone Discourse Docker container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild app
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed
templates:
- "templates/postgres.template.yml"
- "templates/redis.template.yml"
- "templates/web.template.yml"
- "templates/web.ratelimited.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
- "templates/web.ssl.template.yml"
- "templates/web.letsencrypt.ssl.template.yml"
## which TCP/IP ports should this container expose?
## If you want Discourse to share a port with another webserver like Apache or nginx,
## see https://meta.discourse.org/t/17247 for details
expose:
- "80:80" # http
- "443:443" # https
params:
db_default_text_search_config: "pg_catalog.english"
## Set db_shared_buffers to a max of 25% of the total memory.
## will be set automatically by bootstrap based on detected RAM, or you can override
db_shared_buffers: "4096MB"
## can improve sorting performance, but adds memory usage per-connection
#db_work_mem: "40MB"
## Which Git revision should this container use? (default: tests-passed)
#version: tests-passed
env:
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LANGUAGE: en_US.UTF-8
# DISCOURSE_DEFAULT_LOCALE: en
## How many concurrent web requests are supported? Depends on memory and CPU cores.
## will be set automatically by bootstrap based on detected CPUs, or you can override
UNICORN_WORKERS: 8
## TODO: The domain name this Discourse instance will respond to
## Required. Discourse will not work with a bare IP number.
DISCOURSE_HOSTNAME: discussion.scottibyte.com
## Uncomment if you want the container to be started with the same
## hostname (-h option) as specified above (default "$hostname-$config")
#DOCKER_USE_HOSTNAME: true
## TODO: List of comma delimited emails that will be made admin and developer
## on initial signup example 'user1@example.com,user2@example.com'
DISCOURSE_DEVELOPER_EMAILS: 'aaaaaa@gmail.com'
## TODO: The SMTP mail server used to validate new accounts and send notifications
# SMTP ADDRESS, username, and password are required
# WARNING the char '#' in SMTP password can cause problems!
DISCOURSE_SMTP_ADDRESS: smtp.gmail.com
DISCOURSE_SMTP_PORT: 587
DISCOURSE_SMTP_USER_NAME: redacted
DISCOURSE_SMTP_PASSWORD: redacted
DISCOURSE_SMTP_ENABLE_START_TLS: true # (optional, default true)
DISCOURSE_SMTP_DOMAIN: discussion.scottibyte.com
DISCOURSE_NOTIFICATION_EMAIL: noreply@scottibyte.com
## If you added the Lets Encrypt template, uncomment below to get a free SSL certificate
LETSENCRYPT_ACCOUNT_EMAIL: me@example.com
## The http or https CDN address for this Discourse instance (configured to pull)
## see https://meta.discourse.org/t/14857 for details
#DISCOURSE_CDN_URL: https://discourse-cdn.example.com
## The maxmind geolocation IP address key for IP address lookup
## see https://meta.discourse.org/t/-/137387/23 for details
DISCOURSE_MAXMIND_LICENSE_KEY: aaaaaaaa
## The Docker container is stateless; all data is stored in /shared
volumes:
- volume:
host: /var/discourse/shared/standalone
guest: /shared
- volume:
host: /var/discourse/shared/standalone/log/var-log
guest: /var/log
## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
after_code:
- exec:
cd: $home/plugins
cmd:
- git clone https://github.com/discourse/docker_manager.git
## Any custom commands to run after building
run:
- exec: echo "Beginning of custom commands"
## If you want to set the 'From' email address for your first registration, uncomment and change:
## After getting the first signup email, re-comment the line. It only needs to run once.
# - exec: rails r "SiteSetting.notification_email='info@unconfigured.discourse.org'"
- exec: echo "End of custom commands"
Commented out the Maxmind key in the YAML. Rebuild is underway. By the way, email notifications have always worked fine. Lately the email for the backup download link no longer works. I simply go to the folder and download the backup via sftp and that works and the backup log completes successfully. So, what do you think the issue with the Maxmind key is?
As you can see, the rebuild worked and the forum is at the latest build. Taking a new backup worked from the GUI as did the backup notification and even the backup download link. Everything works fine.
Thanks, you are the master! Great comments and wonderful help sir.
So, even with a new Maxmind key, I had no luck. Any advice on that?
Ideally neither will, just need confirmation of what exactly is broken with proper reproduction steps so we can file a but and assign an engineer to fix it.
All I did was to comment the key out and the rebuild worked perfectly. With a valid (and even new key) the build fails with the logs that I have provided.
