Troubleshooting bootstrap failures in Discourse

Support
1

I am still struggling with the new “pre-bundling” process. The bootstrap fails on my test instance and I dont know why. Note that this is is a setup with an external postgres database. The containers run behind a haproxy as ssl accelerator. I only bootstrap the web_only container here. The current build version (before the bootstrap) is 3.5.0.beta8-dev, Commits · discourse/discourse · GitHub

This is the currently failing yml:

# IMPORTANT: SET A SECRET PASSWORD in Postgres for the Discourse User
# TODO: change SOME_SECRET in this template

templates:
  - "templates/web.template.yml"
  ## Uncomment the next line to enable the IPv6 listener
  #- "templates/web.ipv6.template.yml"
  - "templates/web.ratelimited.template.yml"
  - "templates/web.socketed.template.yml"
  ## Uncomment these two lines if you wish to add Lets Encrypt (https)
  #- "templates/web.ssl.template.yml"
  #- "templates/web.letsencrypt.ssl.template.yml"

## which TCP/IP ports should this container expose?
## If you want Discourse to share a port with another webserver like Apache or nginx,
## see https://meta.discourse.org/t/17247 for details
expose:
  - "80:80"   # http
  - "443:443" # https

# Use 'links' key to link containers together, aka use Docker --link flag.
links:
  - link:
      name: redis
      alias: data

# any extra arguments for Docker?
# docker_args:

params:
  ## Which Git revision should this container use? (default: tests-passed)
  #version: tests-passed

env:
  LC_ALL: en_US.UTF-8
  LANG: en_US.UTF-8
  LANGUAGE: en_US.UTF-8
  DISCOURSE_DEFAULT_LOCALE: de

  ## How many concurrent web requests are supported? Depends on memory and CPU cores.
  ## will be set automatically by bootstrap based on detected CPUs, or you can override
  UNICORN_WORKERS: 4

  ## TODO: The domain name this Discourse instance will respond to
  DISCOURSE_HOSTNAME: 'forum2.netzwissen.de'

  ## Uncomment if you want the container to be started with the same
  ## hostname (-h option) as specified above (default "$hostname-$config")
  #DOCKER_USE_HOSTNAME: true

  ## TODO: List of comma delimited emails that will be made admin and developer
  ## on initial signup example 'user1@example.com,user2@example.com'
  DISCOURSE_DEVELOPER_EMAILS: 'admin@netzwissen.de'

  ## TODO: The SMTP mail server used to validate new accounts and send notifications
  # SMTP ADDRESS, username, and password are required
  # WARNING the char '#' in SMTP password can cause problems!
  DISCOURSE_SMTP_ADDRESS: mail.netzwissen.de
  DISCOURSE_SMTP_PORT: 587
  DISCOURSE_SMTP_USER_NAME: discourse@netzwissen.de
  DISCOURSE_SMTP_PASSWORD: [xxxxxxxxxxxxxxxxx}
  #DISCOURSE_SMTP_ENABLE_START_TLS: true           # (optional, default true)
  #DISCOURSE_SMTP_DOMAIN: discourse.example.com    # (required by some providers)
  DISCOURSE_NOTIFICATION_EMAIL: discourse@netzwissen.de    # (address to send notifications from)

  ## If you added the Lets Encrypt template, uncomment below to get a free SSL certificate
  #LETSENCRYPT_ACCOUNT_EMAIL: me@example.com

## configure connectivity to the databases
  DISCOURSE_DB_SOCKET: ''
  DISCOURSE_DB_USERNAME: [xxxxxxxxxxx]
  DISCOURSE_DB_NAME: [xxxxxxxxxxxxxxx]
  DISCOURSE_DB_HOST: [xxxxxxxxxxxxxxx]
  DISCOURSE_DB_PASSWORD: [xxxxxxxxxxxxxxx]
  ## redis caching on the neighbouring container
  DISCOURSE_REDIS_HOST: redis

  ## The http or https CDN address for this Discourse instance (configured to pull)
  ## see https://meta.discourse.org/t/14857 for details
  #DISCOURSE_CDN_URL: https://discourse-cdn.example.com

  ## The maxmind geolocation IP account ID and license key for IP address lookups
  ## see https://meta.discourse.org/t/-/173941 for details
  #DISCOURSE_MAXMIND_ACCOUNT_ID: 123456
  #DISCOURSE_MAXMIND_LICENSE_KEY: 1234567890123456

volumes:
  - volume:
      host: /mnt/data/discourse/shared/web-only
      guest: /shared
  - volume:
      host: /mnt/data/discourse/shared/web-only/log/var-log
      guest: /var/log

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git
          - git clone https://github.com/discourse/discourse-shared-edits.git
# some plugins are removed, as they are now bundled with core
# https://meta.discourse.org/t/bundling-more-popular-plugins-with-discourse-core/373574
## = now bundled
##          - git clone https://github.com/discourse/discourse-chat-integration.git
##          - git clone https://github.com/discourse/discourse-openid-connect.git
##          - git clone https://github.com/discourse/discourse-calendar.git
##          - git clone https://github.com/angusmcleod/discourse-events.git
##          - git clone https://github.com/discourse/discourse-data-explorer.git
##          - git clone https://github.com/discourse/discourse-reactions.git
##          - git clone https://github.com/discourse/discourse-chat.git
##          - git clone https://github.com/discourse/discourse-ai.git
##          - git clone https://github.com/discourse/discourse-topic-voting.git
##          - git clone https://github.com/discourse/discourse-post-voting.git
##          - git clone https://github.com/discourse/discourse-user-notes.git
##          - git clone https://github.com/discourse/discourse-solved.git
          - git clone https://github.com/discourse/discourse-docs-card-filter.git
          - git clone https://github.com/discourse/discourse-doc-categories.git
##          - git clone https://github.com/discourse/discourse-assign.git
##          - git clone https://github.com/discourse/discourse-templates.git
          - git clone https://github.com/discourse/discourse-saved-searches.git
          - git clone https://github.com/discourse/discourse-tooltips.git
          - git clone https://github.com/discourse/discourse-category-experts.git
          - git clone https://github.com/discourse/discourse-activity-pub.git
          - git clone https://github.com/discourse/discourse-follow.git
          - git clone https://github.com/nathan-nz/discourse-wikified-posts.git
          - git clone https://github.com/discourse/discourse-whos-online.git
          - git clone https://github.com/merefield/discourse-workflow.git

## Remember, this is YAML syntax - you can only have one block with a name
run:
  - exec: echo "Beginning of custom commands"

  ## If you want to configure password login for root, uncomment and change:
  ## Use only one of the following lines:
  #- exec: /usr/sbin/usermod -p 'PASSWORD_HASH' root
  - exec: /usr/sbin/usermod -p "$(mkpasswd -m sha-256 'xxxxxxxxxxxxxxx')" root

## for operation behind haproxy load balancer with local nginx on the container host 
  - replace:
      filename: /etc/nginx/conf.d/discourse.conf
      from: "types {"
      to: |
        set_real_ip_from 127.0.0.1/24;
        set_real_ip_from 10.0.0.0/24;
        real_ip_header X-Forwarded-For;
        real_ip_recursive on;
        proxy_set_header Host $http_host;
        proxy_set_header X-Request-Start “t=${msec}”;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https; # replaced $thescheme;
        proxy_set_header X-Real-IP $remote_addr;
        types {

  ## If you want to authorized additional users, uncomment and change:
  #- exec: ssh-import-id username
  #- exec: ssh-import-id anotherusername

  - exec: echo "End of custom commands"
  - exec: awk -F\# '{print $1;}' ~/.ssh/authorized_keys | awk 'BEGIN { print "Authorized SSH keys for this container:"; } NF>=2 {print $NF;}'

Even when I add a rm -rf block into the ymll:

- rm -rf discourse-chat-integration  
- rm -rf discourse-openid-connect         
- rm -rf discourse-calendar         
- rm -rf discourse-events         
- rm -rf discourse-data-explorer         
- rm -rf discourse-reactions         
- rm -rf discourse-chat         
- rm -rf discourse-ai         
- rm -rf discourse-topic-voting         
- rm -rf discourse-post-voting         
- rm -rf discourse-user-notes         
- rm -rf discourse-solved         
- rm -rf discourse-assign         
- rm -rf discourse-templates

The bootstrap still fails with

/usr/local/lib/ruby/gems/3.3.0/gems/pups-1.3.0/lib/pups.rb
/usr/local/bin/pups --stdin

FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && su discourse -c 'bundle exec rake db:migrate' failed with return #<Process::Status: pid 669 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.3.0/lib/pups/exec_command.rb:131:in `spawn'
exec failed with the params {"cd"=>"$home", "tag"=>"migrate", "hook"=>"db_migrate", "cmd"=>["su discourse -c 'bundle exec rake db:migrate'"]}

I would interprete that as an error with the database migration at “bundle exec rake db:migrate”. But what is the root cause here? The discourse2 database already has the vector extension which was required for the (previously already installed) discourse-ai plugin:

postgres=# \c discourse2  
psql (15.12 (Ubuntu 15.12-1.pgdg22.04+1), server 13.20 (Ubuntu 13.20-1.pgdg22.04+1))  
You are now connected to database "discourse2" as user "postgres".  
discourse2=# \dx  
                                   List of installed extensions  
  Name   | Version |   Schema   |                            Description                               
----------+---------+------------+-------------------------------------------------------------------  
hstore   | 1.7     | public     | data type for storing sets of (key, value) pairs  
pg_trgm  | 1.5     | public     | text similarity measurement and index searching based on trigrams  
plpgsql  | 1.0     | pg_catalog | PL/pgSQL procedural language  
unaccent | 1.1     | public     | text search dictionary that removes accents  
vector   | 0.8.0   | public     | vector data type and ivfflat and hnsw access methods  
(5 rows)

Later on I also see the plugin hints although the “git clone” lines are all commented in the yml:

---
HINT: The plugin 'discourse-openid-connect' is now bundled with Discourse and should not be included in your container configuration.
Remove the line 'git clone https://github.com/discourse/discourse-openid-connect' from your containers/web_only.yml file, then try again.
For more information, see https://meta.discourse.org/t/373574
---

To be honest, I am a bit lost now ::sleepy_face: …

2

what deos free -h return?

3

root@docker3a:/var/discourse/containers# free -h
gesamt benutzt frei gemns. Puffer/Cache verfügbar
Speicher: 3,8Gi 2,0Gi 1,1Gi 3,8Mi 969Mi 1,8Gi

4

probably 4GB of swap

5

Sorry, my fault, the output was cut off

root@docker3a:~# free -h
gesamt benutzt frei gemns. Puffer/Cache verfügbar
Speicher: 3,8Gi 2,0Gi 1,1Gi 3,8Mi 971Mi 1,8Gi
Swap: 974Mi 0B 974Mi

The VM already has 4 GB of memory. I added 2 GB more and started another ./launcher rebuild web_only^, resulting with the same error. So I doubt we have a space-related build issue here …

6

Well, I needed 2 GB swap with 8 GB RAM.

1 Like
7

I tried again with 8 GB RAM, with the same error result. Note that this is a Kernel-based virtual machine (KVM) on a proxmox host (Features - Proxmox Virtual Environment).

A QEMU/KVM guest requests memory from the host as the guest OS needs it, but it does not return the memory back to the host automatically if the guest OS doesn’t need it anymore. If the host needs memory, the kernel will ask processes to return unused RAM, thus QEMU will try to recover some memory but it depends on guest OS to release it first.

I never had memory issues with the 4 GB RAM and 4 CPU setup, even not with discourse-ai plugin. Bootstrap problems only appeared after this bundling announcement …

1 Like
8

Wait is this discourse-openid-connect plugin anywhere in your yml file?

This is an issue migrating, can you scroll up to find what failed to migrate?

1 Like