Rm: cannot remove '/var/run/postgresql': Directory not empty

After having a full disk and not managing to move my images, I decided to delete everything, start a new discourse container and use the backup.

I deleted /var/discourse and did a docker system prune -a

Then I followed the official guide. But… something is wrong:

 ./launcher rebuild app
x86_64 arch detected.
Ensuring launcher is up to date
Fordere an von origin
Launcher is up-to-date
2.0.20230222-0048: Pulling from discourse/base
Digest: sha256:XXXX
Status: Image is up to date for discourse/base:2.0.20230222-0048
docker.io/discourse/base:2.0.20230222-0048
/usr/local/lib/ruby/gems/3.2.0/gems/pups-1.1.1/lib/pups.rb
/usr/local/bin/pups --stdin
I, [2023-03-04T12:04:27.482399 #1]  INFO -- : Reading from stdin
I, [2023-03-04T12:04:27.486188 #1]  INFO -- : > locale-gen $LANG && update-locale
I, [2023-03-04T12:04:28.456925 #1]  INFO -- : Generating locales (this might take a while)...
Generation complete.

I, [2023-03-04T12:04:28.457319 #1]  INFO -- : > mkdir -p /shared/postgres_run
I, [2023-03-04T12:04:28.581971 #1]  INFO -- :
I, [2023-03-04T12:04:28.583082 #1]  INFO -- : > chown postgres:postgres /shared/postgres_run
I, [2023-03-04T12:04:28.588444 #1]  INFO -- :
I, [2023-03-04T12:04:28.588792 #1]  INFO -- : > chmod 775 /shared/postgres_run
I, [2023-03-04T12:04:28.591995 #1]  INFO -- :
I, [2023-03-04T12:04:28.592147 #1]  INFO -- : > rm -fr /var/run/postgresql
rm: cannot remove '/var/run/postgresql': Directory not empty
I, [2023-03-04T12:04:28.616079 #1]  INFO -- :


FAILED
--------------------
Pups::ExecError: rm -fr /var/run/postgresql failed with return #<Process::Status: pid 16 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.2.0/gems/pups-1.1.1/lib/pups/exec_command.rb:117:in `spawn'
exec failed with the params "rm -fr /var/run/postgresql"
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.

Can anyone help me there?

1 Like

Did you stop the old container?

docker stop app
docker rm app

2 Likes

Well, while that would totally be like me, this time it’s not that simple:

# sudo -s
# cd /var/discourse/
# docker stop app
Error response from daemon: No such container: app
# docker rm app
Error: No such container: app

I think the problem might be that I moved everything from the old default location from docker images to the new place and now the discourse docker is not able to access the filesystem it expects?

1 Like

/var/run/postgresql does not exist.

I have no idea why this isn’t working anymore. :cry:

1 Like

What does your app.yml look like?

1 Like
$# cat app.yml
## this is the all-in-one, standalone Discourse Docker container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild app
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed

templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
#  - "templates/web.ssl.template.yml"
#  - "templates/web.letsencrypt.ssl.template.yml"

## which TCP/IP ports should this container expose?
## If you want Discourse to share a port with another webserver like Apache or nginx,
## see https://meta.discourse.org/t/17247 for details
expose:
#  - "7890:80"   # http
#  - "7891:443" # https

labels:
  app_name: discourse
  traefik.enable: true
  traefik.http.routers.discourse.entrypoints: http
  traefik.http.routers.discourse.rule: Host(`talk.spielleiten.de`)      ## Hier eure Domain eintragen
  traefik.http.middlewares.discourse-https-redirect.redirectscheme.scheme: https
  traefik.http.routers.discourse.middlewares: discourse-https-redirect
  traefik.http.routers.discourse-secure.entrypoints: https
  traefik.http.routers.discourse-secure.rule: Host(`talk.spielleiten.de`)      ## Hier eure Domain eintragen
  traefik.http.routers.discourse-secure.tls: true
  traefik.http.routers.discourse-secure.tls.certresolver: http
  traefik.http.routers.discourse-secure.service: discourse
  traefik.http.services.discourse.loadbalancer.server.port: 80
  traefik.docker.network: proxy

docker_args:
  - "--network=proxy"

networks:
  - proxy

params:
  db_default_text_search_config: "pg_catalog.english"

  ## Set db_shared_buffers to a max of 25% of the total memory.
  ## will be set automatically by bootstrap based on detected RAM, or you can override
  #db_shared_buffers: "256MB"

  ## can improve sorting performance, but adds memory usage per-connection
  #db_work_mem: "40MB"

  ## Which Git revision should this container use? (default: tests-passed)
  #version: tests-passed

env:
  LC_ALL: de_DE.UTF-8
  LANG: de_DE.UTF-8
  LANGUAGE: dn_DE.UTF-8
  EMBER_CLI_PROD_ASSETS: 1
  # DISCOURSE_DEFAULT_LOCALE: en

  ## How many concurrent web requests are supported? Depends on memory and CPU cores.
  ## will be set automatically by bootstrap based on detected CPUs, or you can override
  #UNICORN_WORKERS: 3

  ## TODO: The domain name this Discourse instance will respond to
  ## Required. Discourse will not work with a bare IP number.
  DISCOURSE_HOSTNAME: talk.spielleiten.de

  ## Uncomment if you want the container to be started with the same
  ## hostname (-h option) as specified above (default "$hostname-$config")
  #DOCKER_USE_HOSTNAME: true

  ## TODO: List of comma delimited emails that will be made admin and developer
  ## on initial signup example 'user1@example.com,user2@example.com'
  DISCOURSE_DEVELOPER_EMAILS: 'XXX'

  ## TODO: The SMTP mail server used to validate new accounts and send notifications
  # SMTP ADDRESS, username, and password are required
  # WARNING the char '#' in SMTP password can cause problems!
[deleted for privacy reasons]

  ## The http or https CDN address for this Discourse instance (configured to pull)
  ## see https://meta.discourse.org/t/14857 for details
  #DISCOURSE_CDN_URL: https://discourse-cdn.example.com

  ## The maxmind geolocation IP address key for IP address lookup
  ## see https://meta.discourse.org/t/-/137387/23 for details
  DISCOURSE_MAXMIND_LICENSE_KEY: XXX

## The Docker container is stateless; all data is stored in /shared
volumes:
  - volume:
      host: /var/discourse/shared/standalone
      guest: /shared
  - volume:
      host: /var/discourse/shared/standalone/log/var-log
      guest: /var/log

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git

## Any custom commands to run after building
run:
  - exec: echo "Beginning of custom commands"
  ## If you want to set the 'From' email address for your first registration, uncomment and change:
  ## After getting the first signup email, re-comment the line. It only needs to run once.
  #- exec: rails r "SiteSetting.notification_email='info@unconfigured.discourse.org'"
  - exec: echo "End of custom commands"

This worked till I changed daemon.json:

"data-root": "/srv/new-drive/new-docker-root"
1 Like

Then that’s your problem. This is a docker issue, not a Discourse issue.

My guess is that it has to do with permissions or the drive not being there somehow.

1 Like

I assume so. But all the other services in dockers work as intended! It’s just discourse that won’t try to access the new data-root.
That’s why I hoped to find a solution here.

2 Likes

You might try changing the volumes to be somewhere other than /var/discourse/shared/standalone

I presume that you already deleted /var/discourse/shared?

1 Like

I did a rm -R /var/discourse and followed the official install guide as one of the first steps.

I’ll try that!

1 Like

I tried both a directoy on the same disk as well als a folder in “data-root” - nothing changed.

Why does the rm command fail? I assume this is still the file system of the operating system (as opposed to inside the container) - and that operation works if I repeat the command in the shell.

2 Likes

I believe that delete command is inside the container that it is building.

I don’t think I have any more ideas.

1 Like

Thank you very, very much anyway!

2 Likes

That directory maps to /var/discourse/shared/standalone/postgres_run in the host.

2 Likes
# ls -lsa /var/discourse/shared/standalone
insgesamt 16
4 drwxr-xr-x 4 root            root             4096 Mär  6 08:34 .
4 drwxr-xr-x 3 root            root             4096 Mär  6 08:34 ..
4 drwxr-xr-x 3 root            root             4096 Mär  6 08:34 log
4 drwxrwxr-x 2 systemd-resolve systemd-timesync 4096 Mär  6 08:34 postgres_run
# ls -lsa /var/discourse/shared/standalone/postgres_run/
insgesamt 8
4 drwxrwxr-x 2 systemd-resolve systemd-timesync 4096 Mär  6 08:34 .
4 drwxr-xr-x 4 root            root             4096 Mär  6 08:34 ..

Is this what you would expect? df shows ample space, so it’s not a full disk issue.

# ./discourse-doctor
DISCOURSE DOCTOR Mo 6. Mär 18:54:01 CET 2023
OS: Linux dracula 5.4.0-139-generic #156-Ubuntu SMP Fri Jan 20 17:27:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux


Found containers/app.yml

==================== YML SETTINGS ====================
[deleted for privacy reasons]

==================== DOCKER INFO ====================
DOCKER VERSION: Docker version 20.10.12, build 20.10.12-0ubuntu2~20.04.1

DOCKER PROCESSES (docker ps -a)

CONTAINER ID   IMAGE                                        COMMAND                  CREATED      STATUS                 PORTS                                                                      NAMES
f6d3f502384f   cthulhoo/ttrss-fpm-pgsql-static:latest       "/opt/tt-rss/updater…"   4 days ago   Up 9 hours             9000/tcp                                                                   ttrss-docker_updater_1
fb0c774b31cb   cthulhoo/ttrss-web-nginx:latest              "/docker-entrypoint.…"   4 days ago   Up 9 hours (healthy)   127.0.0.1:8280->80/tcp                                                     ttrss-docker_web-nginx_1
83a32795b305   cthulhoo/ttrss-fpm-pgsql-static:latest       "/opt/tt-rss/dcron.s…"   4 days ago   Up 9 hours             9000/tcp                                                                   ttrss-docker_backups_1
c211bc93f368   cthulhoo/ttrss-fpm-pgsql-static:latest       "/bin/sh -c ${SCRIPT…"   4 days ago   Up 9 hours             9000/tcp                                                                   ttrss-docker_app_1
4ecb89c6bb9f   postgres:12-alpine                           "docker-entrypoint.s…"   4 days ago   Up 9 hours             5432/tcp                                                                   ttrss-docker_db_1
c668703d805d   traefik:latest                               "/entrypoint.sh trae…"   4 days ago   Up 9 hours             0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   traefik
f67a63243f13   ghcr.io/paperless-ngx/paperless-ngx:latest   "/sbin/docker-entryp…"   4 days ago   Up 9 hours (healthy)   0.0.0.0:8000->8000/tcp, :::8000->8000/tcp                                  paperless-web
d2fbf8682efb   postgres:13                                  "docker-entrypoint.s…"   4 days ago   Up 9 hours             5432/tcp                                                                   paperless-db
8284f03fdcd0   redis:6.0                                    "docker-entrypoint.s…"   4 days ago   Up 9 hours             6379/tcp                                                                   paperless-redis
ab29487743b0   tootsuite/mastodon                           "/usr/bin/tini -- no…"   4 days ago   Up 9 hours (healthy)   3000/tcp, 4000/tcp                                                         mastodon_streaming_1
a117f5e785f8   tootsuite/mastodon                           "/usr/bin/tini -- bu…"   4 days ago   Up 9 hours (healthy)   3000/tcp, 4000/tcp                                                         mastodon_sidekiq_1
5a4a7e99230c   tootsuite/mastodon                           "/usr/bin/tini -- ba…"   4 days ago   Up 9 hours (healthy)   3000/tcp, 4000/tcp                                                         mastodon_web_1
5484d4dd39b6   postgres:14-alpine                           "docker-entrypoint.s…"   4 days ago   Up 9 hours (healthy)   5432/tcp                                                                   mastodon_db_1
ecb2e7cbdc0b   redis:7-alpine                               "docker-entrypoint.s…"   4 days ago   Up 9 hours (healthy)   6379/tcp                                                                   mastodon_redis_1

c211bc93f368   cthulhoo/ttrss-fpm-pgsql-static:latest       "/bin/sh -c ${SCRIPT…"   4 days ago   Up 9 hours             9000/tcp                                                                   ttrss-docker_app_1

Discourse container app is running


==================== PLUGINS ====================
          - git clone https://github.com/discourse/docker_manager.git

No non-official plugins detected.

See https://github.com/discourse/discourse/blob/main/lib/plugin/metadata.rb for the official list.

========================================
Discourse version at talk.spielleiten.de: NOT FOUND
Discourse version at localhost: NOT FOUND


==================== MEMORY INFORMATION ====================
RAM (MB):

              gesamt      belegt       frei     gemeinsam    Zwischen   verfügbar
Speicher:        7870        2292         141        4207        5436        1101
Auslager:        2047         580        1467

==================== DISK SPACE CHECK ====================
---------- OS Disk Space ----------
Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
/dev/sda2       109G     68G   36G   66% /
/dev/md0        3,7T    3,1T  581G   85% /mnt/raid

==================== DISK INFORMATION ====================
Festplatte /dev/sda: 111,81 GiB, 120034123776 Bytes, 234441648 Sektoren
Festplattenmodell: Samsung SSD 840
Einheiten: Sektoren von 1 * 512 = 512 Bytes
Sektorgröße (logisch/physikalisch): 512 Bytes / 512 Bytes
E/A-Größe (minimal/optimal): 512 Bytes / 512 Bytes
Festplattenbezeichnungstyp: gpt
Festplattenbezeichner: F889FA6C-2FB7-44B6-8CDF-1E6A05B9E33D

Gerät       Anfang      Ende  Sektoren  Größe Typ
/dev/sda1     2048   1050623   1048576   512M EFI-System
/dev/sda2  1050624 234440703 233390080 111,3G Linux-Dateisystem


Festplatte /dev/sdb: 186,32 GiB, 200049647616 Bytes, 390721968 Sektoren
Festplattenmodell: SAMSUNG HD200HJ
Einheiten: Sektoren von 1 * 512 = 512 Bytes
Sektorgröße (logisch/physikalisch): 512 Bytes / 512 Bytes
E/A-Größe (minimal/optimal): 512 Bytes / 512 Bytes
Festplattenbezeichnungstyp: dos
Festplattenbezeichner: 0x13e913e8

Gerät      Boot Anfang      Ende  Sektoren  Größe Kn Typ
/dev/sdb1         2048 390721967 390719920 186,3G 83 Linux


Festplatte /dev/sdc: 1,84 TiB, 2000398934016 Bytes, 3907029168 Sektoren
Festplattenmodell: SAMSUNG HD203WI
Einheiten: Sektoren von 1 * 512 = 512 Bytes
Sektorgröße (logisch/physikalisch): 512 Bytes / 512 Bytes
E/A-Größe (minimal/optimal): 512 Bytes / 512 Bytes
Festplattenbezeichnungstyp: dos
Festplattenbezeichner: 0x00000000

Gerät      Boot Anfang       Ende   Sektoren Größe Kn Typ
/dev/sdc1           63 3907024064 3907024002  1,8T fd Linux RAID-Autoerkennung


Festplatte /dev/sde: 1,84 TiB, 2000398934016 Bytes, 3907029168 Sektoren
Festplattenmodell: SAMSUNG HD203WI
Einheiten: Sektoren von 1 * 512 = 512 Bytes
Sektorgröße (logisch/physikalisch): 512 Bytes / 512 Bytes
E/A-Größe (minimal/optimal): 512 Bytes / 512 Bytes
Festplattenbezeichnungstyp: dos
Festplattenbezeichner: 0x54cd8732

Gerät      Boot Anfang       Ende   Sektoren Größe Kn Typ
/dev/sde1           63 3907024064 3907024002  1,8T fd Linux RAID-Autoerkennung


Festplatte /dev/sdd: 1,84 TiB, 2000398934016 Bytes, 3907029168 Sektoren
Festplattenmodell: SAMSUNG HD203WI
Einheiten: Sektoren von 1 * 512 = 512 Bytes
Sektorgröße (logisch/physikalisch): 512 Bytes / 512 Bytes
E/A-Größe (minimal/optimal): 512 Bytes / 512 Bytes
Festplattenbezeichnungstyp: dos
Festplattenbezeichner: 0xd4a5e07f

Gerät      Boot Anfang       Ende   Sektoren Größe Kn Typ
/dev/sdd1           63 3907024064 3907024002  1,8T fd Linux RAID-Autoerkennung


Festplatte /dev/md0: 3,65 TiB, 4000792182784 Bytes, 7814047232 Sektoren
Einheiten: Sektoren von 1 * 512 = 512 Bytes
Sektorgröße (logisch/physikalisch): 512 Bytes / 512 Bytes
E/A-Größe (minimal/optimal): 65536 Bytes / 131072 Bytes

==================== END DISK INFORMATION ====================

==================== DONE! ====================

1 Like