I used this weekend to install some external tools on the VM that runs our Discourse installation (grafana, prometheus, some exporters) as well as some analytics (matomo) and setting up the backups on an external location (AWS S3). It’s been two months since our migration and I finally got some free time to do it.
After a couple days left to have metrics and stuff to simmer, I went back to check on it and noticed that the container running discourse is soaking up basically all the RAM available according to cadvisor:
Which is weird to me because checking from other sources it doesn’t seem so, however, I noticed that other aspects still have some things I’d like to understand better.
CPU usage for example have spikes but tend to stay on average well over 100% nearly at all times:
However this is the output of Docker Stats:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
254d80933447 grafana 0.02% 83.34MiB / 29.38GiB 0.28% 4.2MB / 10.1MB 88.2MB / 13MB 14
78d8a523c667 prometheus 0.00% 114.6MiB / 29.38GiB 0.38% 741MB / 50.7MB 190MB / 201MB 14
d602e2724c7a cadvisor 1.48% 67.52MiB / 29.38GiB 0.22% 12.3MB / 691MB 166MB / 4.08MB 24
4718b3629c8e docker_state_exporter 0.00% 11.54MiB / 29.38GiB 0.04% 2.85MB / 38.8MB 2.7MB / 90.1kB 14
c5a211185855 app 337.52% 7.543GiB / 29.38GiB 25.67% 365MB / 883MB 360GB / 67.3GB 282
9b95fa3156bb matomo_cron 0.00% 7.504MiB / 29.38GiB 0.02% 1.48kB / 0B 762MB / 0B 3
553a3e7389eb matomo_web 0.11% 9.832MiB / 29.38GiB 0.03% 106MB / 203MB 8.89MB / 33MB 9
adf21bdea1e5 matomo_app 0.01% 113.3MiB / 29.38GiB 0.38% 166MB / 146MB 1.26GB / 153MB 4
96d873027990 matomo_db 0.03% 99.66MiB / 29.38GiB 0.33% 63.8MB / 126MB 118MB / 310MB 15
9d21fdde2ec9 node_exporter 0.00% 9.887MiB / 29.38GiB 0.03% 3MB / 48.9MB 10.5MB / 299kB 6
Reading around this forum I already tried reducing the numbers of unicorn processes from the auto-detection (8) to 4 but I don’t see any relevant change in terms of used CPU/Memory.
Last but not least, when we imported out db from vbulletin3 to discourse, the database itself was around 7GB. Checking today I can see that it grew tenfold.
du -sh /var/discourse/shared/standalone/* | sort -hr | head -n 10
70G /var/discourse/shared/standalone/postgres_data
1.6G /var/discourse/shared/standalone/uploads
807M /var/discourse/shared/standalone/log
69M /var/discourse/shared/standalone/redis_data
200K /var/discourse/shared/standalone/postgres_run
28K /var/discourse/shared/standalone/state
12K /var/discourse/shared/standalone/tmp
12K /var/discourse/shared/standalone/ssl
8.0K /var/discourse/shared/standalone/backups
4.0K /var/discourse/shared/standalone/postgres_backup
I suppose this is postgresql doing its thing in the background and creating tons extra data but is there anything that can be done to at least control it?
Output of the Discourse Doctor in case it could help:
DISCOURSE DOCTOR Mon May 15 09:44:17 AM CEST 2023
OS: Linux vmi1229594.OMITTED.net 5.15.0-67-generic #74-Ubuntu SMP Wed Feb 22 14:14:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Found containers/app.yml
==================== YML SETTINGS ====================
DISCOURSE_HOSTNAME=OMITTED
SMTP_ADDRESS=OMITTED
DEVELOPER_EMAILS=OMITTED
SMTP_PASSWORD=OMITTED
SMTP_PORT=OMITTED
SMTP_USER_NAME=OMITTED
LETSENCRYPT_ACCOUNT_EMAIL=OMITTED
==================== DOCKER INFO ====================
DOCKER VERSION: Docker version 23.0.1, build a5ee5b1
DOCKER PROCESSES (docker ps -a)
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
254d80933447 grafana/grafana:latest "/run.sh" 8 hours ago Up 7 hours 0.0.0.0:8443->3000/tcp grafana
78d8a523c667 prom/prometheus:latest "/bin/prometheus --c…" 8 hours ago Up 8 hours 0.0.0.0:9090->9090/tcp prometheus
d602e2724c7a gcr.io/cadvisor/cadvisor:latest "/usr/bin/cadvisor -…" 8 hours ago Up 8 hours (healthy) cadvisor
4718b3629c8e karugaru/docker_state_exporter "/go/bin/docker_stat…" 8 hours ago Up 8 hours docker_state_exporter
c5a211185855 local_discourse/app "/sbin/boot" 9 hours ago Up 7 hours 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp app
9b95fa3156bb matomo:fpm "bash -c 'bash -s <<…" 20 hours ago Up 20 hours 9000/tcp matomo_cron
553a3e7389eb nginx:alpine "/docker-entrypoint.…" 21 hours ago Up 21 hours 80/tcp, 0.0.0.0:2053->443/tcp matomo_web
adf21bdea1e5 matomo:fpm-alpine "/entrypoint.sh php-…" 21 hours ago Up 21 hours 9000/tcp matomo_app
96d873027990 mariadb "docker-entrypoint.s…" 21 hours ago Up 21 hours 3306/tcp matomo_db
9d21fdde2ec9 quay.io/prometheus/node-exporter:latest "/bin/node_exporter …" 36 hours ago Up 36 hours node_exporter
c5a211185855 local_discourse/app "/sbin/boot" 9 hours ago Up 7 hours 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp app
adf21bdea1e5 matomo:fpm-alpine "/entrypoint.sh php-…" 21 hours ago Up 21 hours 9000/tcp matomo_app
Discourse container app is running
==================== PLUGINS ====================
- git clone https://github.com/discourse/docker_manager.git
- git clone https://github.com/discourse/discourse-spoiler-alert.git
- git clone https://github.com/discourse/discourse-animated-avatars.git
- git clone https://github.com/discourse/discourse-whos-online.git
- git clone https://github.com/discourse/discourse-bbcode.git
- git clone https://github.com/discourse/discourse-signatures.git
- git clone https://github.com/discourse/discourse-reactions.git
- git clone https://github.com/paviliondev/discourse-legal-tools.git
- git clone https://github.com/discourse/discourse-patreon.git
- git clone https://github.com/discourse/discourse-yearly-review.git
- git clone https://github.com/discourse/discourse-user-notes.git
- git clone https://github.com/merefield/discourse-user-network-vis.git
- git clone https://github.com/discourse/discourse-calendar.git
- git clone https://github.com/discourse/discourse-prometheus.git
WARNING:
You have what appear to be non-official plugins.
If you are having trouble, you should disable them and try rebuilding again.
See https://github.com/discourse/discourse/blob/main/lib/plugin/metadata.rb for the official list.
========================================
Discourse 3.1.0.beta4
Discourse version at OMITTED: Discourse 3.1.0.beta4
Discourse version at localhost: Discourse 3.1.0.beta4
==================== MEMORY INFORMATION ====================
RAM (MB): 31550
total used free shared buff/cache available
Mem: 30088 3958 1307 4269 24823 21475
Swap: 8191 1140 7051
==================== DISK SPACE CHECK ====================
---------- OS Disk Space ----------
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 194G 93G 91G 51% /
---------- Container Disk Space ----------
Filesystem Size Used Avail Use% Mounted on
overlay 194G 93G 91G 51% /
/dev/sda3 194G 93G 91G 51% /shared
/dev/sda3 194G 93G 91G 51% /var/log
==================== DISK INFORMATION ====================
Disk /dev/sda: 200 GiB, 214748364800 bytes, 419430400 sectors
Disk model: QEMU HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: OMITTED
Device Start End Sectors Size Type
/dev/sda1 2048 4095 2048 1M BIOS boot
/dev/sda2 4096 4194303 4190208 2G Linux filesystem
/dev/sda3 4194304 419428351 415234048 198G Linux filesystem
==================== END DISK INFORMATION ====================
==================== MAIL TEST ====================
For a robust test, get an address from http://www.mail-tester.com/
Or just send a test message to yourself.
Email address for mail test? ('n' to skip) [OMITTED]: n
Mail test skipped.
Replacing: SMTP_PASSWORD
Replacing: LETSENCRYPT_ACCOUNT_EMAIL
Replacing: DEVELOPER_EMAILS
Replacing: DISCOURSE_DB_PASSWORD
Replacing: Sending mail to
==================== DONE! ====================