Can't increase db_shared_buffers (0.1% of total ram) and db_work_mem (0.03% of total ram)

[markersocial]

This is a bit of a head scratcher

Have a freshly migrated Discourse instance on a new server.

Getting the error in logs: PG::DiskFull (ERROR: could not resize shared memory segment "/PostgreSQL.1759815625" to 8388608 bytes: No space left on device )

It’s strange because, the previous server (64gb ram) didn’t have this issue and I had these set:

db_shared_buffers: "25632MB"
db_work_mem: "160MB"

On the new server (128gb ram), I can’t increase above the baseline defaults (I tried tripling the values below and get the same PG DiskFull error):

db_shared_buffers: "128MB"
db_work_mem: "40MB"

Previous machine has docker 27.x on it (auto installed via the discourse installer). New machine installed docker.io as per instructions (so 26.x). I tried switching to docker 27.x to see if that was related - but it didn’t change anything. Both are on the Stable Discourse branch, version 3.3.2.

It seems shm_size might be the main culprit:

PG throws could not resize shared memory segment error

This is due to the fact that docker by default restricts shared memory size to 64MB which makes pg10 run afoul of this limitation when doing a number of scatter/gather jobs in parallel: Outside the container: supermathie@host: ~ $ df -h /dev/shm Filesystem Size Used Avail Use% Mounted on none 63G 1.2M 63G 1% /run/shm Inside the container: supermathie@host: ~ $ docker exec -it postgres-master df -h /dev/shm Filesystem Size Used Avail Use% Mounted on shm …

Though no idea why it wasn’t an issue on the previous server and it is on the new server. The only other main difference is the old server uses Ubuntu 22.04 LTS and the new server is on 24.04 LTS.

I tried this also, but the changes get overwritten when the container is started again

Docker shm_size Option in app.yml

Hey @Ghan In the meantime (for testing purposes, see caveat below), you can change this directly with docker after the container is built, as follows: Edit the /var/lib/docker/containers/$CONTAINER_ID/hostconfig.json file directly. For example, change the value for ShmSize in the file above. Stop and restart the container. In our docker container hostconfig file, it looks like this: "ShmSize":536870912, [Screen Shot 2020-12-29 at 12.39.17 PM] HTH Caveat: Some people have posted that y…

shm_size appears to be hardcoded into the launcher:

discourse/discourse_docker/blob/40169e87630008d00dcc559606eecebca2d82e78/launcher#L600

discourse/discourse_docker/blob/40169e87630008d00dcc559606eecebca2d82e78/launcher#L616

discourse/discourse_docker/blob/40169e87630008d00dcc559606eecebca2d82e78/launcher#L659

Any insights or help would be appreciated!

[pfaffman (Jay Pfaffman)]

@markersocial

: No space left on device

Refers to the hard drive space, not the ram.

[markersocial]

@pfaffman

Refers to the hard drive space, not the ram.

Thanks @pfaffman - I thought that too initially, but then I read this thread:

PG throws could not resize shared memory segment error

After upgrading (with ‘launcher rebuild app’ times two) to Postgres 10, we’ve started seeing errors like this in /logs: Failed to handle exception in exception app middleware : PG::DiskFull: ERROR: could not resize shared memory segment "/PostgreSQL.682207201" to 283432 bytes: No space left on device : SELECT "topics"."id" AS t0_r0, "topics"."title" AS t0_r1, "topics"."last_posted_at" AS t0_r2, … Ideas? Looks like PG is unable to increase the size of a shared memory segment, can this be tuned …

There is a lot of free space also

[markersocial]

So a quick duct tape fix is to simply edit the launcher file like:

cd /var/discourse
vi launcher

Then replace all 3 occurances of
--shm-size=512m
with the desired ram amount (I went with 50% of system ram).

Then rebuild. Will need to edit it again every time the launcher updates.

Seems to be doing the job for the time being.

[markersocial]

@mcdanlj

Kernel configuration

Redis (one of the key components on which Discourse is built) strongly recommends disabling transparent huge pages when using on-disk persistence (which Discourse does), and I also allow memory overcommit.

echo 'sys.kernel.mm.transparent_hugepage.enabled=never' > /etc/sysctl.d/10-huge-pages.conf
echo 'vm.overcommit_memory=1' > /etc/sysctl.d/90-vm_overcommit_memory.conf
sysctl --system

So to follow up in case someone else faces this issue - the above solved this for me. I do not need to edit shm-size in the launcher anymore.

This was found after noticing this warning during rebuild:
WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.