After upgrading (with ‘launcher rebuild app’ times two) to Postgres 10, we’ve started seeing errors like this in /logs:
Failed to handle exception in exception app middleware :
PG::DiskFull: ERROR: could not resize shared memory segment "/PostgreSQL.682207201" to 283432 bytes: No space left on device :
SELECT "topics"."id" AS t0_r0, "topics"."title" AS t0_r1, "topics"."last_posted_at" AS t0_r2, …
Ideas? Looks like PG is unable to increase the size of a shared memory segment, can this be tuned somewhere?
This is due to the fact that docker by default restricts shared memory size to 64MB which makes pg10 run afoul of this limitation when doing a number of scatter/gather jobs in parallel:
Outside the container:
supermathie@host: ~ $ df -h /dev/shm
Filesystem Size Used Avail Use% Mounted on
none 63G 1.2M 63G 1% /run/shm
Inside the container:
supermathie@host: ~ $ docker exec -it postgres-master df -h /dev/shm
Filesystem Size Used Avail Use% Mounted on
shm 64M 8.0K 64M 1% /dev/shm
You should raise the limit when launching the postgres container by including the --shm-size=1g parameter.
We’ll look at fixing this in the launcher.
(or as the link suggests, disable parallelism in pg10 with max_parallel_workers_per_gather = 0)
You need to make sure that’s on your data container.
EDIT:
OK, adding the following:
docker_args: "--shm-size=512m"
to your app.yml (if all-in-one) or data.yml (if separate pg) containers should prevent this from happening. I’ve left the shm size at a conservative 512MB.
Or if you pull latest and relaunch it’ll now start the container with that shm size.
@tgxworld I’ve pushed a change that changes the launcher to always launch with 512MB SHM size under bootstrap or run. Should be safe.