Docker shm_size Option in app.yml

Ghan · December 23, 2020, 2:32am

Would it be possible to make the shm_size param an option in app.yml? I ran into this error today when trying to do a vacuum analyze:

ERROR: could not resize shared memory segment

It was trying to increase the shm usage to 1 GB and failed, as it’s only 512 MB out of the box. This param looks to be in the launcher script, but it might be nice to have this as an option (possibly commented out by default) in the .yml file.

tgxworld · December 23, 2020, 4:01am

You can pass custom docker args to override the default we provide with --docker-args.

github.com/discourse/discourse_docker

launcher

master


      
            echo "    enter:       Open a shell to run commands inside the container"
            echo "    logs:        View the Docker logs for a container"
            echo "    bootstrap:   Bootstrap a container for the config based on a template"
            echo "    run:         Run the given command with the config in the context of the last bootstrapped image"
            echo "    rebuild:     Rebuild a container (destroy old, bootstrap, start new)"
            echo "    cleanup:     Remove all containers that have stopped for > 24 hours"
            echo "    start-cmd:   Generate docker command used to start container"
            echo
            echo "Options:"
            echo "    --skip-prereqs             Don't check launcher prerequisites"
            echo "    --docker-args              Extra arguments to pass when running docker"
            echo "    --skip-mac-address         Don't assign a mac address"
            echo "    --run-image                Override the image used for running the container"
            exit 1
          }
          
          # for potential re-exec later
          SAVED_ARGV=("$@")
          
          command=$1
          config=$2

I believe it’ll look something like ./launcher start app --docker-args "--shm-size=1024m"

Ghan · December 23, 2020, 1:44pm

This didn’t work when I tried it unfortunately. I think maybe the container needs to be rebuild to increase the size of the shared memory space? I had to change the value in the hard-coded areas in the launcher script and then rebuild the container for it to actually take effect.

This likely wouldn’t persist across updates/rebuilds either - you’d need to be manually changing it every time.

tgxworld · December 24, 2020, 1:03am

What commands did you run? You’ll definitely have to rebuild the container for the --shm-size option to take effect.

Ghan · December 24, 2020, 2:40am

All I did at first was stop the container, then run the launcher start command with that parameter. Then I manually edited the --shm-size params inside the launcher script itself and rebuilt the container. Then it adjusted the value correctly. If you override the param on the command line to rebuild the container, then won’t the change be lost if you perform an update/rebuild through the /admin/upgrade interface? That’s why I thought it might be best to make it a field in app.yml - then it would persist through rebuilds and code updates.

tgxworld · December 24, 2020, 5:46am

Hmm I can see this being useful. @sam What do you think about this?

sam · December 29, 2020, 4:52am

That never rebuilds the container though, so it is safe.

Don’t we already have this per:

github.com/discourse/discourse_docker

launcher

0326f200e


      
          local docker_args
          
          docker_args=`cat $config_file | $docker_path run $user_args --rm -i -a stdout -a stdin $image ruby -e \
                  "require 'yaml'; puts YAML.load(STDIN.readlines.join)['docker_args']"`
          
          if [[ -n "$docker_args" ]]; then
            user_args="$user_args_argv $docker_args"
          fi

docker_args: .... in your yaml.

Ghan · December 29, 2020, 5:12am

I’m not sure. Does that overwrite the launcher file itself? I see 3 places where the option is hardcoded:

github.com/discourse/discourse_docker

launcher

0326f200e


      
          $docker_path run --shm-size=512m $links $attach_on_run $restart_policy "${env[@]}" "${labels[@]}" -h "$hostname" \
             -e DOCKER_HOST_IP="$docker_ip" --name $config -t "${ports[@]}" $volumes $mac_address $user_args \
             $run_image $boot_command

github.com/discourse/discourse_docker

launcher

0326f200e


      
          (exec $docker_path run --rm --shm-size=512m $user_args $links "${env[@]}" -e DOCKER_HOST_IP="$docker_ip" -i -a stdin -a stdout -a stderr $volumes $run_image \
            /bin/bash -c "$run_command") || ERR=$?

github.com/discourse/discourse_docker

launcher

0326f200e


      
          (exec cat "$tmp_input_file" | $docker_path run --shm-size=512m $user_args $links "${env[@]}" -e DOCKER_HOST_IP="$docker_ip" --cidfile $cidbootstrap -i -a stdin -a stdout -a stderr $volumes $image \
            /bin/bash -c "$run_command") || ERR=$?

sam · December 29, 2020, 5:17am

We should fix this then, but the fix is very very complicated bash script. Feel free to give this a shot.

Our docker args getting function should be responsible for shm-size. Handling user overrides and setting a default.

neounix · December 29, 2020, 5:39am

Hey @Ghan

In the meantime (for testing purposes, see caveat below), you can change this directly with docker after the container is built, as follows:

Edit the /var/lib/docker/containers/$CONTAINER_ID/hostconfig.json file directly.
For example, change the value for ShmSize in the file above.
Stop and restart the container.

In our docker container hostconfig file, it looks like this:

"ShmSize":536870912,

HTH

Caveat: Some people have posted that you must make sure that docker service is stopped before you modify hostconfig.json (or the changes will be overwritten). However, I have not actually tested this. Obviously a launcher solution is better

markersocial · November 29, 2024, 5:42pm

Anyone know what the current solution for increasing shm_size is?

Edit: Appears like it is still hardcoded.

Relevant thread:

Topic		Replies	Views
PG throws could not resize shared memory segment error Self-hosting	5	27496	April 6, 2018
Can't increase db_shared_buffers (0.1% of total ram) and db_work_mem (0.03% of total ram) Product Support	4	398	December 5, 2024
Followed github guide, vps. help me apply ram/vcpu limits Self-hosting server-resources	2	247	September 29, 2025
Change attachment size limit without rebuilding? Self-hosting	2	489	May 2, 2023
Stuck in a loop of freeing up space and filling it up again when rebuilding Self-hosting server-resources	30	4252	March 20, 2019

Docker shm_size Option in app.yml

AI-generated summary

Related topics