Docker shm_size Option in app.yml

Ghan · December 23, 2020, 2:32am

Would it be possible to make the shm_size param an option in app.yml? I ran into this error today when trying to do a vacuum analyze:

ERROR: could not resize shared memory segment

It was trying to increase the shm usage to 1 GB and failed, as it’s only 512 MB out of the box. This param looks to be in the launcher script, but it might be nice to have this as an option (possibly commented out by default) in the .yml file.

tgxworld · December 23, 2020, 4:02am

You can pass custom docker args to override the default we provide with --docker-args.

github.com

discourse/discourse_docker/blob/master/launcher#L20


  echo "    enter:       Open a shell to run commands inside the container"
  echo "    logs:        View the Docker logs for a container"
  echo "    bootstrap:   Bootstrap a container for the config based on a template"
  echo "    run:         Run the given command with the config in the context of the last bootstrapped image"
  echo "    rebuild:     Rebuild a container (destroy old, bootstrap, start new)"
  echo "    cleanup:     Remove all containers that have stopped for > 24 hours"
  echo "    start-cmd:   Generate docker command used to start container"
  echo
  echo "Options:"
  echo "    --skip-prereqs             Don't check launcher prerequisites"
  echo "    --docker-args              Extra arguments to pass when running docker"
  echo "    --skip-mac-address         Don't assign a mac address"
  echo "    --run-image                Override the image used for running the container"
  exit 1
}
# for potential re-exec later
SAVED_ARGV=("$@")
command=$1
config=$2

I believe it’ll look something like ./launcher start app --docker-args "--shm-size=1024m"

Ghan · December 23, 2020, 1:44pm

This didn’t work when I tried it unfortunately. I think maybe the container needs to be rebuild to increase the size of the shared memory space? I had to change the value in the hard-coded areas in the launcher script and then rebuild the container for it to actually take effect.

This likely wouldn’t persist across updates/rebuilds either - you’d need to be manually changing it every time.

tgxworld · December 24, 2020, 1:03am

What commands did you run? You’ll definitely have to rebuild the container for the --shm-size option to take effect.

Ghan · December 24, 2020, 2:40am

All I did at first was stop the container, then run the launcher start command with that parameter. Then I manually edited the --shm-size params inside the launcher script itself and rebuilt the container. Then it adjusted the value correctly. If you override the param on the command line to rebuild the container, then won’t the change be lost if you perform an update/rebuild through the /admin/upgrade interface? That’s why I thought it might be best to make it a field in app.yml - then it would persist through rebuilds and code updates.

tgxworld · December 24, 2020, 5:46am

Hmm I can see this being useful. @sam What do you think about this?

sam · December 29, 2020, 4:52am

That never rebuilds the container though, so it is safe.

Don’t we already have this per:

github.com

discourse/discourse_docker/blob/0326f200eef3c9b18a06f270decb541e8c0020de/launcher#L563-L570


local docker_args
docker_args=`cat $config_file | $docker_path run $user_args --rm -i -a stdout -a stdin $image ruby -e \
        "require 'yaml'; puts YAML.load(STDIN.readlines.join)['docker_args']"`
if [[ -n "$docker_args" ]]; then
  user_args="$user_args_argv $docker_args"
fi

docker_args: .... in your yaml.

Ghan · December 29, 2020, 5:12am

I’m not sure. Does that overwrite the launcher file itself? I see 3 places where the option is hardcoded:

github.com

discourse/discourse_docker/blob/0326f200eef3c9b18a06f270decb541e8c0020de/launcher#L641-L643


$docker_path run --shm-size=512m $links $attach_on_run $restart_policy "${env[@]}" "${labels[@]}" -h "$hostname" \
   -e DOCKER_HOST_IP="$docker_ip" --name $config -t "${ports[@]}" $volumes $mac_address $user_args \
   $run_image $boot_command

github.com

discourse/discourse_docker/blob/0326f200eef3c9b18a06f270decb541e8c0020de/launcher#L657-L658


(exec $docker_path run --rm --shm-size=512m $user_args $links "${env[@]}" -e DOCKER_HOST_IP="$docker_ip" -i -a stdin -a stdout -a stderr $volumes $run_image \
  /bin/bash -c "$run_command") || ERR=$?

github.com

discourse/discourse_docker/blob/0326f200eef3c9b18a06f270decb541e8c0020de/launcher#L702-L703


(exec cat "$tmp_input_file" | $docker_path run --shm-size=512m $user_args $links "${env[@]}" -e DOCKER_HOST_IP="$docker_ip" --cidfile $cidbootstrap -i -a stdin -a stdout -a stderr $volumes $image \
  /bin/bash -c "$run_command") || ERR=$?

sam · December 29, 2020, 5:17am

We should fix this then, but the fix is very very complicated bash script. Feel free to give this a shot.

Our docker args getting function should be responsible for shm-size. Handling user overrides and setting a default.

neounix · December 29, 2020, 5:46am

Hey @Ghan

In the meantime (for testing purposes, see caveat below), you can change this directly with docker after the container is built, as follows:

Edit the /var/lib/docker/containers/$CONTAINER_ID/hostconfig.json file directly.
For example, change the value for ShmSize in the file above.
Stop and restart the container.

In our docker container hostconfig file, it looks like this:

"ShmSize":536870912,

HTH

Caveat: Some people have posted that you must make sure that docker service is stopped before you modify hostconfig.json (or the changes will be overwritten). However, I have not actually tested this. Obviously a launcher solution is better