Launcher rebuild fails with Docker 19.03.0 due to Docker network parameter

With docker rolling out version 19.03.0, the discourse launcher is now failing to start the instance after rebuilding with the following message:

/usr/bin/docker: --network is not a valid mac address.
See '/usr/bin/docker run --help'.

This is the command run (some items changed for security):

+ /usr/bin/docker run --shm-size=512m -d --restart=always -e LANG=en_US.UTF-8 -e RAILS_ENV=production -e UNICORN_WORKERS=2 -e UNICORN_SIDEKIQS=1 -e RUBY_GLOBAL_METHOD_CACHE_SIZE=131072 -e RUBY_GC_HEAP_GROWTH_MAX_SLOTS=40000 -e RUBY_GC_HEAP_INIT_SLOTS=400000 -e RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=1.5 -e DISCOURSE_DB_SOCKET=/var/run/postgresql -e DISCOURSE_DB_HOST= -e DISCOURSE_DB_PORT= -e DISCOURSE_HOSTNAME=discourse.mydomain.com -e VIRTUAL_HOST=discourse.mydomain.com -e DISCOURSE_DEVELOPER_EMAILS=admin@mydomain.com -e DISCOURSE_SMTP_ADDRESS=mail.mydomain.com -e DISCOURSE_SMTP_PORT=587 -e DISCOURSE_SMTP_USER_NAME=discourse@mydomain.com -e DISCOURSE_SMTP_PASSWORD=MySuperDuperPass -h vps-discourse -e DOCKER_HOST_IP=172.17.0.1 --name discourse -t --expose 80 --mac-address --network reverse-proxy --network reverse-proxy local_discourse/discourse /sbin/boot
/usr/bin/docker: --network is not a valid mac address.
See '/usr/bin/docker run --help'.

I am using this with behind a reverse proxy (running on network named reverse-proxy). It has always worked, until today. It seems it expects a mac address value that is not being provided.

2 Likes

Let’s see…

root@endoffice-a:/var/discourse# docker --version
Docker version 19.03.0, build aeac949

I just did a ./launcher rebuild app and it went OK:

+ /usr/bin/docker run --shm-size=512m -d --restart=always -e LANG=en_US.UTF-8 -e RAILS_ENV=production -e UNICORN_WORKERS=8 -e UNICORN_SIDEKIQS=1 -e RUBY_GLOBAL_METHOD_CACHE_SIZE=131072 -e RUBY_GC_HEAP_GROWTH_MAX_SLOTS=40000 -e RUBY_GC_HEAP_INIT_SLOTS=400000 -e RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=1.5 -e DISCOURSE_DB_SOCKET=/var/run/postgresql -e DISCOURSE_DB_HOST= -e DISCOURSE_DB_PORT= -e LETSENCRYPT_DIR=/shared/letsencrypt -e UNICORN_SIDEKIQ_MAX_RSS=1000 -e DISCOURSE_HOSTNAME=discourse.codinghorror.com -e DISCOURSE_DEVELOPER_EMAILS=jatwood@codinghorror.com -e DISCOURSE_SMTP_ADDRESS=smtp.mailgun.org -e DISCOURSE_SMTP_PORT=587 -e DISCOURSE_SMTP_USER_NAME={redacted} -e DISCOURSE_SMTP_PASSWORD={redacted} -e LETSENCRYPT_ACCOUNT_EMAIL={redacted} -e DISCOURSE_CDN_URL=https://discourse-cdn.codinghorror.com -h endoffice-a-app -e DOCKER_HOST_IP=172.17.0.1 --name app -t -p 80:80 -p 443:443 -v /var/discourse/shared/standalone:/shared -v /var/discourse/shared/standalone/log/var-log:/var/log --mac-address 02:72:e3:65:d5:32 local_discourse/app /sbin/boot

Note that --mac-address does appear and is populated, but that --network line is specific to your setup and deviates from a standard standalone Discourse install.

3 Likes

@codinghorror That’s all well and great, but then, if standard standalone installs is the only way Discourse is supposed to operate, why provide support for yml container configuration in ./containers, and including examples of this in ./samples? The file I used is fully commented with instructions on operation, specifically this area:

# any extra arguments for Docker?
# docker_args:

The only thing I am using is:

docker_args: "--network reverse-proxy"

Which again, has been working great for a year now. So, obviously something has changed. Please understand I am not trying to be confrontational (it would be kind of silly since I am asking for help), but I am indeed trying to make the point that I am using options that are made available through the Discourse samples.

Perhaps if I understood how that mac address is being generated it could point me to an environment change issue, that’s what I’m hoping.

I see this in launcher:

     if [ -z "$SKIP_MAC_ADDRESS" ] ; then
      mac_address="--mac-address $($docker_path run $user_args -i --rm -a stdout -a stderr $image /bin/sh -c "echo $hostname | md5sum | sed 's/^\(..\)\(..\)\(..\)\(..\)\(..\).*$/02:\1:\2:\3:\4:\5/'")"
     fi

Will dissect and see if I can figure it out.

UPDATE: And looking at this, I see that I can use --skip-mac-address. Doing this brings up my instance just fine, so it seems the process that figures out the mac address is the issue.

4 Likes

So, in summary. The build operation succeeds no matter what, but in the build operation (./launcher rebuild myinstance), the last step of starting the container fails due to the mac address somehow not being set.

However, I can simply do ./launcher start myinstance after that and then it starts fine, and the mac address is there as well, It’s strange that’s all.

2 Likes

I have the same problem.
@codinghorror Here are the steps to reproduce and a lead to fix this regression

Step to reproduct:

  1. Set in containers config (app):
    docker_args: "--network reverse-proxy"
  2. rebuild
    ./launcher rebuild app
  3. see error
    /usr/bin/docker: --network is not a valid mac address

It’s a bug introduced by commit bfc79e7 (GitHub - discourse/discourse_docker: A Docker image for Discourse)
merge_user_args is called by set_template_info, but set_template_info is called twice when rebuild (run_run and run_bootstrap)
So instead of user_args equals $user_args $docker_args, we have $user_args $docker_args $docker_args

Docker doesn’t allow to configure the same network mutiple times
docker: network "reverse-proxy" is specified multiple times

Thank you to roolback this commit or make idempotent merge_user_args

5 Likes

Thanks for the detective work! :male_detective: @saj looks like you cosigned that commit along with @deargle

https://github.com/discourse/discourse_docker/commit/bfc79e773024fbbc80af3253b7a5aaa904a26079

3 Likes

Here’s a patch -

https://github.com/discourse/discourse_docker/pull/438

With that patch applied, and the docker_args key set in the container definition, the final docker run argument vector looks something like this:

/usr/bin/docker run --shm-size=512m -d --restart=always -e ... -h saj-launcher-testing-app -e DOCKER_HOST_IP=172.17.0.1 --name app -t -p 80:80 -p 443:443 -v /var/discourse/shared/standalone:/shared -v /var/discourse/shared/standalone/log/var-log:/var/log --mac-address 02:07:fa:c6:c4:82 --network foonet local_discourse/app /sbin/boot
5 Likes

Cool! So, glad I was not dreaming this. Thank you guys for looking into this.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.