How can I update the nginx config directly in discourse/docker ..?

You might find a similar technique to edit the config in this old advice works for you:

(taken from a prior version of this wiki post: Set up Let’s Encrypt with multiple domains)

"It’s pretty simple really, add something like the following to your app.yml hooks section:

Step 2 - discourse.conf

In addition to step 1 there is a change needed for nginx to correctly redirect http to https for anything other than DISCOURSE_HOSTNAME

    - replace:
        filename: "/etc/nginx/conf.d/discourse.conf"
        from: /return 301 https.+/
        to: |
          return 301 https://$host$request_uri;

We also need to remove the additional rewrite that web.ssl.template.yml adds since it forces only the DISCOURSE_HOSTNAME and now we have others!

    - replace:
        filename: "/etc/nginx/conf.d/discourse.conf"
        from: /gzip on;[^\}]+\}/m
        to: |
          gzip on;
          add_header Strict-Transport-Security 'max-age=31536000'; # remember the certificate for a year and automatically connect to HTTPS for this domain