Have a look at discourse_docker/templates/web.template.yml at main · discourse/discourse_docker · GitHub for some examples of modifying the nginx config.
What I usually do is modify the config files by hand inside the container to test before making changes to the yml, as it takes a long time to rebuild to see if your changes worked.