Completely forgot to set the proxy headers, added in the missing headers and all works well.
Thanks!
In case someone runs into this issue, the proxy pass headers I was missing:
proxy_set_header HOST $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;