Mixed SSL since upgrade

Andrew_Bond · December 30, 2022, 12:17pm

I have(had!) a default working reverse proxy install.
Everything was being served by HTTPS as designed.

I have jumped up a few versions and experienced the upgrade of death where the screen remains blank as reported here : Upgrade discourse doesn't work - #6 by Andrew_Bond

I have manually updated my discourse with the ./launcher rebuild app and am on build 2.9.0.beta14(0da79561c3)

I noted that if a draft tried to save, or most other actions such as post, logging in I recieved a 403 forbidden error ( via the chrome inspector )

My logo works in the top left of the site though.

If I manually disable the force https element via command line the logo fails to load as I am in mixed mode where the http element is blocked. All other elements of the site continue to work.

I have not made any reverse nginx changes. I have tried to rebuild the app again with no joy.

I am suggesting that something changed with the way discourse operates on the builds I have upgraded from. This was the last build ID I can find that worked with the force https option working : e9f53dbe0028ca7a5d8926711a1944765d34347

Errors trapped now :

david · December 30, 2022, 12:36pm

Hi Andrew - thanks for the report. This issue specifically affects reverse-proxy configurations where multiple X-Forwarded-Proto headers are added to the request before it hits the application container.

I’ve just merged a fix into main - it should hit tests-passed in about 20 minutes

github.com/discourse/discourse

FIX: Update nginx config for v1.23 (#19651)

committed 12:35PM - 30 Dec 22 UTC

davidtaylorhq

+1 -1

NGINX v1.23 concatenates duplicate headers into a single comma-separated string.… We were doing an equality check on `x-forwarded-proto`. If the request includes multiple `x-forwarded-proto` headers then this check would fail under NGINX v1.23, and our config assumed an `http` connection. This commit updates the config to check for `https` at the end of the header, thereby restoring the old behavior when multiple `x-forwarded-proto` request headers are sent.

Once it appears in tests-passed you’ll need to do a full rebuild (nginx config changes are not applied during UI-based upgrades).

Andrew_Bond · December 30, 2022, 12:43pm

Many thanks for the speedy reply and the fast fix!

Andrew_Bond · December 30, 2022, 2:37pm

So, update.

I have rebuilt the app with ./launcher rebuild app

No problems there.

The errors have gone when I force HTTPS with the exception of

I no longer have 403 denied but for example when trying to login I am prompted with the login box which seems to accept the data then presented again with an unlogged in user

The same error as above is present when reloading the page.

I have again disabled force HTTPS and the site works - for clarity though logos are still not being offered

david · December 30, 2022, 3:26pm

That error is unrelated - it’s caused by cloudflare injecting analytics scripts (and Discourse’s default Content Security Policy blocking the injection)

Can you share more about the proxies that you have between users and Discourse? I suppose Cloudflare is the first layer? What else is between cloudflare and the Discourse app container?

Andrew_Bond · December 30, 2022, 3:27pm

Sure!

You are quite correct. Cloudflare is the first hit then we have nginx reverse proxy offered by centminmod onto the docker.

david · December 30, 2022, 3:29pm

Can you share the config you have for this? In particular, does it add an additional X-Forwarded-Proto header?

Andrew_Bond · December 30, 2022, 3:33pm

Sure!

# Centmin Mod Getting Started Guide
# must read https://centminmod.com/getstarted.html
# For HTTP/2 SSL Setup
# read https://centminmod.com/nginx_configure_https_ssl_spdy.html

# redirect from www to non-www  forced SSL
# uncomment, save file and restart Nginx to enable
# if unsure use return 302 before using return 301
 server {
       listen   80;
       server_name exiges.com www.exiges.com;
       return 302 https://$server_name$request_uri;
 }

server {
  listen 443 ssl http2;
  server_name exiges.com www.exiges.com;

  ssl_dhparam /usr/local/nginx/conf/ssl/exiges.com/dhparam.pem;
  ssl_certificate      /usr/local/nginx/conf/ssl/exiges.com/exiges.com.crt;
  ssl_certificate_key  /usr/local/nginx/conf/ssl/exiges.com/exiges.com.key;
  include /usr/local/nginx/conf/ssl_include.conf;

  # cloudflare authenticated origin pull cert community.centminmod.com/threads/13847/
  #ssl_client_certificate /usr/local/nginx/conf/ssl/cloudflare/exiges.com/origin.crt;
  #ssl_verify_client on;
  
  
  
  # mozilla recommended
  ssl_ciphers TLS13-AES-128-GCM-SHA256:TLS13-AES-256-GCM-SHA384:TLS13-CHACHA20-POLY1305-SHA256:TLS13-AES-128-CCM-8-SHA256:TLS13-AES-128-CCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;
  ssl_prefer_server_ciphers   on;
  #add_header Alternate-Protocol  443:npn-spdy/3;

  # before enabling HSTS line below read centminmod.com/nginx_domain_dns_setup.html#hsts
  #add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
  #add_header X-Frame-Options SAMEORIGIN;
  add_header X-Xss-Protection "1; mode=block" always;
  add_header X-Content-Type-Options "nosniff" always;
  #add_header Referrer-Policy "strict-origin-when-cross-origin";
  #add_header Permissions-Policy "accelerometer=(), camera=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), payment=(), usb=()";
  #spdy_headers_comp 5;
  ssl_buffer_size 1369;
  ssl_session_tickets on;
  
  # enable ocsp stapling
  #resolver 8.8.8.8 8.8.4.4 1.1.1.1 1.0.0.1 valid=10m;
  #resolver_timeout 10s;
  #ssl_stapling on;
  #ssl_stapling_verify on;
  #ssl_trusted_certificate /usr/local/nginx/conf/ssl/exiges.com/exiges.com-trusted.crt;

# ngx_pagespeed & ngx_pagespeed handler
#include /usr/local/nginx/conf/pagespeed.conf;
#include /usr/local/nginx/conf/pagespeedhandler.conf;
#include /usr/local/nginx/conf/pagespeedstatslog.conf;

  # limit_conn limit_per_ip 16;
  # ssi  on;

  access_log /home/nginx/domains/exiges.com/log/access.log combined buffer=256k flush=5m;
  error_log /home/nginx/domains/exiges.com/log/error.log;

  include /usr/local/nginx/conf/autoprotect/exiges.com/autoprotect-exiges.com.conf;
  root /home/nginx/domains/exiges.com/public;
  # uncomment cloudflare.conf include if using cloudflare for
  # server and/or vhost site
  #include /usr/local/nginx/conf/cloudflare.conf;
  include /usr/local/nginx/conf/503include-main.conf;

 
location / {
       proxy_pass http://unix:/var/discourse/shared/standalone/nginx.http.sock:;
       proxy_set_header        X-Forwarded-Proto https;
       proxy_http_version 1.1;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Real-IP $remote_addr;
   }   

  include /usr/local/nginx/conf/php.conf;
  
  #include /usr/local/nginx/conf/pre-staticfiles-local-exiges.com.conf;
  #include /usr/local/nginx/conf/pre-staticfiles-global.conf;
  #include /usr/local/nginx/conf/staticfiles.conf;
  #include /usr/local/nginx/conf/drop.conf;
  #include /usr/local/nginx/conf/errorpage.conf;
  #include /usr/local/nginx/conf/vts_server.conf;
}

I see the reference for the x-proto header …

   proxy_set_header X-Forwarded-Proto $scheme;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

david · December 30, 2022, 4:01pm

How is Cloudflare connecting to your origin? Is it using SSL/TLS? Ideally you should be using “Full” or “Full (Strict)” here.

Andrew_Bond · December 30, 2022, 5:19pm

Flexible. I have been lazy and not used server certs

david · December 30, 2022, 5:28pm

I’d recommend you get set up with one of the secure Cloudflare SSL options. “Flexible” means that traffic is not encrypted between Cloudflare and your server.

Taking a look at the config again:

Looks like you’re setting the header to https, and then setting it again two lines below to $scheme (which will be http based on your Cloudflare config). If you remove the second one, then Discourse will be told the connection is secure (even though it’s not).

Andrew_Bond · December 30, 2022, 7:31pm

Interesting.

What I find strange though is it was working.

david · December 30, 2022, 7:50pm

I suspect (but haven’t 100% confirmed) that the Discourse container NGINX was previously looking at the ‘first’ X-Forwarded-Proto header. Now, it is looking at the ‘last’ X-Forwarded-Proto header, which is arguably safer because it is comes from ‘most recent’ proxy which has the most information about the incoming request.

So, before the update, it was looking at the X-Forwarded-Proto introduced by Cloudflare (https). Now, it’s looking at the X-Forwarded-Proto header introduced by your ‘centminmod’ reverse proxy configuration (http).

Andrew_Bond · December 30, 2022, 10:08pm

I have removed proxy_set_header X-Forwarded-Proto $scheme; from the config with a now working config.

For anyone out there who is reading and configing a centminmod there are TWO configs for the nginx.

One for the http and one for the https

I have removed the second proxy_set_header X-Forwarded-Proto $scheme; from the http site, and everything is working as expected now .

This is my exiges.com.conf

# Centmin Mod Getting Started Guide
# must read https://centminmod.com/getstarted.html

# redirect from non-www to www 
# uncomment, save file and restart Nginx to enable
# if unsure use return 302 before using return 301
#server {
#            listen   80;
#            server_name exiges.com;
#            return 301 $scheme://www.exiges.com$request_uri;
#       }

server {
  server_name exiges.com www.exiges.com;
#return 301 https://exiges.com.com$request_uri;
#return 301 $scheme://www.exiges.com$request_uri;

# ngx_pagespeed & ngx_pagespeed handler
#include /usr/local/nginx/conf/pagespeed.conf;
#include /usr/local/nginx/conf/pagespeedhandler.conf;
#include /usr/local/nginx/conf/pagespeedstatslog.conf;

  #add_header X-Frame-Options SAMEORIGIN;
  add_header X-Xss-Protection "1; mode=block" always;
  add_header X-Content-Type-Options "nosniff" always;
  #add_header Referrer-Policy "strict-origin-when-cross-origin";
  #add_header Permissions-Policy "accelerometer=(), camera=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), payment=(), usb=()";

  # limit_conn limit_per_ip 16;
  # ssi  on;

  access_log /home/nginx/domains/exiges.com/log/access.log combined buffer=256k flush=5m;
  error_log /home/nginx/domains/exiges.com/log/error.log;

  include /usr/local/nginx/conf/autoprotect/exiges.com/autoprotect-exiges.com.conf;
  root /home/nginx/domains/exiges.com/public;
  # uncomment cloudflare.conf include if using cloudflare for
  # server and/or vhost site
  #include /usr/local/nginx/conf/cloudflare.conf;
  include /usr/local/nginx/conf/503include-main.conf;

  # prevent access to ./directories and files
  #location ~ (?:^|/)\. {
  # deny all;
  #}

  location / {
 proxy_pass http://unix:/var/discourse/shared/standalone/nginx.http.sock:;
		    proxy_set_header        X-Forwarded-Proto https;	
                #proxy_set_header Host $http_host;
                proxy_http_version 1.1;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                #proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header X-Real-IP $remote_addr;
  #include /usr/local/nginx/conf/503include-only.conf;

# block common exploits, sql injections etc
#include /usr/local/nginx/conf/block.conf;

  # Enables directory listings when index file not found
  #autoindex  on;

  # Shows file listing times as local time
  #autoindex_localtime on;

  # Wordpress Permalinks example
  #try_files $uri $uri/ /index.php?q=$uri&$args;

  }

  include /usr/local/nginx/conf/php.conf;
  
  #include /usr/local/nginx/conf/pre-staticfiles-local-exiges.com.conf;
  #include /usr/local/nginx/conf/pre-staticfiles-global.conf;
  #include /usr/local/nginx/conf/staticfiles.conf;
  #include /usr/local/nginx/conf/drop.conf;
  #include /usr/local/nginx/conf/errorpage.conf;
  #include /usr/local/nginx/conf/vts_server.conf;
}

This is my exiges.com.conf.ssl

# Centmin Mod Getting Started Guide
# must read https://centminmod.com/getstarted.html
# For HTTP/2 SSL Setup
# read https://centminmod.com/nginx_configure_https_ssl_spdy.html

# redirect from www to non-www  forced SSL
# uncomment, save file and restart Nginx to enable
# if unsure use return 302 before using return 301
 server {
       listen   80;
       server_name exiges.com www.exiges.com;
       return 302 https://$server_name$request_uri;
 }

server {
  listen 443 ssl http2;
  server_name exiges.com www.exiges.com;

  ssl_dhparam /usr/local/nginx/conf/ssl/exiges.com/dhparam.pem;
  ssl_certificate      /usr/local/nginx/conf/ssl/exiges.com/exiges.com.crt;
  ssl_certificate_key  /usr/local/nginx/conf/ssl/exiges.com/exiges.com.key;
  include /usr/local/nginx/conf/ssl_include.conf;

  # cloudflare authenticated origin pull cert community.centminmod.com/threads/13847/
  #ssl_client_certificate /usr/local/nginx/conf/ssl/cloudflare/exiges.com/origin.crt;
  #ssl_verify_client on;
  
  
  
  # mozilla recommended
  ssl_ciphers TLS13-AES-128-GCM-SHA256:TLS13-AES-256-GCM-SHA384:TLS13-CHACHA20-POLY1305-SHA256:TLS13-AES-128-CCM-8-SHA256:TLS13-AES-128-CCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;
  ssl_prefer_server_ciphers   on;
  #add_header Alternate-Protocol  443:npn-spdy/3;

  # before enabling HSTS line below read centminmod.com/nginx_domain_dns_setup.html#hsts
  #add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
  #add_header X-Frame-Options SAMEORIGIN;
  add_header X-Xss-Protection "1; mode=block" always;
  add_header X-Content-Type-Options "nosniff" always;
  #add_header Referrer-Policy "strict-origin-when-cross-origin";
  #add_header Permissions-Policy "accelerometer=(), camera=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), payment=(), usb=()";
  #spdy_headers_comp 5;
  ssl_buffer_size 1369;
  ssl_session_tickets on;
  
  # enable ocsp stapling
  #resolver 8.8.8.8 8.8.4.4 1.1.1.1 1.0.0.1 valid=10m;
  #resolver_timeout 10s;
  #ssl_stapling on;
  #ssl_stapling_verify on;
  #ssl_trusted_certificate /usr/local/nginx/conf/ssl/exiges.com/exiges.com-trusted.crt;

# ngx_pagespeed & ngx_pagespeed handler
#include /usr/local/nginx/conf/pagespeed.conf;
#include /usr/local/nginx/conf/pagespeedhandler.conf;
#include /usr/local/nginx/conf/pagespeedstatslog.conf;

  # limit_conn limit_per_ip 16;
  # ssi  on;

  access_log /home/nginx/domains/exiges.com/log/access.log combined buffer=256k flush=5m;
  error_log /home/nginx/domains/exiges.com/log/error.log;

  include /usr/local/nginx/conf/autoprotect/exiges.com/autoprotect-exiges.com.conf;
  root /home/nginx/domains/exiges.com/public;
  # uncomment cloudflare.conf include if using cloudflare for
  # server and/or vhost site
  include /usr/local/nginx/conf/cloudflare.conf;
  include /usr/local/nginx/conf/503include-main.conf;

 
location / {
       proxy_pass http://unix:/var/discourse/shared/standalone/nginx.http.sock:;
       proxy_set_header        X-Forwarded-Proto https;
       proxy_http_version 1.1;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Real-IP $remote_addr;
   }   

  include /usr/local/nginx/conf/php.conf;
  
  #include /usr/local/nginx/conf/pre-staticfiles-local-exiges.com.conf;
  #include /usr/local/nginx/conf/pre-staticfiles-global.conf;
  #include /usr/local/nginx/conf/staticfiles.conf;
  #include /usr/local/nginx/conf/drop.conf;
  #include /usr/local/nginx/conf/errorpage.conf;
  #include /usr/local/nginx/conf/vts_server.conf;
}

@david your time is hugely appreciated in fixing this issue.

For what it is worth I am now full , not flexible with the certs.

system · January 29, 2023, 10:09pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Add an offline page to display when Discourse is rebuilding or starting up Sysadmins configuring , how-to , advanced-setup	135	242102	January 5, 2024
Avatar, Site Logos, and Cert Errors Installation unsupported-install	61	3619	May 25, 2020
Problem with my SSL certificate Installation	17	11233	June 8, 2024
Discourse with jwilder/nginx-proxy howtos? Installation	12	7786	June 19, 2022
Difference between socket- and port-based connection to outer NGINX? Installation	10	12329	December 28, 2019

Mixed SSL since upgrade

Related Topics