Prometheus scrape job cannot reach the metrics

Product Support Self-hosting
16

That’s due to the next line:

GlobalSetting.add_default :prometheus_collector_port, 9405
GlobalSetting.add_default :prometheus_webserver_bind, "localhost"
GlobalSetting.add_default :prometheus_trusted_ip_allowlist_regex, ""

Binding to localhost means that it can only be connected to on the localhost IP, which is why connecting to 172.20.0.2 fails. This is a security measure to ensure it doesn’t accidentally get exposed to a much wider audience than intended.

If you set in the container definition file:

  DISCOURSE_PROMETHEUS_WEBSERVER_BIND: '*'

It’ll listen on all IP addresses and you’ll be able to connect to it from another container.

The reason why this made it work:

server {
    listen 8080;

    location /metrics {
      proxy_set_header Host "YOUR_DOMAIN_HERE.COM";
      proxy_pass https://discourse_app/metrics;
    }
}

is because this nginx container is now talking to prometheus over the localhost IP.

If you’re not sure of the IPs or ports on which services are listening, you can use ss -ltp or netstat -ltp (inside the container! necessary packages are net-tools and iproute2 respectively) to look at them. For instance, I just rebuilt a container with the prometheus plugin and see:

root@discourse-docker-app:/# ss -ltp
State      Recv-Q     Send-Q           Local Address:Port                 Peer Address:Port     Process                             
LISTEN     0          128                  127.0.0.1:3000                      0.0.0.0:*                                            
LISTEN     0          128                    0.0.0.0:postgresql                0.0.0.0:*                                            
LISTEN     0          128                    0.0.0.0:https                     0.0.0.0:*         users:(("nginx",pid=555,fd=7))     
LISTEN     0          128                  127.0.0.1:9405                      0.0.0.0:*                                            
LISTEN     0          128                    0.0.0.0:redis                     0.0.0.0:*                                            
LISTEN     0          128                    0.0.0.0:http                      0.0.0.0:*         users:(("nginx",pid=555,fd=6))     
LISTEN     0          128                       [::]:postgresql                   [::]:*                                            
LISTEN     0          128                       [::]:https                        [::]:*         users:(("nginx",pid=555,fd=8))     
LISTEN     0          128                       [::]:redis                        [::]:*

root@discourse-docker-app:/# curl http://172.17.0.2:9405/metrics
curl: (7) Failed to connect to 172.17.0.2 port 9405: Connection refused

root@discourse-docker-app:/# curl http://localhost:9405/metrics
# HELP discourse_collector_working Is the master process collector able to collect metrics
# TYPE discourse_collector_working gauge
discourse_collector_working 1


# HELP discourse_collector_rss total memory used by collector process
# TYPE discourse_collector_rss gauge
discourse_collector_rss 38178816
…

That’s the nameserver rejecting the IP lookup request for vmi1187507-app. Port 53 is DNS.