Prometheus scrape job cannot reach the metrics

Good find but if I try to target that specific port I get a “connection refused” message.

Get "http://discourse_app:9405/metrics": dial tcp 172.20.0.2:9405: connect: connection refused

Tested with a wget from inside the prometheus container as well just to be sure.

/prometheus # ping discourse_app
PING discourse_app (172.20.0.2): 56 data bytes
64 bytes from 172.20.0.2: seq=0 ttl=64 time=0.223 ms
64 bytes from 172.20.0.2: seq=1 ttl=64 time=0.270 ms
^C
--- discourse_app ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.223/0.246/0.270 ms
/prometheus # wget discourse_app:9405/metrics
Connecting to discourse_app:9405 (172.20.0.2:9405)
wget: can't connect to remote host (172.20.0.2): Connection refused

Yep, tested with wget instead (prometheus container is a barebone busybox) but got to the metrics nonetheless.

So what you are saying is that I should find a way to have the container running prometheus have an entry in the /etc/hosts that resolve… I’ve lost you there sorry :slight_smile:

What I did is adding yet another docker with simply an nginx in it and provide a forward proxy configuration that add the header Host to the requests that it receives. It doesn’t expose any port so it can only be accessed by the internal virtual network anyway.

So how things change?

Prometheus Job:

  - job_name: discourse_exporter_proxy
    scheme: http
    static_configs:
      - targets:
        - 'discourse_forward_proxy:8080'

docker-compose.yaml (just the part with the proxy)

version: "3"

services:
# [...]
    discourse_forward_proxy:
        image: nginx:latest
        container_name: discourse_forward_proxy
        restart: unless-stopped
        volumes:
            - ./discourse_forward_proxy/:/etc/nginx/conf.d
        networks:
            - prometheus-discourse_forward_proxy
            - discourse
# [...]

networks:
    prometheus-discourse_forward_proxy:
        internal: true
    discourse:
        external: true

In the directory in which your docker-compose.yaml is, have ./discourse_forward_proxy/discourse_forward_proxy.conf

server {
    listen 8080;

    location /metrics {
      proxy_set_header Host "YOUR_DOMAIN_HERE.COM";
      proxy_pass https://discourse_app/metrics;
    }
}

There you go:

1 Like