# Prometheus scrape job cannot reach the metrics

**URL:** <https://meta.discourse.org/t/prometheus-scrape-job-cannot-reach-the-metrics/264902>\
**Category:** Self-hosting\
**Tags:** prometheus\
**Created:** [May 13, 2023, 5:23pm UTC](https://meta.discourse.org/t/prometheus-scrape-job-cannot-reach-the-metrics/264902 "2023-05-13T17:23:40Z")\
**Posts on this page:** 1\
**Showing post:** 16

<div class="post-metadata">

**Author:** ![supermathie](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/supermathie/32/507518_2.png) [@supermathie](https://meta.discourse.org/u/supermathie)\
**Post date:** [May 16, 2023, 2:56pm UTC](https://meta.discourse.org/t/prometheus-scrape-job-cannot-reach-the-metrics/264902/16 "2023-05-16T14:56:43Z")

</div>

> [@Crius](#):
>
> Get “http://discourse\_app:9405/metrics”: dial tcp 172.20.0.2:9405: connect: connection refused

That’s due to the next line:

```plaintext
GlobalSetting.add_default :prometheus_collector_port, 9405
GlobalSetting.add_default :prometheus_webserver_bind, "localhost"
GlobalSetting.add_default :prometheus_trusted_ip_allowlist_regex, ""

```

Binding to localhost means that it can only be connected to on the localhost IP, which is why connecting to `172.20.0.2` fails. This is a security measure to ensure it doesn’t accidentally get exposed to a much wider audience than intended.

If you set in the container definition file:

```plaintext
  DISCOURSE_PROMETHEUS_WEBSERVER_BIND: '*'

```

It’ll listen on all IP addresses and you’ll be able to connect to it from another container.

The reason why this made it work:

```plaintext
server {
    listen 8080;

    location /metrics {
      proxy_set_header Host "YOUR_DOMAIN_HERE.COM";
      proxy_pass https://discourse_app/metrics;
    }
}

```

is because this nginx container is now talking to prometheus over the `localhost` IP.

If you’re not sure of the IPs or ports on which services are listening, you can use `ss -ltp` or `netstat -ltp` (_inside_ the container! necessary packages are `net-tools` and `iproute2` respectively) to look at them. For instance, I just rebuilt a container with the prometheus plugin and see:

```plaintext
root@discourse-docker-app:/# ss -ltp
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process                             
LISTEN 0 128 127.0.0.1:3000 0.0.0.0:*                                            
LISTEN 0 128 0.0.0.0:postgresql 0.0.0.0:*                                            
LISTEN 0 128 0.0.0.0:https 0.0.0.0:* users:(("nginx",pid=555,fd=7))     
LISTEN 0 128 127.0.0.1:9405 0.0.0.0:*                                            
LISTEN 0 128 0.0.0.0:redis 0.0.0.0:*                                            
LISTEN 0 128 0.0.0.0:http 0.0.0.0:* users:(("nginx",pid=555,fd=6))     
LISTEN 0 128 [::]:postgresql [::]:*                                            
LISTEN 0 128 [::]:https [::]:* users:(("nginx",pid=555,fd=8))     
LISTEN 0 128 [::]:redis [::]:*

root@discourse-docker-app:/# curl http://172.17.0.2:9405/metrics
curl: (7) Failed to connect to 172.17.0.2 port 9405: Connection refused

root@discourse-docker-app:/# curl http://localhost:9405/metrics
# HELP discourse_collector_working Is the master process collector able to collect metrics
# TYPE discourse_collector_working gauge
discourse_collector_working 1

# HELP discourse_collector_rss total memory used by collector process
# TYPE discourse_collector_rss gauge
discourse_collector_rss 38178816
…

```

> [@Crius](#):
>
> No idea where that `127.0.0.11:53` is coming from to be honest 🤔

That’s the nameserver rejecting the IP lookup request for `vmi1187507-app`. Port 53 is DNS.

---

_[View the full topic](https://meta.discourse.org/t/prometheus-scrape-job-cannot-reach-the-metrics/264902)._