Getting Discourse to see the Prometheus server IP

Installation
1

Hello everyone!

So far, If I understand correctly, To ensure a prometheus instance can scrap data from /metrics endpoint, We have to add IP address of that machine in the allow list by editing DISCOURSE_PROMETHEUS_TRUSTED_IP_WHITELIST_REGEX.

Is there any other way to do this? Our discourse instance sits behind cloudflare’s DNS proxy and doesn’t get to see the actual IP address of end users. I’d really prefer if maybe we can figure out a way to do this with API keys? Although that seems unlikely because of the restrictions in how prometheus can ingest data.

2

Then Discourse is broken and you need to add the cloudflare template.

I think that would work. Did you try? Oh, but I don’t think that you can get grafana to put the keys in the header, which is why I added this feature in the first place.

1 Like
3

Then Discourse is broken and you need to add the cloudflare template.

Ah, What do you mean by “cloudflare template”?

I think that would work. Did you try? Oh, but I don’t think that you can get grafana to put the keys in the header, which is why I added this feature in the first place.

Yes, I did. Prometheus still doesn’t allow arbitrary headers in scraper configs. So, There is no way for me to intake discourse metrics in prometheus if I can’t whitelist prometheus server’s IP address which is not possible in my case.

There are probably other people who run their discourse instances behind proxies like cloudflare. So, With that assumption in mind, I believe we’ll have to tweak the exporter a little bit.

Discourse’s API uses it’s own Api-Key and Api-Username headers instead of something that is somewhat standardized like http basic auth(which is supported by prometheus scraper subsystem). So, I really have no way to use this in my setup.

4

You’ll need to search. But you can find it inside the templates directory and add it to your app.yml.

1 Like
5

Thanks!

Enabled cloudflare.template.yml. :slight_smile: But if I understand correctly, That was just to disable rate limiting on traffic coming from cloudflare’s IP addresses.

For now, This is what my prometheus config looks like,

scrape_configs:
        - job_name: forum
          scrape_interval: 5s
          scheme: https
          static_configs:
                  - targets:
                          - forum-behind-cloudflare-dns-proxy.com

Now, The issue I am facing is, Discourse doesn’t get to see the actual IP address of a request. So, For the allow list approach to work, I have tried this.

  1. Added my prometheus server’s IPv6 address to the DISCOURSE_PROMETHEUS_TRUSTED_IP_WHITELIST_REGEX env variable in app.yml.

  2. Hard coded forum’s actual IPv6 address in /etc/hosts on the prometheus server. Now, The forum can see the IP address of my prometheus server and allow access to it. :slight_smile:

I still have some other problems, Like, I am running prometheus in a container. `/etc/hosts/ from host is not shared inside the container. So, It keeps resolving forum address to a cloudflare IP and fails during authentication.

I can share /etc/hosts from host in docker container like, -v /etc/hosts:/etc/hosts while starting up prometheus but that results in a error like,

Get "https://forum-behind-cloudflare-dns-proxy.com:443/metrics": dial tcp [<ipv6-address>]:443: connect: cannot assign requested address

Now, I just have to resolve this problem.

Edit #1: Figured this out as well. IPv6 was disabled inside docker container. I can fix it by enabling IPv6 in docker container or just --net=host.

6

Are you behind a reverse proxy? You need to configure nginx to let the real remote up get to discourse. I think Running other websites on the same machine as Discourse might give you a hint on getting that figured out.

1 Like
7

The discourse instance is indeed running behind a reverse proxy. The reverse proxy is offered by Cloudflare and we are only on the Pro plan and don’t have the option to pass on real client IP up to the discourse instance from cloudflare. Anyyway, My problem is already resolved. :slight_smile:

The grafana dashboard probably should be updated at this point. Over time, Prometheus exporter was updated so it longer sends out certain metrics but the grafana dashboard is still configured to read values of those (now) non-existent metrics.

1 Like