Prometheus exporter plugin for Discourse

Hi,

I’m trying your suggestion with DISCOURSE_PROMETHEUS_TRUSTED_IP_REGEX, and there is something quite strange. I’ve been checking logs and I’m getting the following:

Started GET "/metrics" for ::ffff:188.184.108.126 at 2020-04-27 10:43:04 +0000
ActionController::RoutingError (No route matches [GET] "/metrics")
/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.2.2/lib/action_dispatch/middleware/debug_exceptions.rb:36:in `call'
  Rendering exceptions/not_found.html.erb within layouts/no_ember
  Rendered exceptions/not_found.html.erb within layouts/no_ember (Duration: 0.2ms | Allocations: 18)
  Rendered layouts/_head.html.erb (Duration: 4.4ms | Allocations: 232)
  Rendered common/_discourse_stylesheet.html.erb (Duration: 0.8ms | Allocations: 310)
  Rendered application/_header.html.erb (Duration: 0.5ms | Allocations: 122)
  Rendering html template
  Rendered html template (Duration: 0.0ms | Allocations: 2)

Also, I did a try with DISCOURSE_PROMETHEUS_TRUSTED_IP_REGEX=(/.*/), it means, let’s allow everything to see if at least it works, and no luck so far.

Any clue?

Cheers,
Ismael

1 Like

I spent a few minutes on this and couldn’t get it figured out. From the spec it seems like it should work. I’ve got a couple other irons in the fire before I’ll have a chance to look at this more closely.

1 Like

It works as expected! In my case I ended up by setting prometheus_trusted_ip_whitelist_regex variable under discourse.conf and :rocket:

1 Like

Great! Maybe I’ll get a few minutes to again start monitoring remote sites.

You probably know that discourse.conf gets re-written when you rebuild the container, and putting it in DISCOURSE_PROMETHEUS_TRUSTED_IP_REGEX in your app.yml pushes it into discourse.conf when you start the container.

2 Likes

Yes, the result is the same, since environment variables are eventually stripped and written under discourse.conf at the end. But many thanks, above all for your PR, extremely useful for us.

Cheers,
Ismael

1 Like

I just tried it again too and it worked as expected. No idea what I did wrong the day. . .

Thanks!

1 Like

Hi,
I’ve try to use metrics xith Grafana but ive got problems
(Cross-Origin Request) : politic « Same Origin »
hearder CORS « Access-Control-Allow-Origin » missing

any idea to resolve it.

Hi. I was hoping for step by step instructions for setting this up and am a bit confused.

  • Installed the plugin on latest tests-pass
  • I see /metrics as admin.

Just unsure of next steps for adding this to our grafana instance. Read the thread, but still unsure on steps.

Next step is to install Prometheus and have it scrape the metrics path

3 Likes

Okay I’ll point it at domain.com/metrics

edit:
Still running into issues with this,

grafana.noisebride.net shows 404 error when attempting to access discuss.noisebridget.info/metrics

I am a discourse admin, but the admin of grafana is not. Still confused on how this works.

It appears Prometheus is accessing the /metrics endpoint from a different server. Therefore you need to allow this server to access the endpoint. To do that, you need the external IP address of the server that Prometheus runs on.

You then need to add this line to your app.yml file in the env: section:

DISCOURSE_PROMETHEUS_TRUSTED_IP_WHITELIST_REGEX: '(12\.34\.56\.78)'

In this example the IP address is 12.34.56.78, replace it with your own address. Also make sure the indentation is correct.

After rebuilding the container, Prometheus should be able to access the /metrics endpoint.

3 Likes

Ha! I’m so glad to see that I"m not the only one who wanted this feature!

I’m glad this feature exists, thanks for adding it :slight_smile:

In general, it’s pretty straight-forward. The only issue I faced was that the environment variable is called DISCOURSE_PROMETHEUS_TRUSTED_IP_WHITELIST_REGEX, but in this thread it was sometimes referred to as DISCOURSE_PROMETHEUS_TRUSTED_IP_REGEX. But now it works :+1:

3 Likes

Hello everyone!

So far, If I understand correctly, To ensure a prometheus instance can scrap data from /metrics endpoint, We have to add IP address of that machine in the allow list by editing DISCOURSE_PROMETHEUS_TRUSTED_IP_WHITELIST_REGEX.

Is there any other way to do this? Our discourse instance sits behind cloudflare’s DNS proxy and doesn’t get to see the actual IP address of end users. I’d really prefer if maybe we can figure out a way to do this with API keys? Although that seems unlikely because of the restrictions in how prometheus can ingest data.

Then Discourse is broken and you need to add the cloudflare template.

I think that would work. Did you try? Oh, but I don’t think that you can get grafana to put the keys in the header, which is why I added this feature in the first place.

1 Like

Then Discourse is broken and you need to add the cloudflare template.

Ah, What do you mean by “cloudflare template”?

I think that would work. Did you try? Oh, but I don’t think that you can get grafana to put the keys in the header, which is why I added this feature in the first place.

Yes, I did. Prometheus still doesn’t allow arbitrary headers in scraper configs. So, There is no way for me to intake discourse metrics in prometheus if I can’t whitelist prometheus server’s IP address which is not possible in my case.

There are probably other people who run their discourse instances behind proxies like cloudflare. So, With that assumption in mind, I believe we’ll have to tweak the exporter a little bit.

Discourse’s API uses it’s own Api-Key and Api-Username headers instead of something that is somewhat standardized like http basic auth(which is supported by prometheus scraper subsystem). So, I really have no way to use this in my setup.

You’ll need to search. But you can find it inside the templates directory and add it to your app.yml.

1 Like

Thanks!

Enabled cloudflare.template.yml. :slight_smile: But if I understand correctly, That was just to disable rate limiting on traffic coming from cloudflare’s IP addresses.

For now, This is what my prometheus config looks like,

scrape_configs:
        - job_name: forum
          scrape_interval: 5s
          scheme: https
          static_configs:
                  - targets:
                          - forum-behind-cloudflare-dns-proxy.com

Now, The issue I am facing is, Discourse doesn’t get to see the actual IP address of a request. So, For the allow list approach to work, I have tried this.

  1. Added my prometheus server’s IPv6 address to the DISCOURSE_PROMETHEUS_TRUSTED_IP_WHITELIST_REGEX env variable in app.yml.

  2. Hard coded forum’s actual IPv6 address in /etc/hosts on the prometheus server. Now, The forum can see the IP address of my prometheus server and allow access to it. :slight_smile:

I still have some other problems, Like, I am running prometheus in a container. `/etc/hosts/ from host is not shared inside the container. So, It keeps resolving forum address to a cloudflare IP and fails during authentication.

I can share /etc/hosts from host in docker container like, -v /etc/hosts:/etc/hosts while starting up prometheus but that results in a error like,

Get "https://forum-behind-cloudflare-dns-proxy.com:443/metrics": dial tcp [<ipv6-address>]:443: connect: cannot assign requested address

Now, I just have to resolve this problem.

Edit #1: Figured this out as well. IPv6 was disabled inside docker container. I can fix it by enabling IPv6 in docker container or just --net=host.

Are you behind a reverse proxy? You need to configure nginx to let the real remote up get to discourse. I think Running other websites on the same machine as Discourse might give you a hint on getting that figured out.

1 Like

The discourse instance is indeed running behind a reverse proxy. The reverse proxy is offered by Cloudflare and we are only on the Pro plan and don’t have the option to pass on real client IP up to the discourse instance from cloudflare. Anyyway, My problem is already resolved. :slight_smile:

The grafana dashboard probably should be updated at this point. Over time, Prometheus exporter was updated so it longer sends out certain metrics but the grafana dashboard is still configured to read values of those (now) non-existent metrics.

1 Like