Discourse Prometheus 404 metrics for Grafana

Support
1

Continuing the discussion from Discourse Prometheus:

Not sure what’s happening but this integration is not working anymore. I couldn’t get the dashboard in grafana to display anything… the /metrics does return something but anything beyond that does not really work

Many errors like the one below

Started GET "/metrics/api/v1/label/instance/values?limit=40000&start=1760980620&end=1760982480&match%5B%5D=discourse_page_views" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
Started GET "/metrics/api/v1/label/job/values?limit=40000&start=1760980620&end=1760982480&match%5B%5D=discourse_page_views%7B%7D" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
  Rendered layout layouts/no_ember.html.erb (Duration: 21.1ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 26.0ms | GC: 0.0ms)
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
  Rendered layout layouts/no_ember.html.erb (Duration: 19.8ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 28.2ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 34.6ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 44.4ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 41.7ms | GC: 0.0ms)
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:08 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:08 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:08 +0000
  Rendered layout layouts/no_ember.html.erb (Duration: 23.6ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 24.9ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 19.9ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 21.6ms | GC: 0.0ms)
Started GET "/metrics/api/v1/label/job/values?limit=40000&start=1760980620&end=1760982480&match%5B%5D=discourse_page_views%7B%7D" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
Started GET "/metrics/api/v1/label/instance/values?limit=40000&start=1760980620&end=1760982480&match%5B%5D=discourse_page_views" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
  Rendered layout layouts/no_ember.html.erb (Duration: 19.5ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 20.9ms | GC: 0.0ms)
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
  Rendered layout layouts/no_ember.html.erb (Duration: 22.4ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 22.0ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 40.0ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 37.0ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 24.8ms | GC: 1.6ms)
2

I have managed to make this work. My problems were:

  1. My instance redirects to `https`
  2. The default grafana configuration for `alloy` was trying to reach localhost:80 (if you don’t know, `alloy` is a prometheus-like server that grafana spins and reads `/metrics` to expose them to grafana)

this is what my config looks like in case it helps anyone

... removed the part that's auto generated when installing alloy ...

discovery.relabel "metrics_integrations_integrations_discourse" {
	targets = [{
		__address__ = "your.domain.com", # <--- THIS IS A CHANGE I DID 
		__scheme__  = "https", # <--- THIS IS A CHANGE I DID 
	}]

	rule {
		target_label = "instance"
		replacement  = constants.hostname
	}
}

prometheus.scrape "metrics_integrations_integrations_discourse" {
	targets    = discovery.relabel.metrics_integrations_integrations_discourse.output
	forward_to = [prometheus.relabel.metrics_integrations_integrations_discourse.receiver]
	job_name   = "integrations/discourse"

	tls_config {
		insecure_skip_verify = true # <--- THIS IS A CHANGE I DID 
	}
}

prometheus.relabel "metrics_integrations_integrations_discourse" {
	forward_to = [prometheus.remote_write.metrics_service.receiver]

	rule {
		source_labels = ["__name__"]
		regex         = "up|discourse_active_app_reqs|discourse_http_duration_seconds|discourse_http_requests|discourse_page_views|discourse_queued_app_reqs|discourse_rss|discourse_scheduled_job_count|discourse_scheduled_job_duration_seconds|discourse_sidekiq_job_count|discourse_sidekiq_job_duration_seconds|discourse_sidekiq_jobs_enqueued|discourse_v8_used_heap_size"
		action        = "keep"
	}
}

with this grafana started getting data…

Don’t forget to set DISCOURSE_PROMETHEUS_TRUSTED_IP_ALLOWLIST_REGEX

I searched for grafana’s IPs and added them all as a long line, there may be a better or smarter way but I couldn’t find it lol

here’s an example of adding multiple ips

DISCOURSE_PROMETHEUS_TRUSTED_IP_ALLOWLIST_REGEX: "^(localhost|127\\.0\\.0\\.1|::1)$