Discourse Prometheus 404 metrics for Grafana

Product Support

AI-generated summary

The discussion revolves around issues with Discourse Prometheus metrics not working in Grafana, resulting in 404 errors. DevTeVe reports that while the /metrics endpoint returns data, Grafana fails to display anything due to errors. The problem is traced to two main issues: the instance redirecting to HTTPS and the default Alloy configuration in Grafana attempting to connect to localhost:80. DevTeVe resolved the issue by updating the Alloy configuration to use the correct domain and HTTPS scheme, and by adjusting the discovery and scrape configurations to properly target the Discourse instance.

1

Continuing the discussion from Discourse Prometheus:

Not sure what’s happening but this integration is not working anymore. I couldn’t get the dashboard in grafana to display anything… the /metrics does return something but anything beyond that does not really work

Many errors like the one below

Started GET "/metrics/api/v1/label/instance/values?limit=40000&start=1760980620&end=1760982480&match%5B%5D=discourse_page_views" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
Started GET "/metrics/api/v1/label/job/values?limit=40000&start=1760980620&end=1760982480&match%5B%5D=discourse_page_views%7B%7D" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
  Rendered layout layouts/no_ember.html.erb (Duration: 21.1ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 26.0ms | GC: 0.0ms)
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
  Rendered layout layouts/no_ember.html.erb (Duration: 19.8ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 28.2ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 34.6ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 44.4ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 41.7ms | GC: 0.0ms)
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:08 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:08 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:08 +0000
  Rendered layout layouts/no_ember.html.erb (Duration: 23.6ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 24.9ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 19.9ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 21.6ms | GC: 0.0ms)
Started GET "/metrics/api/v1/label/job/values?limit=40000&start=1760980620&end=1760982480&match%5B%5D=discourse_page_views%7B%7D" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
Started GET "/metrics/api/v1/label/instance/values?limit=40000&start=1760980620&end=1760982480&match%5B%5D=discourse_page_views" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
  Rendered layout layouts/no_ember.html.erb (Duration: 19.5ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 20.9ms | GC: 0.0ms)
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
  Rendered layout layouts/no_ember.html.erb (Duration: 22.4ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 22.0ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 40.0ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 37.0ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 24.8ms | GC: 1.6ms)
2

I have managed to make this work. My problems were:

  1. My instance redirects to `https`
  2. The default grafana configuration for `alloy` was trying to reach localhost:80 (if you don’t know, `alloy` is a prometheus-like server that grafana spins and reads `/metrics` to expose them to grafana)

this is what my config looks like in case it helps anyone

... removed the part that's auto generated when installing alloy ...

discovery.relabel "metrics_integrations_integrations_discourse" {
	targets = [{
		__address__ = "your.domain.com", # <--- THIS IS A CHANGE I DID 
		__scheme__  = "https", # <--- THIS IS A CHANGE I DID 
	}]

	rule {
		target_label = "instance"
		replacement  = constants.hostname
	}
}

prometheus.scrape "metrics_integrations_integrations_discourse" {
	targets    = discovery.relabel.metrics_integrations_integrations_discourse.output
	forward_to = [prometheus.relabel.metrics_integrations_integrations_discourse.receiver]
	job_name   = "integrations/discourse"

	tls_config {
		insecure_skip_verify = true # <--- THIS IS A CHANGE I DID 
	}
}

prometheus.relabel "metrics_integrations_integrations_discourse" {
	forward_to = [prometheus.remote_write.metrics_service.receiver]

	rule {
		source_labels = ["__name__"]
		regex         = "up|discourse_active_app_reqs|discourse_http_duration_seconds|discourse_http_requests|discourse_page_views|discourse_queued_app_reqs|discourse_rss|discourse_scheduled_job_count|discourse_scheduled_job_duration_seconds|discourse_sidekiq_job_count|discourse_sidekiq_job_duration_seconds|discourse_sidekiq_jobs_enqueued|discourse_v8_used_heap_size"
		action        = "keep"
	}
}

with this grafana started getting data…

Don’t forget to set DISCOURSE_PROMETHEUS_TRUSTED_IP_ALLOWLIST_REGEX

I searched for grafana’s IPs and added them all as a long line, there may be a better or smarter way but I couldn’t find it lol

here’s an example of adding multiple ips

DISCOURSE_PROMETHEUS_TRUSTED_IP_ALLOWLIST_REGEX: "^(localhost|127\\.0\\.0\\.1|::1)$