Grafana용 Discourse Prometheus 404 메트릭

Discourse Prometheus 토론을 이어갑니다:

무슨 일이 일어나고 있는지 정확히 모르겠지만, 이 통합 기능이 더 이상 작동하지 않습니다. Grafana 대시보드에 아무것도 표시되지 않았습니다… /metrics는 응답을 반환하지만, 그 이후의 요청은 제대로 작동하지 않습니다.

아래와 같은 오류가 많이 발생합니다.

Started GET "/metrics/api/v1/label/instance/values?limit=40000&start=1760980620&end=1760982480&match%5B%5D=discourse_page_views" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
Started GET "/metrics/api/v1/label/job/values?limit=40000&start=1760980620&end=1760982480&match%5B%5D=discourse_page_views%7B%7D" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
  Rendered layout layouts/no_ember.html.erb (Duration: 21.1ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 26.0ms | GC: 0.0ms)
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
  Rendered layout layouts/no_ember.html.erb (Duration: 19.8ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 28.2ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 34.6ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 44.4ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 41.7ms | GC: 0.0ms)
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:07 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:08 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:08 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:08 +0000
  Rendered layout layouts/no_ember.html.erb (Duration: 23.6ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 24.9ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 19.9ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 21.6ms | GC: 0.0ms)
Started GET "/metrics/api/v1/label/job/values?limit=40000&start=1760980620&end=1760982480&match%5B%5D=discourse_page_views%7B%7D" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
Started GET "/metrics/api/v1/label/instance/values?limit=40000&start=1760980620&end=1760982480&match%5B%5D=discourse_page_views" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
  Rendered layout layouts/no_ember.html.erb (Duration: 19.5ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 20.9ms | GC: 0.0ms)
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
Started POST "/metrics/api/v1/query_range" for 16.63.57.20 at 2025-10-20 17:47:37 +0000
  Rendered layout layouts/no_ember.html.erb (Duration: 22.4ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 22.0ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 40.0ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 37.0ms | GC: 0.0ms)
  Rendered layout layouts/no_ember.html.erb (Duration: 24.8ms | GC: 1.6ms)

드디어 이 문제를 해결했습니다. 제 문제는 다음과 같았습니다:

  1. 제 인스턴스가 https로 리다이렉트되고 있습니다.
  2. alloy의 기본 grafana 설정이 localhost:80에 접근하려고 하고 있었습니다. (alloy가 무엇인지 모른다면, grafana가 실행하여 /metrics를 읽어 grafana에 노출하는 prometheus 유사 서버입니다.)

혹시 도움이 될 수 있을까 해서 제 설정을 공유합니다.

... alloy 설치 시 자동 생성되는 부분은 제거했습니다 ...

discovery.relabel "metrics_integrations_integrations_discourse" {
	targets = [{
		__address__ = "your.domain.com", # <--- 제가 변경한 부분 
		__scheme__  = "https", # <--- 제가 변경한 부분 
	}]

	rule {
		target_label = "instance"
		replacement  = constants.hostname
	}
}

prometheus.scrape "metrics_integrations_integrations_discourse" {
	targets    = discovery.relabel.metrics_integrations_integrations_discourse.output
	forward_to = [prometheus.relabel.metrics_integrations_integrations_discourse.receiver]
	job_name   = "integrations/discourse"

	tls_config {
		insecure_skip_verify = true # <--- 제가 변경한 부분 
	}
}

prometheus.relabel "metrics_integrations_integrations_discourse" {
	forward_to = [prometheus.remote_write.metrics_service.receiver]

	rule {
		source_labels = ["__name__"]
		regex         = "up|discourse_active_app_reqs|discourse_http_duration_seconds|discourse_http_requests|discourse_page_views|discourse_queued_app_reqs|discourse_rss|discourse_scheduled_job_count|discourse_scheduled_job_duration_seconds|discourse_sidekiq_job_count|discourse_sidekiq_job_duration_seconds|discourse_sidekiq_jobs_enqueued|discourse_v8_used_heap_size"
		action        = "keep"
	}
}

이 설정으로 grafana가 데이터를 받기 시작했습니다…

DISCOURSE_PROMETHEUS_TRUSTED_IP_ALLOWLIST_REGEX를 설정하는 것을 잊지 마세요.

grafana의 IP를 검색하여 모두 긴 한 줄로 추가했습니다. 더 좋거나 지능적인 방법이 있을 수 있지만 저는 찾지 못했습니다 ㅋㅋ

여러 IP를 추가하는 예시는 다음과 같습니다.

DISCOURSE_PROMETHEUS_TRUSTED_IP_ALLOWLIST_REGEX: "^(localhost|127\\.0\\.0\\.1|::1)$