Grok Pattern for Nginx logs

Development

AI-generated summary

Grok Pattern for Nginx logs

The discussion revolves around finding a Grok pattern for custom Nginx logs generated by Discourse. Initially, mpalmer was unaware of any existing pattern, but sam pointed to a Ruby-based log analyzer in the discourse-nginx-performance-report plugin.

jackzampolin was looking for a Grok pattern and eventually found a solution using the Telegraf logparser plugin. The plugin configuration and custom patterns are provided to parse the Discourse logs, and the resulting data is stored in InfluxDB line protocol. The configuration includes the following:

  • Measurement: nginx_access
  • Tags: method, discourse_route, response_code, username
  • Fields: client_ip, request_path, http_version, user_agent, resp_bytes, x_referer, resp_time

The provided configuration can be used as a starting point for creating a Grok pattern for Discourse Nginx logs.

1

It looks like discourse spits out custom Nginx logs. There wouldn’t happen to be a grok pattern for those logs anywhere? I looked and was unable to find anything.

2

I’m not aware of one. We haven’t written one ourselves because we collect all access log data through haproxy.

3

We have this :slight_smile:

https://github.com/discourse/discourse/blob/master/plugins/discourse-nginx-performance-report/lib/log_analyzer.rb#L11-L15

It is in ruby and probably easily portable to an elastic grok rule

4

Thanks for the pointer! I’ll drop this here for anyone looking:

I’m parsing the discourse logs with the telegraf logparser plugin. The following config works to parse this:

[[inputs.logparser]]
  files = [
    "/var/discourse/shared/standalone/log/var-log/nginx/access.log",
    "/var/discourse/shared/standalone/log/var-log/nginx/access.log.1"
  ]
  from_beginning = true
  [inputs.logparser.grok]
    measurement="nginx_access"
    patterns=[
      '''\[%{CUSTOM_TIMESTAMP:timestamp:ts-httpd}\] %{IP:client_ip} "%{NOTSPACE:method:tag} %{NOTSPACE:request_path} %{NOTSPACE:http_version}" "%{DATA:user_agent}" "%{DATA:discourse_route:tag}" %{NUMBER:response_code:tag} %{NUMBER:resp_bytes:int} "%{DATA:x_referrer}" %{NUMBER} %{NUMBER:resp_time:float} "%{DATA:username:tag}"'''
    ]
    custom_patterns = '''
      CUSTOM_TIMESTAMP %{MONTHDAY}/%{MONTH}/%{YEAR}:%{HOUR}:%{MINUTE}:%{SECOND} %{CUSTOM_TZ}
      CUSTOM_TZ [+-][0-9]{4}
    '''

The data is as follows in InfluxDB line protocol:

{
  measurement: nginx_access,
  tags: [
    method,
    discourse_route,
    response_code,
    username
  ],
  fields: {
    client_ip: string,
    request_path: string,
    http_version: string,
    user_agent: string,
    resp_bytes: int,
    x_referer: string,
   resp_time: float,
  }
}