Convalesco | Extract HAProxy status metrics using mtail

HAProxy provides Prometheus metrics by default. The provided metrics group HTTP status codes as 2xx, 3xx, 4xx, and 5xx. This is intentional, as these groupings are often sufficient at the reverse proxy layer.

For more detailed information, logs can be utilised. HAProxy supports customised log formats, offering numerous options.

Managing and storing logs can be costly. Metrics are efficient and cheap store and easier to process.

mtail is a tool that extracts metrics from logs using regular expressions.

HAProxy does not support writing logs directly to the filesystem. To save logs to a filesystem, a specialised tool is required. rsyslog can handle the task easily. The rsyslog configuration is simple:

# Load UDP module
module(load="imudp")
input(type="imudp" port="5113")

# Define template for log file path
template(name="Everything" type="string" string="/opt/log/everything.log")

# Send HAProxy messages to a dedicated logfile
:programname, startswith, "haproxy" {
  /opt/log/haproxy.log
  stop
}

# Log everything to the specified template
*.* ?Everything

# Provide logging through standard error, useful when running in Docker
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$RepeatedMsgReduction on
$FileCreateMode 0644
$DirCreateMode 0755
$Umask 0022

# Ensure all log files have the correct permissions
$FileCreateMode 0644
$DirCreateMode 0755
$Umask 0022

The HAProxy log format used is as follows:

%ci:%cp %ft %b/%s %ST %B %Tr %ac/%fc/%bc/%sc/%rc %sq/%bq %{+Q}r

HAProxy supports an extremely wide range of log formats. The above example is a simple yet functional one. The mtail configuration is as follows:

counter mtail_line_count by process, pid
counter mtail_haproxy_http_requests_total by frontend_name_transport, backend_name, status_code, request_method, http_version
counter mtail_haproxy_http_nosrv_total by frontend_name_transport, backend_name, status_code, request_method, http_version

histogram mtail_haproxy_http_response_duration_ms buckets 0.0, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0 by backend_name, server_name

hidden gauge response_time_ms

def syslog {
  /^(?P<date>(?P<legacy_date>\w+\s+\d+\s+\d+:\d+:\d+)|(?P<rfc3339_date>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\+\d{2}:\d{2}))/ +
    /\s+(?P<hostname>[\w\.-]+)\s+(?P<application>[\w\.-]+)(?:\[(?P<pid>\d+)\])?:\s+(?P<message>.*)/ {
      len($legacy_date) > 0 {
        strptime($2, "Jan _2 15:04:05")
      }
      len($rfc3339_date) > 0 {
        strptime($rfc3339_date, "2006-01-02T15:04:05-07:00")
      }
      mtail_line_count[$application][$pid]++
      next
    }
}

@syslog {
  /\d+(\.\d+){3}:\d+ / +
    /(?P<frontend_name_transport>http-\w+) / +
    /(?P<backend_name>\w+)\/<?(?P<server_name>[<>\w\.-]+)>? / +
    /(?P<status_code>\d{3}) / +
    /(?P<bytes_read>\d+) / +
    /(?P<response_time>-?\d+) / +
    /(?P<actconn>-?\d+)\// +
    /(?P<feconn>-?\d+)\// +
    /(?P<beconn>-?\d+)\// +
    /(?P<srv_conn>\d+)\// +
    /(?P<retries>\d+) / +
    /(?P<srv_queue>\d+)\/(?P<backend_queue>\d+) / +
    /"(?P<request_method>[A-Z]+) (?P<URI>\S+) (?P<http_version>HTTP\/[0-9\.]+)"/ +
    /$/ {

      # Handle response_time (%Tr):
      $response_time =~ /^-?\d+$|^0$/ {
        response_time_ms = 0.0
      } else {
        response_time_ms = $response_time / 1000.0
      }

      mtail_haproxy_http_requests_total[$frontend_name_transport][$backend_name][$status_code][$request_method][$http_version]++

      /<NOSRV>/ {
        mtail_haproxy_http_nosrv_total[$frontend_name_transport][$backend_name][$status_code][$request_method][$http_version]++
      } else {
        mtail_haproxy_http_response_duration_ms[$backend_name][$server_name] = response_time_ms
      }
    }
}

The @syslog decorator is likely unnecessary here. The rest is a regular expression that matches the HAProxy log format. In this case, we are extracting mtail_haproxy_http_requests_total and a few other metrics. With mtail, we can extract any type of metrics we need from logs.

The setup can be visualized as follows:

+------------------+                +------------------+                       +------------------+
|                  |                |                  |                       |                  |
|    HAProxy       |  logs to UDP   |    rsyslogd      |  saves to file        |    mtail pod     |
|                  |  port 5113/udp |  (port 5113)     | /opt/log/haproxy.log  |   transforms     |
|                  |--------------->|                  |                       | logs into metrics|
|                  |                |                  |<----------------------|  exposed at port |
|                  |                |                  |                       |      3903        |
+------------------+                +------------------+                       +------------------+

The metrics can be scraped by Prometheus. The mtail metrics are exposed on port 3903 by default.