Documentation
a project

Monitoring Caddy with Prometheus metrics

Whether you're running thousands of Caddy instances in the cloud, or a single Caddy server on an embedded device, it's likely that at some point you'll want to have a high-level overview of what Caddy is doing, and how long it's taking. In other words, you're going to want to be able to monitor Caddy.

Prometheus

Prometheus is a monitoring platform that collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets. As well as helping you to display metrics with a dashboarding tool like Grafana, Prometheus is also used for alerting.

Like Caddy, Prometheus is written in Go and distributed as a single binary. To install it, see the Prometheus Installation docs, or on MacOS just run brew install prometheus.

Read the Prometheus docs if you're brand new to Prometheus, otherwise read on!

To configure Prometheus to scrape from Caddy you'll need a YAML configuration file similar to this:

# prometheus.yaml
global:
  scrape_interval: 15s # default is 1 minute

scrape_configs:
  - job_name: caddy
    static_configs:
      - targets: ['localhost:2019']

You can then start up Prometheus like this:

$ prometheus --config.file=prometheus.yaml

Caddy's metrics

Like any process monitored with Prometheus, Caddy exposes an HTTP endpoint that responds in the Prometheus exposition format. Caddy's Prometheus client is also configured to respond with the OpenMetrics exposition format if negotiated (that is, if the Accept header is set to application/openmetrics-text; version=0.0.1).

By default, there is a /metrics endpoint available at the admin API (i.e. http://localhost:2019/metrics). But if the admin API is disabled or you wish to listen on a different port or path, you can use the metrics handler to configure this.

You can see the metrics with any browser or HTTP client like curl:

$ curl http://localhost:2019/metrics
# HELP caddy_admin_http_requests_total Counter of requests made to the Admin API's HTTP endpoints.
# TYPE caddy_admin_http_requests_total counter
caddy_admin_http_requests_total{code="200",handler="metrics",method="GET",path="/metrics"} 2
# HELP caddy_http_request_duration_seconds Histogram of round-trip request durations.
# TYPE caddy_http_request_duration_seconds histogram
caddy_http_request_duration_seconds_bucket{code="308",handler="static_response",method="GET",server="remaining_auto_https_redirects",le="0.005"} 1
caddy_http_request_duration_seconds_bucket{code="308",handler="static_response",method="GET",server="remaining_auto_https_redirects",le="0.01"} 1
caddy_http_request_duration_seconds_bucket{code="308",handler="static_response",method="GET",server="remaining_auto_https_redirects",le="0.025"} 1
...

There are a number of metrics you'll see, that broadly fall under 3 categories:

  • Runtime metrics
  • Admin API metrics
  • HTTP Middleware metrics

Runtime metrics

These metrics cover the internals of the Caddy process, and are provided automatically by the Prometheus Go Client. They are prefixed with go_* and process_*.

Note that the process_* metrics are only collected on Linux and Windows.

See the documentation for the Go Collector, Process Colletor, and BuildInfo Collector.

Admin API metrics

These are metrics that help to monitor the Caddy admin API. Each of the admin endpoints is instrumented to track request counts and errors.

These metrics are prefixed with caddy_admin_*.

For example:

$ curl -s http://localhost:2019/metrics | grep ^caddy_admin
caddy_admin_http_requests_total{code="200",handler="admin",method="GET",path="/config/"} 1
caddy_admin_http_requests_total{code="200",handler="admin",method="GET",path="/debug/pprof/"} 2
caddy_admin_http_requests_total{code="200",handler="admin",method="GET",path="/debug/pprof/cmdline"} 1
caddy_admin_http_requests_total{code="200",handler="load",method="POST",path="/load"} 1
caddy_admin_http_requests_total{code="200",handler="metrics",method="GET",path="/metrics"} 3

caddy_admin_http_requests_total

A counter of the number of requests handled by admin endpoints, including modules in the admin.api.* namespace.

Label Description
code HTTP status code
handler The handler or module name
method The HTTP method
path The URL path the admin endpoint was mounted to

caddy_admin_http_request_errors_total

A counter of the number of errors encountered in admin endpoints, including modules in the admin.api.* namespace.

Label Description
handler The handler or module name
method The HTTP method
path The URL path the admin endpoint was mounted to

HTTP Middleware metrics

All Caddy HTTP middleware handlers are instrumented automatically for determining request latency, time-to-first-byte, errors, and request/response body sizes.

For the histogram metrics below, the buckets are currently not configurable. For durations, the default (prometheus.DefBuckets set of buckets is used (5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, and 10s). For sizes, the buckets are 256b, 1kiB, 4kiB, 16kiB, 64kiB, 256kiB, 1MiB, and 4MiB.

caddy_http_requests_in_flight

A gauge of the number of requests currently being handled by this server.

Label Description
server The server name
handler The handler or module name

caddy_http_request_errors_total

A counter of middleware errors encountered while handling requests.

Label Description
server The server name
handler The handler or module name

caddy_http_requests_total

A counter of HTTP(S) requests made.

Label Description
server The server name
handler The handler or module name

caddy_http_request_duration_seconds

A histogram of the round-trip request durations.

Label Description
server The server name
handler The handler or module name
code HTTP status code
method The HTTP method

caddy_http_request_size_bytes

A histogram of the total (estimated) size of the request. Includes body.

Label Description
server The server name
handler The handler or module name
code HTTP status code
method The HTTP method

caddy_http_response_size_bytes

A histogram of the size of the returned response body.

Label Description
server The server name
handler The handler or module name
code HTTP status code
method The HTTP method

caddy_http_response_duration_seconds

A histogram of time-to-first-byte for responses.

Label Description
server The server name
handler The handler or module name
code HTTP status code
method The HTTP method

Sample Queries

Once you have Prometheus scraping Caddy's metrics, you can start to see some interesting metrics about how Caddy's performing.

For example, to see the per-second request rate, as averaged over 5 minutes:

rate(caddy_http_requests_total{handler="file_server"}[5m])

To see the rate at which your latency threshold of 100ms is being exceeded:

sum(rate(caddy_http_request_duration_seconds_count{server="srv0"}[5m])) by (handler)
-
sum(rate(caddy_http_request_duration_seconds_bucket{le="0.100", server="srv0"}[5m])) by (handler)

To find the 95th percentile request duration on the file_server handler, you can use a query like this:

histogram_quantile(0.95, sum(caddy_http_request_duration_seconds_bucket{handler="file_server"}) by (le))

Or to see the median response size in bytes for successful GET requests on the file_server handler:

histogram_quantile(0.5, caddy_http_response_size_bytes_bucket{method="GET", handler="file_server", code="200"})