Skip to main content

Metrics

Traefik Hub API Gateway provides metrics in the OpenTelemetry format as well as the following vendor specific backends:

Official Dashboard

Traefik Proxy has an official Grafana dashboard for both on-premises and Kubernetes deployments that are can be used to monitor Hub API Gateway metrics too.


Helm Chart Configuration Options

Traefik Hub API Gateway can be configured to provide metrics to the OpenTelemetry format using the Helm Chart values. To know more about the Helm Chart options, refer to Helm Chart reference page (Options metrics.otlp).

Disable Prometheus Metrics

By default, the Traefik Hub API Gateway enables the Prometheus metrics. Disbale them adding the following option:

metrics:
prometheus:
enabled: false

Configuration Options

The section below describe how to configure Traefik Hub API Gateway to provide metrics using the static configuration.

Open Telemetry

FieldDescriptionDefaultRequired
metrics.addInternalsEnables metrics for internal resources (e.g.: ping@internals).falseNo
metrics.otlp.addEntryPointsLabelsEnable metrics on entry points.trueNo
metrics.otlp.addRoutersLabelsEnable metrics on routers.falseNo
metrics.otlp.addServicesLabelsEnable metrics on services.trueNo
metrics.otlp.explicitBoundariesExplicit boundaries for Histogram data points.".005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10"No
metrics.otlp.pushIntervalInterval at which metrics are sent to the OpenTelemetry Collector.10sNo
metrics.otlp.serviceNameDefines the OTEL service name to use.traefikNo
metrics.otlp.httpThis instructs the exporter to send the metrics to the OpenTelemetry Collector using HTTP.
Setting the sub-options with their default values.
null/falseNo
metrics.otlp.http.endpointURL of the OpenTelemetry Collector to send metrics to.
Format="<scheme>://<host>:<port><path>"
"http://localhost:4318/v1/metrics"Yes
metrics.otlp.http.headersAdditional headers sent with metrics by the exporter to the OpenTelemetry Collector.No
metrics.otlp.http.tls.caPath to the certificate authority used for the secure connection to the OpenTelemetry Collector,
it defaults to the system bundle.
""No
metrics.otlp.http.tls.certPath to the public certificate used for the secure connection to the OpenTelemetry Collector.
When using this option, setting the key option is required.
""No
metrics.otlp.http.tls.keyThis instructs the exporter to send the metrics to the OpenTelemetry Collector using HTTP.
Setting the sub-options with their default values.
""null/false ""No
metrics.otlp.http.tls.insecureskipverifyAllow the TLS connection to the OpenTelemetry Collector accepts any certificate presented by the server regardless of the hostnames it covers.falseYes
metrics.otlp.grpcThis instructs the exporter to send metrics to the OpenTelemetry Collector using gRPC.null/falseNo
metrics.otlp.grpc.endpointAddress of the OpenTelemetry Collector to send metrics to.
Format="<host>:<port>"
"localhost:4317"Yes
metrics.otlp.grpc.headersAdditional headers sent with metrics by the exporter to the OpenTelemetry Collector.No
metrics.otlp.http.grpc.insecureAllows exporter to send metrics to the OpenTelemetry Collector without using a secured protocol.falseYes
metrics.otlp.grpc.tls.caPath to the certificate authority used for the secure connection to the OpenTelemetry Collector,
it defaults to the system bundle.
""No
metrics.otlp.grpc.tls.certPath to the public certificate used for the secure connection to the OpenTelemetry Collector.
When using this option, setting the key option is required.
""No
metrics.otlp.grpc.tls.keyThis instructs the exporter to send the metrics to the OpenTelemetry Collector using HTTP.
Setting the sub-options with their default values.
""null/false ""No
metrics.otlp.grpc.tls.insecureskipverifyAllow the TLS connection to the OpenTelemetry Collector accepts any certificate presented by the server regardless of the hostnames it covers.falseYes

Datadog

FieldDescriptionDefaultRequired
metrics.addInternalsEnables metrics for internal resources (e.g.: ping@internals).falseNo
metrics.datadog.addEntryPointsLabelsEnable metrics on entry points.trueNo
metrics.datadog.addRoutersLabelsEnable metrics on routers.falseNo
metrics.datadog.addServicesLabelsEnable metrics on services.trueNo
metrics.datadog.pushIntervalThe interval used by the exporter to push metrics to DataDog server.10sNo
metrics.datadog.addressAddress instructs exporter to send metrics to datadog-agent at this address.
This address can be a Unix Domain Socket (UDS) address with the following form: unix:///path/to/datadog.socket.
"127.0.0.1:8125"Yes
metrics.datadog.prefixThe prefix to use for metrics collection."traefik"No

InfluxDB v2

FieldDescriptionDefaultRequired
metrics.addInternalsEnables metrics for internal resources (e.g.: ping@internals).falseNo
metrics.influxDB2.addEntryPointsLabelsEnable metrics on entry points.trueNo
metrics.influxDB2.addRoutersLabelsEnable metrics on routers.falseNo
metrics.influxDB2.addServicesLabelsEnable metrics on services.trueNo
metrics.influxDB2.additionalLabelsAdditional labels (InfluxDB tags) on all metrics.{}]No
metrics.influxDB2.pushIntervalThe interval used by the exporter to push metrics to InfluxDB server.10sNo
metrics.influxDB2.addressAddress of the InfluxDB v2 instance."http://localhost:8086"Yes
metrics.influxDB2.tokenToken with which to connect to InfluxDB v2.""Yes
metrics.influxDB2.orgOrganisation where metrics will be stored.""Yes
metrics.influxDB2.bucketBucket where metrics will be stored.""Yes

Prometheus

FieldDescriptionDefaultRequired
metrics.prometheus.addInternalsEnables metrics for internal resources (e.g.: ping@internals).falseNo
metrics.prometheus.addEntryPointsLabelsEnable metrics on entry points.trueNo
metrics.prometheus.addRoutersLabelsEnable metrics on routers.falseNo
metrics.prometheus.addServicesLabelsEnable metrics on services.trueNo
metrics.prometheus.bucketsBuckets for latency metrics."0.100000, 0.300000, 1.200000, 5.000000"No
metrics.prometheus.manualRoutingSet to true, it disables the default internal router in order to allow creating a custom router for the prometheus@internal service.falseNo
metrics.prometheus.entryPointTraefik Entrypoint name used to expose metrics."traefik"No
metrics.prometheus.headerLabelsDefines the extra labels for the requests_total metrics.
More information here.
Yes

headerLabels

Defines the extra labels for the requests_total metrics, and for each of them, the request header containing the value for this label. If the header is not present in the request it will be added nonetheless with an empty value. The label must be a valid label name for Prometheus metrics, otherwise, the Prometheus metrics provider will fail to serve any Traefik-related metric.

How to provide the `Host` header value

The Host header is never present in the Header map of a request, as per go documentation says:

// For incoming requests, the Host header is promoted to the
// Request.Host field and removed from the Header map.

As a workaround, to obtain the Host of a request as a label, use instead the X-Forwarded-Host header.

Configuration Example

Here is an example of the entryPoint requests_total metric with an additional "useragent" label.

When configuring the label in Static Configuration:

# static_configuration.yaml
metrics:
prometheus:
headerLabels:
useragent: User-Agent

StatsD

FieldDescriptionDefaultRequired
metrics.addInternalsEnables metrics for internal resources (e.g.: ping@internals).falseNo
metrics.statsD.addEntryPointsLabelsEnable metrics on entry points.trueNo
metrics.statsD.addRoutersLabelsEnable metrics on routers.falseNo
metrics.statsD.addServicesLabelsEnable metrics on services.trueNo
metrics.statsD.pushIntervalThe interval used by the exporter to push metrics to DataDog server.10sNo
metrics.statsD.addressAddress instructs exporter to send metrics to statsd at this address."127.0.0.1:8125"Yes
metrics.statsD.prefixThe prefix to use for metrics collection."traefik"No

Metrics Provided

Global Metrics

MetricTypeLabelsDescription
traefik_config_reloads_totalCountThe total count of configuration reloads.
traefik_config_last_reload_successGaugeThe timestamp of the last configuration reload success.
traefik_open_connectionsGaugeentrypoint, protocolThe current count of open connections, by entrypoint and protocol.
traefik_tls_certs_not_afterGaugeThe expiration date of certificates.

Labels

Here is a comprehensive list of labels that are provided by the global metrics:

LabelDescriptionExample
entrypointEntrypoint that handled the connection"example_entrypoint"
protocolConnection protocol"TCP"

OpenTelemetry Semantic Conventions

Traefik Proxy follows official OpenTelemetry semantic conventions v1.23.1.

HTTP Server

MetricTypeLabelsDescription
http.server.request.durationHistogramerror.type, http.request.method, http.response.status_code, network.protocol.name, server.address, server.port, url.schemeDuration of HTTP server requests
Labels

Here is a comprehensive list of labels that are provided by the metrics:

LabelDescriptionexample
error.typeDescribes a class of error the operation ended with"500"
http.request.methodHTTP request method"GET"
http.response.status_codeHTTP response status code"200"
network.protocol.nameOSI application layer or non-OSI equivalent"http/1.1"
network.protocol.versionVersion of the protocol specified in network.protocol.name"1.1"
server.addressName of the local HTTP server that received the request"example.com"
server.portPort of the local HTTP server that received the request"80"
url.schemeThe URI scheme component identifying the used protocol"http"

HTTP Client

MetricTypeLabelsDescription
http.client.request.durationHistogramerror.type, http.request.method, http.response.status_code, network.protocol.name, server.address, server.port, url.schemeDuration of HTTP client requests
Labels

Here is a comprehensive list of labels that are provided by the metrics:

LabelDescriptionexample
error.typeDescribes a class of error the operation ended with"500"
http.request.methodHTTP request method"GET"
http.response.status_codeHTTP response status code"200"
network.protocol.nameOSI application layer or non-OSI equivalent"http/1.1"
network.protocol.versionVersion of the protocol specified in network.protocol.name"1.1"
server.addressName of the local HTTP server that received the request"example.com"
server.portPort of the local HTTP server that received the request"80"
url.schemeThe URI scheme component identifying the used protocol"http"

HTTP Metrics

On top of the official OpenTelemetry semantic conventions, Traefik provides its own metrics to monitor the incoming traffic.

EntryPoint Metrics

MetricTypeLabelsDescription
traefik_entrypoint_requests_totalCountcode, method, protocol, entrypointThe total count of HTTP requests received by an entrypoint.
traefik_entrypoint_requests_tls_totalCounttls_version, tls_cipher, entrypointThe total count of HTTPS requests received by an entrypoint.
traefik_entrypoint_request_duration_secondsHistogramcode, method, protocol, entrypointRequest processing duration histogram on an entrypoint.
traefik_entrypoint_requests_bytes_totalCountcode, method, protocol, entrypointThe total size of HTTP requests in bytes handled by an entrypoint.
traefik_entrypoint_responses_bytes_totalCountcode, method, protocol, entrypointThe total size of HTTP responses in bytes handled by an entrypoint.

Router Metrics

MetricTypeLabelsDescription
traefik_router_requests_totalCountcode, method, protocol, router, serviceThe total count of HTTP requests handled by a router.
traefik_router_requests_tls_totalCounttls_version, tls_cipher, router, serviceThe total count of HTTPS requests handled by a router.
traefik_router_request_duration_secondsHistogramcode, method, protocol, router, serviceRequest processing duration histogram on a router.
traefik_router_requests_bytes_totalCountcode, method, protocol, router, serviceThe total size of HTTP requests in bytes handled by a router.
traefik_router_responses_bytes_totalCountcode, method, protocol, router, serviceThe total size of HTTP responses in bytes handled by a router.

Service Metrics

MetricTypeLabelsDescription
traefik_service_requests_totalCountcode, method, protocol, serviceThe total count of HTTP requests processed on a service.
traefik_service_requests_tls_totalCounttls_version, tls_cipher, serviceThe total count of HTTPS requests processed on a service.
traefik_service_request_duration_secondsHistogramcode, method, protocol, serviceRequest processing duration histogram on a service.
traefik_service_retries_totalCountserviceThe count of requests retries on a service.
traefik_service_server_upGaugeservice, urlCurrent service's server status, 0 for a down or 1 for up.
traefik_service_requests_bytes_totalCountcode, method, protocol, serviceThe total size of requests in bytes received by a service.
traefik_service_responses_bytes_totalCountcode, method, protocol, serviceThe total size of responses in bytes returned by a service.

Labels

Here is a comprehensive list of labels that are provided by the metrics:

LabelDescriptionexample
cnCertificate Common Name"example.com"
codeRequest code"200"
entrypointEntrypoint that handled the request"example_entrypoint"
methodRequest Method"GET"
protocolRequest protocol"http"
routerRouter that handled the request"example_router"
sansCertificate Subject Alternative NameS"example.com"
serialCertificate Serial Number"123..."
serviceService that handled the request"example_service@provider"
tls_cipherTLS cipher used for the request"TLS_FALLBACK_SCSV"
tls_versionTLS version used for the request"1.0"
urlService server url"http://example.com"
"method label value"

If the HTTP method verb on a request is not one defined in the set of common methods for HTTP/1.1 or the PRI verb (for HTTP/2), then the value for the method label becomes EXTENSION_METHOD.

Generative AI Metrics

note

The Generative AI client metrics are only available when you enable the AI Gateway

The following metrics are specifically designed to monitor Generative AI client applications, adhering to OpenTelemetry’s conventions.

MetricTypeDescription
gen_ai.client.token.usageHistogramMeasures the number of input and output tokens used in AI operations.
gen_ai.client.operation.durationHistogramMeasures the duration of Generative AI operations in seconds.

For more detailed information on the available metrics & their attributes, refer to the OpenTelemetry Gen AI client metrics documentation.

Labels

LabelDescriptionexample
app_idUnique Identifier for the application"app-id"
app_nameName of the Application"myApp"
error.typeDescribes a class of error the operation ended with.timeout; java.net.UnknownHostException; server_certificate_invalid; 500
genai.operation.nameThe name of the operation performed"chat"
genai.response_modelThe model used by the AI service in the response"gpt-4o-2024-08-06"
genai.request_modelThe model requested by the client"gpt-3.5-turbo"
genai.systemIdentifier for the AI provider system"openai"
server.addressThe address of the AI service server"api.openai.com"
server.portThe port number of the AI service server443