Prometheus and metrics

In short: Prometheus is an open-source system that regularly fetches numeric values (metrics) from services, stores them and provides them for charts and alerts.

In more detail: Instead of each service pushing its numbers, Prometheus pulls them over HTTP from an endpoint such as /metrics. Values are stored as time series with labels and queried with the query language PromQL.

In Depth

• Metric types: Counter (only increases, for example requests), gauge (rises and falls, for example memory), histogram (distribution, for example response times).
• Labels: http_requests_total{method="GET",status="500"} – dimensions for filtering. Too many distinct label values (high cardinality) make the system slow.
• Alerts: Alertmanager sends warnings when a rule matches.
• Dashboards: Grafana draws the time series.
• Four golden signals: Latency, traffic, errors, saturation.

See also: Logging and log levels