Understanding Prometheus Metrics: A Complete Guide for SREs

What Are Prometheus Metrics?

Prometheus has become the de facto standard for monitoring cloud-native applications, and understanding its metrics system is fundamental to building reliable observability pipelines. At its core, Prometheus collects and stores metrics as time series data, meaning each metric value is recorded with a timestamp, allowing you to track changes over time and analyze system behavior.

Metrics in Prometheus are presented in a human-readable text-based format called the Prometheus exposition format. This standardized format makes it easy for Prometheus to parse and process data from various sources, whether they're applications, services, or infrastructure components.

The Pull-Based Architecture

Unlike traditional monitoring systems that rely on agents pushing data to a central server, Prometheus uses a pull-based model. The Prometheus server actively scrapes metrics from configured targets at specified intervals, typically through HTTP endpoints exposed at /metrics. This approach offers several advantages in dynamic environments where services frequently scale up or down.

The scraping process is controlled through the Prometheus configuration file, where you define scrape intervals and targets:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  scrape_timeout: 10s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

This configuration tells Prometheus to scrape metrics every 15 seconds from the specified targets, with a 10-second timeout for each scrape operation.

Core Metric Types

Prometheus defines four fundamental metric types, each serving a specific monitoring purpose:

Counters

Counters are cumulative metrics that only increase over time (or reset to zero on restart). They're ideal for tracking events like the total number of HTTP requests, errors, or completed tasks. For example, tracking total HTTP requests:

http_requests_total{method="GET", status="200"} 100

Gauges

Gauges represent values that can fluctuate up or down, making them perfect for monitoring current states like memory usage, active connections, or queue sizes. Unlike counters, gauges reflect instantaneous measurements rather than cumulative totals.

Histograms

Histograms sample observations and count them in configurable buckets, along with providing a sum of all observed values. They're particularly useful for tracking request durations or response sizes, allowing you to calculate percentiles and analyze distribution patterns.

Summaries

Summaries are similar to histograms but calculate quantiles on the client side. They're helpful for performance analysis when you need precomputed percentiles, such as tracking the 95th percentile of request durations across your infrastructure.

Understanding Labels and Multi-Dimensional Data

One of Prometheus's most powerful features is its multi-dimensional data model using labels. Labels are key-value pairs that add context and metadata to metrics, enabling flexible querying and aggregation. This approach allows you to slice and dice your metrics data in countless ways.

Consider this example of an HTTP request counter with labels:

http_requests_total{method="GET", status="200", endpoint="/api/users"} 1500
http_requests_total{method="POST", status="201", endpoint="/api/users"} 300
http_requests_total{method="GET", status="404", endpoint="/api/products"} 25

With these labels, you can query specific combinations—like all GET requests, all 404 errors, or traffic to a particular endpoint—without needing separate metric names for each scenario. This dramatically simplifies your monitoring setup while maintaining granular visibility.

The Metrics Exposition Format

When Prometheus scrapes a target, it expects metrics in a specific text format. Each metric includes several components:

• Metric Name: A descriptive identifier like http_requests_total
• Labels: Optional key-value pairs providing context
• Timestamp: When the measurement was recorded
• Value: The actual measurement
• Comments: Documentation prefixed with # for HELP and TYPE declarations

Here's a complete example:

# HELP http_requests_total Total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="GET", status="200"} 100
http_requests_total{method="POST", status="200"} 50
http_requests_total{method="GET", status="404"} 5

The HELP line provides human-readable documentation, while the TYPE line declares the metric type, helping both users and tools understand how to interpret the data.

Working with Exporters

Not all applications natively expose Prometheus metrics. This is where exporters come in—specialized components that bridge the gap between various technologies and Prometheus. Exporters collect metrics from databases, web servers, hardware, and other systems, converting them into the Prometheus format.

Popular exporters include node_exporter for system metrics, mysql_exporter for database monitoring, and blackbox_exporter for probing endpoints. These exporters expose their own /metrics endpoints that Prometheus can scrape, extending monitoring capabilities across your entire infrastructure.

Querying Metrics with PromQL

Prometheus includes PromQL (Prometheus Query Language), a powerful tool for filtering and analyzing metrics. PromQL lets you aggregate data, calculate rates, and create complex expressions that power both dashboards and alerting rules. For instance, calculating the per-second rate of HTTP requests over the last 5 minutes:

rate(http_requests_total[5m])

This query capability transforms raw metrics into actionable insights, enabling you to detect anomalies, track trends, and understand system behavior at scale.

Best Practices for Prometheus Metrics

To maximize the effectiveness of your Prometheus monitoring:

• Use consistent naming conventions that follow Prometheus standards
• Keep label cardinality reasonable to avoid performance issues
• Choose appropriate metric types based on what you're measuring
• Document your metrics with HELP and TYPE annotations
• Set realistic scrape intervals based on your data resolution needs
• Leverage service discovery in dynamic environments

By understanding these fundamentals of Prometheus metrics, you'll be well-equipped to build robust monitoring solutions that provide deep visibility into your systems' health and performance.