Scalable Metrics Processing Architectures Begin with Boundaries

Scalable Metrics Processing Architectures: Stop Letting Cardinality Set Your Cloud Bill

At 02:17, a payment service started timing out across a Johannesburg Kubernetes cluster. The application was healthy enough to serve requests, but its Prometheus pods were restarting under memory pressure. A dashboard query had introduced customer_id into a metric label; within minutes, a normal deployment had become a time-series explosion.

That incident is the practical reason to design Scalable Metrics Processing Architectures before the estate becomes large. The problem is not simply storing more samples. It is controlling cardinality, separating ingestion from querying, surviving connectivity gaps, and keeping enough regional data available to diagnose an outage without sending every byte to Europe.

Scalable Metrics Processing Architectures Begin with Boundaries

A useful architecture separates four jobs:

• Collection: Prometheus or an agent discovers targets and scrapes metrics.
• Local reliability: a regional component buffers, evaluates urgent alerts, and continues operating when the WAN is impaired.
• Durable storage: Mimir receives remote-written metrics and stores them for longer retention.
• Investigation: Grafana queries metrics alongside Loki logs and Tempo traces.

Prometheus remains excellent at scraping, rule evaluation, and short-term local storage. It is not normally the right place to keep years of metrics from every cluster. A common pattern is to run one or more Prometheus instances per site or Kubernetes cluster, then remote-write selected data to Mimir. Mimir provides horizontally scalable, highly available, multi-tenant long-term storage for Prometheus and OpenTelemetry metrics.[3]

The boundary matters in South African environments. A cluster in Cape Town may have low-latency access to local services but variable latency to eu-west. If every alert depends on a remote query, a fibre fault can become an observability outage. Local Prometheus rules should therefore cover symptoms that require immediate action: node availability, disk exhaustion, API error rates, and service-level objectives.

Use the Edge as a Reliability Layer, Not Just a Scraper

For a hybrid estate, place collection close to the workloads. Prometheus can scrape on-premise servers, cloud workloads, and Kubernetes services without making the central metrics backend part of the request path. Remote write then transfers batches asynchronously.

A minimal example looks like this:

global:
  scrape_interval: 30s
  evaluation_interval: 30s

remote_write:
  - url: https://metrics.example.net/api/v1/push
    queue_config:
      capacity: 10000
      max_shards: 20
      min_shards: 1
      max_samples_per_send: 2000

rule_files:
  - rules/local-alerts.yml

The values are not universal defaults. Queue capacity must be tested against the expected outage window, sample rate, disk capacity, and available bandwidth. A remote-write queue is not infinite storage. If the link to a central Mimir cluster remains unavailable for long enough, samples will eventually be dropped unless the local design includes an intentional buffer or a second destination.

Load-shedding changes the failure model. Power may return in stages, network paths may flap, and a large group of exporters may reconnect simultaneously. Avoid designing alerts that treat every short scrape gap as a customer-impacting incident. Use a suitable for duration, alert on sustained loss of coverage, and keep power-related maintenance windows separate from genuine service failures.

Grafana is useful here because the same dashboard can show local scrape health, remote-write backlog, Mimir ingestion status, and the application’s own service indicators. That makes “the service is down” distinguishable from “the monitoring path is down”.

Cardinality Is an Engineering Constraint

Most metrics incidents are label-design incidents wearing a storage problem’s clothing. A label such as method usually has a bounded set of values. Labels such as user_id, request_id, full URL, or an unnormalised error message do not.

Before adding a label, ask whether its values are:

• Bounded and operationally meaningful.
• Stable enough to aggregate over time.
• Useful in an alert or dashboard query.
• Worth paying for at the expected traffic volume.

Keep high-cardinality investigation data in logs and traces. Loki is designed for horizontally scalable, multi-tenant log aggregation and uses labels to organise streams; its documentation explicitly distinguishes logs from Prometheus-style metrics and describes deployments ranging from a single binary to fine-grained microservices.[8] Do not turn every log field into a Loki label either. Store volatile fields in the log body and filter them with LogQL when needed.

For example, the metric below is generally more manageable than one labelled by customer:

sum by (service, route, status_class) (
  rate(http_requests_total[5m])
)

Normalise routes before instrumentation. /orders/12345 and /orders/12346 should usually become /orders/:id. If a team needs per-customer analysis, emit a structured log or trace attribute and apply access controls appropriate to the data.

Choose Mimir’s Shape Around the Failure You Expect

Mimir’s microservices architecture allows ingestion, querying, compaction, and storage-related components to scale separately. Its documented ingest-storage architecture uses Kafka to decouple reads and writes; the classic architecture uses stateful ingesters with local write-ahead logs.[7]

That choice should follow the operational problem rather than fashion:

• Smaller deployments: a simpler architecture reduces the number of components to monitor and upgrade.
• Growing multi-tenant estates: separate scaling paths become valuable when dashboards, recording rules, and ingestion compete for resources.
• Unreliable links: retain a local collection and alerting layer; do not expect Mimir to repair a broken last-mile connection.
• Strict retention requirements: use object storage and define retention by tenant or data class, rather than keeping everything at maximum resolution forever.

Tenant boundaries are particularly important when several business units or customers share the platform. Enforce limits for ingestion rate, series count, and query capacity. A single team’s accidental label can otherwise degrade everyone’s dashboards.

Cloud cost also needs to be visible in local currency. ZAR pressure often encourages aggressive retention cuts, but deleting the wrong data can make incident analysis impossible. A better approach is to retain lower-resolution recording rules for long-term trends, keep raw data for a shorter period, and sample traces according to service criticality. Model object storage, cross-region transfer, query compute, and operational labour together; the cheapest storage tier is not necessarily the cheapest architecture.

Correlate Metrics, Logs, and Traces Without Centralising Everything

Metrics answer how often and how badly. Logs explain what the application reported. Traces show where time was spent across service boundaries. Tempo provides a high-scale distributed tracing backend and integrates with Mimir, Prometheus, Loki, and Grafana; exemplars can link a metric observation directly to a trace.[10]

Use correlation deliberately:

• Add exemplars to latency and error metrics for sampled requests.
• Use trace IDs in structured logs, but avoid turning them into metric labels.
• Keep service, cluster, region, and environment naming consistent across all signals.
• Use Loki derived fields to jump from a log record to a Tempo trace.

For POPIA-sensitive workloads, decide where logs, traces, and metric labels may travel. Metrics can contain sensitive information if labels include account numbers, email addresses, or geographic detail. Keep raw telemetry in the required jurisdiction where possible, redact before export, and document which aggregated signals may leave the region. Data sovereignty is not solved by calling a dataset “observability data”.

Modernise Carefully: Native Histograms and OpenTelemetry

Prometheus 3.0, released in November 2024, highlighted OpenTelemetry support and native histograms. Native histograms can reduce the need to preselect fixed bucket boundaries and can provide higher-resolution latency data, but adoption still requires compatibility testing across clients, collectors, remote-write paths, storage, and query tooling.[13]

Do not migrate every latency metric on a deadline. Start with one service whose latency distribution is difficult to represent with classic buckets. Compare ingestion volume, query behaviour, dashboard results,