Start with the shape of the metrics problem

Scalable Metrics Processing Architectures: Designing for Growth Without Burning the ZAR Budget

At 02:17, a load-shedding event switched a Johannesburg data centre to generators. The applications stayed up, but the monitoring stack did not: Prometheus queues filled, remote-write retries multiplied, and the central metrics cluster received a burst of delayed samples when connectivity stabilised. The dashboards eventually recovered. The alert storm took longer.

That incident exposed a common architectural mistake: treating metrics processing as a single pipeline instead of a set of independently scalable paths. Scalable Metrics Processing Architectures need to absorb bursts, tolerate disconnected sites, control cardinality and keep query traffic from starving ingestion. Grafana Mimir’s microservices architecture reflects this separation by scaling distributors, ingesters, query frontends and queriers independently.[1]

Start with the shape of the metrics problem

Before choosing components, classify the workload. A small Kubernetes cluster and a national retail platform may both use Prometheus, but their failure modes differ significantly.

• Collection: exporters, application instrumentation, Kubernetes service discovery and infrastructure agents generate samples.
• Transport: remote write or OTLP moves data between sites, clusters and regions.
• Processing: recording rules, relabelling, aggregation and deduplication reduce or transform the stream.
• Storage: short-term local retention supports fast troubleshooting; long-term storage supports trends and compliance.
• Querying: dashboards, alerts, ad hoc PromQL and automated SLO calculations compete for resources.

Capacity planning should therefore track more than samples per second. Measure active series, label churn, remote-write queue depth, ingestion latency, query concurrency, rule evaluation duration and object-storage growth. A ZAR-denominated cloud bill can rise sharply from a single unbounded label such as request_id, even when CPU usage looks modest.

Cardinality is usually the first scaling boundary. Keep stable dimensions such as service, environment, region and status class. Push high-volume detail into logs and traces, where it can be sampled or searched when needed. Grafana makes the relationship visible by placing metrics, Loki logs and Tempo traces in the same investigation workflow rather than forcing every diagnostic field into a time-series label.

A resilient architecture for hybrid African estates

A practical design uses a local collection tier at each meaningful failure domain, followed by a central, horizontally scalable metrics tier.

1. Prometheus or an agent scrapes workloads inside each Kubernetes cluster, branch network or on-premises environment.
2. Local agents apply relabelling, drop unsafe labels and evaluate only the recording and alerting rules that must work during isolation.
3. Buffered remote write sends selected metrics to a regional Mimir cluster.
4. Mimir stores durable data in object storage and exposes PromQL for dashboards, alerts and automation.
5. A second region receives replicated or independently collected data where the recovery objective justifies the cost.

The local tier is not redundant decoration. It protects operations when a site loses its WAN route, power, DNS dependency or upstream cloud connection. Local alerts should cover immediate hazards: node temperature, UPS state, generator fuel, disk exhaustion and service availability. Central alerts can handle fleet-wide conditions, capacity trends and cross-region SLOs.

Prometheus remote write sends batched, compressed samples to Mimir, where distributors validate and route incoming data to ingesters.[1] This makes the write path independently scalable. In a hybrid estate, run more than one remote-write agent or collector per site so a maintenance event does not turn into a monitoring blind spot; Mimir documentation explicitly supports paired remote-write agents for this purpose.[8]

Separate the write path from the read path

Metrics ingestion and dashboard querying have different resource profiles. Ingestion needs predictable memory, WAL durability and stable network capacity. Queries need CPU, memory and concurrency controls. Combining both on the same oversized virtual machines is simple initially and painful during an incident.

Use a topology in which:

• Distributors remain stateless and scale with incoming traffic.
• Ingesters provide replication and recent-data availability.
• Query frontends queue, split and cache expensive PromQL requests.
• Queriers scale separately for dashboard and API demand.
• Compactors and store-gateway components are sized around object-storage retention and long-range queries.

This separation also improves failure behaviour. A dashboard query storm should not prevent applications from writing samples. A slow object-storage operation should not necessarily block local ingestion. Mimir’s 2025 architecture work specifically focused on decoupling read and write paths so an outage in one does not automatically affect the other.[9]

Set explicit tenant and query limits. Limit maximum samples per query, query range, concurrency and series returned. Add recording rules for common dashboards instead of allowing every panel to calculate a multi-month aggregation independently. A query frontend can help, but it cannot rescue an unbounded data model.

Use metrics, logs and traces for different jobs

Scalability improves when each signal carries the detail it handles best. Metrics should answer “how many, how often and how bad”. Loki should retain event context such as deployment messages, upstream error bodies and power-system transitions. Tempo should show where latency accumulates across services and regions.

For example, a slow checkout alert might begin with:

histogram_quantile(
  0.99,
  sum by (le, service, region) (
    rate(http_request_duration_seconds_bucket{service="checkout"}[5m])
  )
) > 1.5

The metric identifies the symptom without creating a series for every customer or request. A trace can then reveal whether the delay occurs between Johannesburg and eu-west, while Loki can show a timeout or deployment event. This is more economical than encoding every diagnostic attribute in Prometheus labels.

Recent ecosystem changes make the boundaries less rigid. Loki 3.0 added native OpenTelemetry support, reducing the need for a separate Loki exporter in some pipelines, while Tempo releases in 2024 improved TraceQL performance and added broader gRPC streaming support.[5] OpenTelemetry can simplify collection across mixed languages and vendors, but standardisation does not remove the need for sampling, limits and local buffering.

Design for outages, not just steady state

Load-shedding-aware observability needs its own runbook. During planned outages, silence alerts that depend on unavailable infrastructure, but keep alerts for unexpected loss of redundancy, battery failure and unsafe temperature. Do not simply mute every alert in a region: that converts a controlled power event into an information blackout.

For disconnected sites, configure bounded queues and monitor them as first-class signals:

remote_write:
  - url: https://metrics.example.invalid/api/v1/push
    queue_config:
      capacity: 20000
      max_shards: 8
      min_shards: 1
      max_samples_per_send: 2000
      batch_send_deadline: 5s
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "debug_.*"
        action: drop

The exact values require load testing. A queue is not infinite storage: once its capacity is exhausted, samples are lost or delayed. Alert on queue saturation, retry rate and oldest pending sample age. Test the full sequence—WAN loss, prolonged outage, recovery burst and object-storage degradation—instead of testing only whether a Prometheus pod restarts.

Regional placement matters too. Keep South African operational data in an approved location when POPIA, contractual controls or internal policy require data sovereignty. Send only aggregated, non-sensitive metrics across borders where possible. Logs and traces often contain more identifying information than metrics, so applying sovereignty rules only to the metrics backend creates a false sense of safety.

Control cost with deliberate resolution

Long-term storage does not mean every metric deserves the same resolution forever. Retain high-resolution infrastructure and application metrics for incident response, then use recording rules for lower-resolution business and capacity trends. Keep raw data longer only when it supports a defined investigation, audit or engineering decision.

• Drop exporter metrics that nobody queries or alerts on.
• Aggregate per-instance series into service-level series where instance detail is not operationally useful.
• Use separate tenants or policies for production, non-production and experimentation.
• Limit dashboard refresh intervals for expensive, wide-range panels.
• Estimate object-storage, egress and query costs before extending retention.

Native histograms are another option for latency-heavy systems. They can provide higher-resolution distributions with less manual bucket design than classic histograms, although support and operational maturity must be checked across the complete Prometheus, Mimir and query path.[2][4] They are not a licence to instrument every endpoint indiscriminately; histogram volume still needs a budget.

Prove the architecture with failure tests

A scalable design is demonstrated by controlled degradation, not by a diagram. Test these cases quarterly:

• One collector disappears while applications continue serving traffic.
• A site loses WAN connectivity for longer than the local queue can retain safely.
• Remote write returns errors during a recovery burst.
• A dashboard fleet issues expensive, concurrent range queries.
• One Mimir zone becomes unavailable.
• Object storage is slow or temporarily unreachable.
• A new label causes a sudden cardinality increase.

Define success in operational terms: no loss of critical local alerts, bounded ingestion delay, controlled query latency and a known maximum recovery gap. The best Scalable Metrics Processing Architectures do not promise perfect visibility during every failure. They preserve the signals needed to make the next safe decision—and recover cleanly when the network, power or cloud bill stops behaving.