Start with the failure boundaries, not the product list

Scalable Metrics Processing Architectures: Keeping Prometheus Useful When the Estate Outgrows One Cluster

At 02:13 during another round of load-shedding, a Kubernetes cluster in Johannesburg started buffering telemetry. Prometheus kept scraping, but its remote-write queue grew faster than the backup power could stabilise the node pool. By the time the site recovered, dashboards were missing the exact window needed to explain why checkout latency had increased.

That failure was not caused by an undersized Grafana dashboard. It exposed an architectural problem: collection, transport, storage and querying had been treated as one system. Scalable Metrics Processing Architectures separate those concerns so that a noisy workload, a broken link or a regional outage does not take observability down with the application.

Start with the failure boundaries, not the product list

A useful design begins by mapping where metrics are generated and where they must remain available. A South African organisation may have workloads in a Cape Town data centre, a Johannesburg Kubernetes platform, a small branch estate and a cloud region in eu-west. Those locations have different network costs, latency and failure modes.

• Collection: Prometheus or an OpenTelemetry Collector gathers metrics close to the workload.
• Processing: Relabelling, filtering, aggregation and enrichment reduce unnecessary traffic and cardinality.
• Transport: Remote write or OTLP moves data across a network that may be slow, expensive or intermittently unavailable.
• Storage: A durable backend retains samples beyond the lifetime of an individual Prometheus server.
• Query and alerting: Engineers need recent, fast data even while long-term storage or another region is degraded.

Grafana is most useful when it exposes these boundaries rather than hiding them. Dashboards should show scrape health, remote-write backlog, ingestion latency, dropped samples and query duration alongside application indicators.

Pattern one: local Prometheus with central Mimir

The simplest scalable pattern keeps Prometheus near the targets and sends a second copy of the data to a central metrics backend. Prometheus remains responsible for scraping and local rule evaluation, while Grafana Mimir provides horizontally scalable, long-term storage and multi-tenant querying. Mimir receives Prometheus samples through the remote-write API; its distributors handle writes and query frontends handle reads.[1]

global:
  scrape_interval: 30s
  external_labels:
    site: jhb
    environment: production

remote_write:
  - url: https://metrics.example.net/api/v1/push
    headers:
      X-Scope-OrgID: platform-jhb
    queue_config:
      capacity: 10000
      max_shards: 20
      min_shards: 2
      max_samples_per_send: 2000
    send_exemplars: true

The local Prometheus instance should still retain enough data to answer immediate operational questions during an upstream outage. A 6- to 24-hour retention period is often more practical than trying to make every query depend on the WAN. The remote-write queue is a buffer, not infinite storage: measure its age and saturation, and alert before it becomes a silent data-loss mechanism.

For a Johannesburg-to-eu-west deployment, avoid routing every scrape across the ocean. Scrape in Johannesburg, apply local filtering, and forward only the resulting stream. This reduces egress costs in ZAR terms and prevents eu-west latency from becoming a dependency for collection.

Pattern two: an edge collector for unstable links

Remote sites with unreliable last-mile connectivity need an intermediary. Grafana Alloy can collect Prometheus and OpenTelemetry telemetry, apply processing pipelines, and forward metrics to Mimir while also supporting logs and traces for Loki and Tempo.[2]

Deploying an Alloy instance per site creates a controlled choke point. It can drop debug series, normalise labels, batch traffic and provide a local queue. The branch Prometheus server then sends to Alloy over the LAN, rather than each application or exporter opening a separate path to the central platform.

prometheus.relabel "edge_filter" {
  forward_to = [prometheus.remote_write.mimir.receiver]

  rule {
    source_labels = ["__name__"]
    regex         = "go_.*|process_.*"
    action        = "keep"
  }

  rule {
    source_labels = ["pod"]
    regex         = ".{81,}"
    action        = "labeldrop"
  }
}

prometheus.remote_write "mimir" {
  endpoint {
    url = "https://metrics.example.net/api/v1/push"
  }
}

Filtering must be deliberate. Dropping every infrastructure metric to save bandwidth will make an outage harder to diagnose. Keep service-level indicators, saturation signals, request rates, error rates and the metrics needed for recording rules. Remove duplicate exporter output and labels that encode request IDs, full URLs or other unbounded values.

Cardinality is the processing bill you choose to receive

Most metrics platforms fail economically before they fail technically. A label such as customer_id, session_id or an unnormalised URL path can create a new time series for every value. In a ZAR-sensitive cloud budget, that series explosion appears as storage growth, higher ingestion volume and slower queries.

Use bounded dimensions for metrics: service, route template, HTTP method, status class, region and workload. Keep high-cardinality detail in Loki logs or Tempo traces, where it can be sampled and queried when needed. Exemplars can connect a latency time series to a trace without turning every trace identifier into a permanent metric label.

Recording rules are another form of processing. Calculate expensive ratios once and store the result rather than repeatedly scanning months of raw counters:

- record: service:http_error_rate5m
  expr: |
    sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
    /
    sum by (service) (rate(http_requests_total[5m]))

Rules should run close to the data they evaluate. Local rules protect alert latency when the central backend is unreachable; central rules provide fleet-wide views. Do not create both blindly. Duplicate evaluation increases cost and can produce confusing differences during partial outages.

Pattern three: federated regions with a shared query layer

Large estates often need regional autonomy rather than one global ingestion endpoint. Run a Mimir deployment, or a durable Prometheus-compatible backend, in each major region. Applications write locally; a global Grafana view queries selected regions through a consistent tenancy and label model.

This design reduces cross-region write traffic and keeps alerting close to the failure. A payment API in Cape Town should not wait for eu-west to acknowledge every sample before its availability alert can fire. Global dashboards can tolerate a little more latency; paging cannot.

Standardise external labels such as cluster, site, region and environment. Without them, a global PromQL query can merge identical service names from separate estates or make an apparent fleet-wide improvement that is actually one healthy region hiding one failing region.

Multi-tenancy also matters when several teams share the platform. Mimir supports tenant isolation and object-storage-backed long-term retention, while scaling its components independently.[1] Query frontends can protect ingestion from expensive dashboard requests, but engineers still need recording rules, query limits and dashboards that do not request five years of one-second data by default.

Design alerting for load-shedding and degraded telemetry

Power interruptions create a dangerous false sense of calm: if a site disappears, “no traffic” can resemble “healthy”. Alert on telemetry availability as a separate signal. A missing-site alert should include the last sample timestamp, the Prometheus up state and the age of the remote-write queue.

• Use local alerts for service failures that require immediate action.
• Use central alerts for fleet-wide capacity, regional comparison and compliance reporting.
• Include site and power-zone labels in notifications where they help the responder route the incident.
• Suppress derivative alerts when a site-level telemetry outage is already firing.
• Test alert behaviour while remote storage, DNS, VPN links and backup power are deliberately unavailable.

POPIA and data-sovereignty requirements may also rule out sending certain labels or exemplars outside the country. Metrics should contain operational dimensions, not personal information. Keep sensitive request context in an approved local logging or tracing boundary, and document exactly which telemetry crosses the regional link.

A practical architecture decision

For a small estate, start with Prometheus, short local retention and remote write to Mimir. Add an edge Alloy layer when connectivity, protocol diversity or filtering justifies it. Move to regional Mimir clusters when cross-region links, sovereignty requirements or alert latency make central ingestion a liability.

Validate the design with failure tests rather than dashboard screenshots:

1. Block the WAN and measure how long local collection and alerting remain useful.
2. Generate a controlled cardinality spike and verify that ingestion limits protect the platform.
3. Stop Mimir ingestion and confirm remote-write queues recover without exhausting disk.
4. Query during a regional outage and check that global dashboards fail partially, not completely.
5. Restore the link and verify that backfill traffic does not starve current samples.

Scalable Metrics Processing Architectures are less about selecting the biggest metrics database than choosing where to collect, where to compute, where to retain and where to fail. Prometheus provides excellent local control; Mimir supplies durable scale; Alloy makes the edges manageable; Loki and Tempo preserve the detail that metrics should not carry. The winning design is the one that still explains an incident after the network, the power and the most expensive dashboard have all had a bad day.