scalable

Scalable Metrics Processing Architectures for a Hybrid South African Estate

Two hours into a Monday morning incident review, the pattern finally became obvious: every time the Cape Town Kubernetes cluster hit 40k active time-series, the entire metrics pipeline wobbled, alerts arrived late, and Grafana dashboards froze. The services…

Opsgenie

— 4 min read
Scalable Metrics Processing Architectures for a Hybrid South African Estate

Scalable Metrics Processing Architectures for a Hybrid South African Estate

Two hours into a Monday morning incident review, the pattern finally became obvious: every time the Cape Town Kubernetes cluster hit 40k active time-series, the entire metrics pipeline wobbled, alerts arrived late, and Grafana dashboards froze. The services survived; the observability layer did not. That meeting led directly to a redesign of our Scalable Metrics Processing Architectures — and more importantly, to a shift in how we think about metrics as an infrastructure problem, not a tooling checkbox.

Why Scale Breaks Differently in South African Observability Setups

Scaling metrics sounds generic, but local constraints change the shape of the problem. A typical South African or African estate might have:

  • On-prem Prometheus scraping legacy workloads in a Johannesburg data centre.
  • Cloud Kubernetes in eu-west-1 for latency and global backbone benefits.
  • Edge nodes in Lusaka, Gaborone, or Nairobi on flaky last-mile connectivity.
  • Strict POPIA requirements limiting what can leave the country, and when.

Layer ZAR-denominated cloud costs on top, with everyone chasing cheaper storage tiers and occasional spot instances, and the result is a metrics pipeline that is constantly juggling four pressures:

  • Cardinality: exploding label sets from microservices, multitenancy and “everything gets a tag”.
  • Retention: needing 90 days of data for capacity planning, but not wanting to pay Dublin prices for it.
  • Network: cross-region latency and intermittent connectivity from regional POPs.
  • Compliance: POPIA and local data sovereignty expectations from customers and auditors.

Scalable Metrics Processing Architectures in this environment demand more than “run a bigger Prometheus”. They need deliberate sharding, aggregation, tiering, and clear data ownership across Prometheus, Mimir, Loki, Tempo and the visual layer in Grafana.

From One Big Prometheus to a Layered Metrics Pipeline

Most teams start with one or two Prometheus servers and organically grow into a mess: ad-hoc federation, overlapping scrape configs, and dashboards wired to random endpoints. The turning point usually arrives when either:

  • Scrape intervals are silently extended (15s to 60s) to keep load under control.
  • Alerts fire late because recording rules are queued behind a flood of scrapes.

A layered pipeline fixes this by separating concerns into three stages:

  1. Edge collection and local resilience.
  2. Centralised, horizontally scalable storage and query.
  3. Derived insights and long-term cost optimisation.

Stage 1: Edge Prometheus with POPIA-aware Scopes

At the edge — whether that’s a small on-prem cluster in Durban or a VM in Lagos — the goal is stability and legal clarity:

  • Scrape everything locally at 15–30s intervals.
  • Store 24–48 hours of high-resolution data.
  • Tag metrics with environment, region, and data-sensitivity labels.

A typical YAML snippet for an edge scrape config in a Johannesburg DC might look like:

scrape_configs:
  - job_name: 'k8s-apps'
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets:
          - '10.0.12.21:8080'
          - '10.0.12.22:8080'
        labels:
          region: 'za-jhb-dc1'
          env: 'prod'
          data_classification: 'internal'

That data_classification label matters later when deciding what is eligible for remote write to Mimir in a foreign region, and what must stay inside South African borders. This is not theoretical; POPIA enforcement risk has made teams more cautious about blanket exports of fine-grained telemetry data.

Stage 2: Mimir and Cross-Region Realities

Once local Prometheus instances are stable, the second stage is a horizontally scalable metrics backend. Grafana Mimir offers multi-tenant, highly available, long-term storage and query, built to handle millions of active series and decades of data at scale. In practice, for many African teams, Mimir is deployed either:

  • In eu-west-1, riding on AWS and other hyperscaler services.
  • In a local cloud provider with peering to corporate networks.

The trade-off is stark:

  • eu-west-1 brings lower latency to global users, but longer RTT from ZA and regional sites.
  • Local hosting reduces network distance but can be more expensive or limited in managed options.

Remote write from local Prometheus to Mimir needs to be tuned deliberately for our conditions:

remote_write:
  - url: 'https://mimir-gateway.mycorp.eu-west-1/api/v1/push'
    queue_config:
      capacity: 20000
      max_shards: 8
      min_shards: 1
      batch_send_deadline: 10s
      retry_on_http_429: true
    write_relabel_configs:
      - source_labels: ['data_classification']
        regex: 'restricted'
        action: 'drop'

Key points here:

  • queue_config absorbs short connectivity issues and limits the number of concurrent shards to keep bandwidth predictable over constrained links.
  • write_relabel_configs enforces our POPIA stance by dropping metrics labelled as restricted before they leave the DC.

Once data lands in Mimir, Grafana dashboards and alerts can query a single, centralised metrics lake, freeing SREs from remembering which Prometheus instance holds which data. This central point also becomes the anchor for Tempo (traces) and Loki (logs), enabling true correlation across signals.

Stage 3: Derived Metrics and Cost-Controlled Retention

Keeping every raw time-series for 180 days is rarely viable at South African cloud price points. The emerging pattern — informed by recent global practice — is:

  • Retain raw, high-cardinality metrics for 7–14 days.
  • Pre-aggregate and downsample business-critical metrics for 90–365 days.

PromQL recording rules on Mimir help here. For instance, instead of storing per-pod latency for a year, we can store per-service SLO metrics:

groups:
  - name: service_slo_rules
    interval: 60s
    rules:
      - record: slo:request_latency_p90_seconds:service
        expr: histogram_quantile(
          0.9,
          sum by (service, le) (
            rate(http_request_duration_seconds_bucket[5m])
          )
        )
      - record: slo:error_rate:service
        expr: sum by (service) (
          rate(http_requests_total{status=~"5.."}[5m])
        ) /
        sum by (service) (
          rate(http_requests_total[5m])
        )

These aggregated metrics are cheaper to store and query over long periods, and they align directly with SLO dashboards and alerting in Grafana. The shift is conceptual: we scale not by hoarding every detail, but by consciously selecting the metrics that encode our service-level reality.

Design Patterns for Resilient Metrics at Load-Shedding Time

Load-shedding introduces a uniquely South African failure mode: entire racks, on-prem DC segments, or access links vanish on a schedule that is semi-predictable but not guaranteed. Resilient Scalable Metrics Processing Architectures need specific patterns to cope:

Pattern 1: Regional Observability Cells

Instead of one giant “prod” metrics estate, split into regional cells:

  • Each cell has its own Prometheus, Loki, and Tempo.
  • Cells are loosely federated to Mimir and a central Grafana deployment.
  • Global dashboards are built off Mimir; local dashboards fall back to edge Prometheus when needed.

During a stage 4 load-shedding window, if the Johannesburg cell goes dark, the Cape Town and eu-west-1 cells continue to produce metrics. Alerts can be scoped so that “regional outage” alerts fire from a

Read more