Real-Time Infrastructure Health Analytics when the lights might go out

Real-Time Infrastructure Health Analytics when the lights might go out

At 18:07 on a winter evening in Johannesburg, the production dashboard looked perfectly calm: CPU under 40%, latency under 120 ms, error rates flat. At 18:09, Eskom flipped the Stage 4 switch, a regional edge node dropped off the map, and checkout failures spiked for users on a specific fibre provider. The metrics caught up only after users had already complained on Twitter. That gap – between what the dashboards showed and what was really happening – is exactly where Real-Time Infrastructure Health Analytics earns its keep.

DevOps teams talk a lot about “real time”, but in practice many stacks are running on five‑minute scrapes, sporadic log shipping, and dashboards that hide locality. In a South African and broader African context – with load shedding, patchy last‑mile connectivity, POPIA constraints, and ZAR‑denominated cloud bills – an observability stack has to be sharp, cost‑aware, and topology‑aware, or it’s just colourful graphs.

From pretty dashboards to an actual health model

The core argument: real‑time analytics for infrastructure health is less about faster charts and more about a better model of “healthy” that reflects real constraints. For a hybrid estate spanning on‑prem, local DCs in Midrand or Cape Town, and workloads in eu‑west‑1, health is multidimensional:

• Can this service still meet its SLOs when we lose a regional DC during Stage 6?
• Will our eu‑west‑1 dependency survive the next undersea cable wobble?
• Can we see POPIA‑sensitive data paths clearly enough to prove compliance?
• Are we spending more ZAR on metrics than on the underlying microservice?

To capture this, a real‑time health analytics layer needs three ingredients:

• High‑resolution signals from Prometheus, Loki and Tempo, aggregated at scale by Mimir.
• Topology awareness – regions, ISPs, DCs, energy state, regulatory zones.
• Decision‑ready views in Grafana: not just panels, but health scores, risk levels and impact estimates.

Teams that get this right end up using Grafana as a living model of their infrastructure, not merely a passive reporting tool. That shift is where real‑time analytics starts to change incident outcomes instead of just documenting them.

Designing a load-shedding-aware health pipeline

Load shedding is predictable enough to model, but disruptive enough to take out entire racks, access networks, and edge caches. The pipeline below has emerged as a practical pattern in local teams:

1. Collect granular infrastructure signals:
   • Node exporter and k8s metrics in Prometheus (CPU, memory, disk, pod status).
   • Power and environmental metrics where possible (UPS status, generator fuel, battery levels) from DC telemetry.
   • Network metrics split by ISP/region (loss, latency, jitter).
2. Ingest event context:
   • Schedules from EskomSePush or internal ops feeds as logs via Loki.
   • Change windows and deployment markers as structured logs or Tempo spans.
3. Aggregate and normalise:
   • Push TSDB blocks into Mimir for long‑term, multi‑tenant analysis.
   • Derive “risk state” labels per node: normal, load_shedding_imminent, on_generator, etc.
4. Surface health scores and alerts:
   • Expose composite health metrics in Grafana dashboards and alert rules.
   • Colour and threshold by risk state, not only by raw utilisation.

The point is not to predict Eskom, but to stop treating infrastructure health as purely technical. A DC running “normally” with 99.95% uptime and clean metrics means something different when its UPS battery has 15 minutes left and the next load shedding window starts in 20.

PromQL: turning raw signals into a health score

A common pattern is a composite health score per node or service. For a Kubernetes node, we can define something like:

100 - (
  (node_load15 > 4) * 20
+ (node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.1) * 20
+ (kube_pod_status_ready{condition="false"} > 0) * 30
+ (infra_power_risk{state=~"load_shedding_imminent|on_generator"} > 0) * 30
)

This PromQL expression assumes:

• infra_power_risk is a custom metric exported by a small power‑status service.
• Node load, filesystem usage, and pod readiness are already collected.

The result is a score from 0–100, where anything under, say, 60 merits attention. Panels can show this as a heatmap across clusters, with additional labels for DC, province, or ISP. When paired with annotations from Tempo for recent deployments, patterns become obvious: nodes already in a risky power state are far less tolerant of heavy rollouts.

Latency and health across Africa’s network reality

Cross‑region latency is a reality for teams depending on eu‑west‑1 or eu‑central‑1 from South Africa, Kenya or Nigeria. Average RTTs are survivable, but cable cuts and peering quirks can turn routing into chaos. In 2024, multiple reports highlighted undersea cable incidents causing widespread degradation across the continent, and teams relying solely on generic “up/down” checks found out the hard way that “mostly up” can still mean “functionally broken” for users.

For Real-Time Infrastructure Health Analytics, latency is not a single stat; it’s a distribution across:

• Access networks (LTE vs fibre vs fixed wireless).
• Regions (local DC vs eu‑west‑1 vs on‑prem mainframe).
• Workloads (API gateway vs internal admin interfaces).

Prometheus should be ingesting per‑client and per‑region latency histograms. Mimir makes it possible to store and query those high‑cardinality metrics without collapsing the platform. Grafana dashboards can then visualise not only p95 per service, but p95 per region‑ISP combination.

PromQL: latency heatmap per ISP and region

Assuming an HTTP metrics series annotated with isp and region labels:

histogram_quantile(
  0.95,
  sum(rate(http_request_duration_seconds_bucket{job="edge-api"}[5m]))
    by (le, isp, region)
)

This exposes how, for example, users on a particular mobile network traversing to eu‑west‑1 suddenly spike to 800 ms p95 while others stay under 200 ms. Health analytics is then:

• Flagging nodes or paths whose latency patterns deviate from normal for that ISP.
• Feeding that into traffic‑shaping: sending certain flows to a local DC when eu‑west‑1 degrades for a specific network.
• Highlighting to business stakeholders the real user impact of “minor” routing incidents.

POPIA-aware observability: health without leaking data

Another axis of “health” in South African infrastructure is compliance. POPIA and related regional regulations push teams to keep personal information close, minimise cross‑border replication, and prove that they understand where data flows. That clashes with traditional observability defaults that happily ship everything to a single US‑hosted backend.

A real‑time health view that ignores data locality is incomplete. For hybrid stacks with on‑prem clusters and cloud workloads, a practical pattern looks like this:

• Prometheus and Loki inside each POPIA zone, scraping and tailing only anonymised or aggregated signals.
• Mimir for centralised metrics, but with strict data‑classification labels; sensitive metrics remain local and only derived aggregates are shipped.
• Tempo