The incident was not a CPU problem

Real-Time Infrastructure Health Analytics: From Noisy Dashboards to Faster Incident Decisions

At 18:07, a payment platform’s Johannesburg API started timing out. CPU stayed below 40%, memory looked healthy, and the Kubernetes dashboard showed green workloads. The useful clue was elsewhere: latency from the Cape Town application tier to a dependency in eu-west had doubled, while a load-shedding event forced one local network path onto backup power.

That is the practical problem Real-Time Infrastructure Health Analytics must solve. It is not a prettier collection of dashboards. It is the ability to combine metrics, logs, traces and infrastructure context quickly enough to distinguish a failing service from a failing dependency, network path, power event or deployment.

The incident was not a CPU problem

Traditional infrastructure monitoring tends to ask whether a resource has crossed a threshold. That works for a saturated database host. It is less useful when a service is technically “up” but its users experience intermittent failures.

A useful health view combines four questions:

• Is the service available?
• How long does a successful request take?
• Where is time being spent across dependencies?
• What changed in the environment when the symptom began?

Prometheus remains the foundation for numeric signals such as request rate, error rate, saturation, queue depth and network loss. Mimir provides a horizontally scalable, long-term metrics store when multiple Prometheus servers or regions need a common query layer. Loki handles structured application and system logs without forcing every event into a high-cardinality metrics label.

Tempo adds the missing request path. Its traces can connect a slow API request to a database span, an external payment provider or a cross-region hop. Tempo is designed to connect traces with logs and metrics, and supports OpenTelemetry alongside Jaeger and Zipkin protocols.[2]

Grafana turns those signals into an investigation surface rather than separate consoles. A panel showing rising latency should lead to the relevant trace; a trace span should lead to matching Loki entries; and the dashboard should retain enough regional and deployment context to explain what changed.

Build health around symptoms, not server folklore

Start with user-facing service-level indicators. For an API, that normally means availability, latency and correctness. Host CPU is still useful, but it should explain a symptom rather than define health on its own.

A latency panel can begin with a Prometheus histogram:

histogram_quantile(
  0.95,
  sum by (le, service, region) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

Break the result down by region and service. A single global percentile can hide the fact that Johannesburg is healthy while Cape Town has packet loss, or that local traffic is fine while calls to eu-west are slow.

Pair latency with an error-budget view. This query calculates the proportion of failed requests over five minutes:

sum(rate(http_requests_total{
  status=~"5..",
  service="checkout"
}[5m]))
/
sum(rate(http_requests_total{
  service="checkout"
}[5m]))

Labels deserve discipline. Adding user_id, full URLs or request IDs to a metric can create expensive cardinality. Put those values in logs or traces instead. Keep metric labels bounded: service, method, status class, region, cluster and perhaps a carefully controlled route name.

Correlate the three signals during the first five minutes

The first five minutes of an incident should not require manually copying timestamps between tools. Instrument services with OpenTelemetry or another consistent tracing approach, propagate trace IDs through HTTP and messaging, and include the trace ID in structured logs.

A Loki query can then narrow the operational noise:

{service="checkout", region="jhb"}
| json
| level="error"
| duration_ms > 1000

From there, a derived field can open the matching Tempo trace. The trace may show that the checkout service spent 80 milliseconds processing locally but waited 1.4 seconds on an inventory call in eu-west. That is a very different remediation from increasing the checkout deployment’s CPU limit.

For infrastructure logs, normalise fields such as cluster, node, availability_zone, region, power_state and network_path. The last two are especially useful in South African estates where generator transitions, UPS alarms, fibre failover and last-mile instability can coincide with application symptoms.

Recent observability practice is moving towards unified infrastructure and application visibility rather than isolated host monitoring. Grafana’s 2025 survey reported that 85% of organisations were using or investigating unified application and infrastructure observability, while OpenTelemetry adoption continued to gain momentum.[8] The implication is practical: telemetry pipelines should be designed for correlation from the start, not retrofitted after the next outage.

Make regional and power context first-class data

Load-shedding-aware alerting does not mean suppressing alerts whenever the grid is under pressure. It means adding context so the on-call engineer can interpret symptoms correctly.

• Record UPS runtime, battery state, generator state and transfer events as metrics.
• Annotate Grafana dashboards with planned maintenance and load-shedding windows.
• Alert on reduced redundancy before the application fails.
• Separate “service unavailable” from “service reachable only through a degraded path”.
• Measure packet loss and round-trip latency between offices, data centres, cloud regions and critical providers.

For a hybrid platform, deploy Prometheus agents or collectors close to each estate and remote-write selected metrics to Mimir. Keep scraping local systems local when connectivity is unreliable, then forward data when the path recovers. This avoids turning a temporary WAN problem into a complete observability blind spot.

Use region labels consistently across Johannesburg, Cape Town, Durban, cloud zones and eu-west. A dashboard should answer whether an incident is regional, cross-region or global without requiring a spreadsheet of naming conventions.

Control cost, retention and POPIA exposure

Real-time analytics becomes wasteful when every signal receives the same retention period. Keep high-resolution metrics for active troubleshooting, downsample or aggregate older data, and retain traces according to their operational value. Tempo’s object-storage model can support high-volume trace retention without requiring a large relational backend.[7]

Logs require stricter handling. Avoid shipping identity numbers, payment details, access tokens and unnecessary customer payloads into Loki. Scrub sensitive fields at the collector or application boundary. Decide which telemetry must remain in South Africa and which can be replicated to another region, documenting the decision against organisational POPIA requirements and contractual obligations. South Africa’s 2024 National Policy on Data and Cloud also highlights data storage, interoperability, scalability and cost optimisation as policy concerns.[9]

ZAR cloud-cost pressure makes cardinality governance an engineering control, not a finance exercise. Track active series, ingestion volume, query cost and retention by team. A weekly report that identifies the ten fastest-growing labels often finds accidental dimensions introduced by a release.

Alert on decisions the team can act on

Alerts should describe an operational decision. “Node CPU high” is weak if no service is affected. “Checkout error budget burning in Johannesburg while eu-west dependency latency exceeds 1 second” tells the responder where to start.

Useful alerts combine symptoms with duration and scope:

• Page when availability breaches the service objective for a sustained period.
• Page when latency and error rate rise together for a customer-facing route.
• Ticket when telemetry ingestion falls below normal but service health remains intact.
• Warn when a region loses redundancy, even before requests fail.
• Suppress duplicate dependency alerts when a known upstream incident explains them.

Use recording rules for expensive, frequently viewed calculations. Link each alert to a dashboard containing the relevant PromQL, LogQL and trace navigation. The runbook should say what evidence confirms a regional network fault, what traffic can be shifted, and which data must not be copied into an incident channel.

The successful platform is not the one with the most panels. It is the one that reduces the distance between a user symptom and a defensible action. When metrics show the blast radius, logs explain the event, traces reveal the waiting dependency and regional context explains the environment, Real-Time Infrastructure Health Analytics becomes operational decision support rather than another monitoring project.