Define health as a chain, not a single gauge

Real-Time Infrastructure Health Analytics: Finding the Failure Before Customers Do

At 18:07, the Johannesburg API was still returning a healthy 200 status. The incident started elsewhere: a brief utility interruption switched a rack to battery power, packet loss appeared between the data centre and a cloud database, and request queues began growing. The first alert fired 11 minutes later, when checkout latency crossed its threshold.

That delay is the difference between Real-Time Infrastructure Health Analytics and a collection of attractive dashboards. The objective is not to watch every CPU graph. It is to correlate infrastructure signals quickly enough to explain what is changing, where it is changing, and whether customers are already affected.

Define health as a chain, not a single gauge

A server can have 40% CPU utilisation and still be unhealthy. A Kubernetes node can be Ready while its attached storage is timing out. An application can return successful HTTP responses while its dependency calls consume the entire latency budget.

A practical health model connects four layers:

• Availability: Can users and critical services reach the endpoint?
• Performance: Are latency, throughput and saturation within the service-level objective?
• Dependency health: Are databases, queues, identity providers and third-party APIs responding?
• Operating conditions: Are power events, network paths, capacity limits or deployments changing the environment?

Prometheus is well suited to the numerical layer: request rates, error ratios, queue depth, node pressure and probe results. Mimir provides horizontally scalable, highly available, multi-tenant long-term storage for Prometheus and OpenTelemetry metrics, which matters when several regions or business units need a common historical view.

Grafana should then present health as a decision surface rather than a wall of panels. The first row might show customer-facing availability, error-budget burn and p95 latency. The next row can expose the likely cause: packet loss, database saturation, pod restarts or power-related capacity changes.

Real-Time Infrastructure Health Analytics across Johannesburg, Cape Town and eu-west

South African estates often combine on-premises systems, local cloud regions and workloads in Europe. That topology creates a trap: a dashboard showing an average latency of 180 ms can conceal a completely different experience for users in Cape Town, Johannesburg and London.

Attach geography and service identity to every useful metric. Labels such as region, availability_zone, service, environment and dependency make comparisons possible, but uncontrolled labels can make a metrics system expensive and slow. Do not label metrics with request IDs, full URLs or user identifiers.

A simple PromQL query for a regional success ratio might look like this:

sum by (region) (
  rate(http_requests_total{
    job="checkout",
    status=~"2.."
  }[5m])
)
/
sum by (region) (
  rate(http_requests_total{
    job="checkout"
  }[5m])
)

Pair that ratio with a black-box probe from more than one network. A healthy result from a monitoring pod in the same cluster proves very little about a customer travelling over a congested last-mile link. Synthetic checks should measure DNS, TLS negotiation, time to first byte and total response time, not merely whether a port is open.

For cross-region dependencies, track both application latency and network latency. If requests from Johannesburg to eu-west suddenly slow down while local calls remain normal, the incident is probably not a general application regression. That distinction saves time during an escalation.

Use logs and traces to explain the metric

Metrics detect a pattern; logs and traces explain it. Loki is effective when logs are indexed around a small set of stable labels and the message body is searched with LogQL. Avoid turning every field into an index label. High-cardinality log labels create operational cost without improving incident response.

For example, this query can identify timeout messages from a checkout service while preserving the region dimension:

{service="checkout", environment="production"}
|= "context deadline exceeded"
| json
| duration_ms > 2000

Tempo completes the chain for distributed requests. A trace can show that the visible checkout delay was actually spent waiting for an inventory call, a token exchange or a database connection. With exemplars, an engineer can move from a latency time series to a representative trace instead of searching blindly through thousands of events.

The useful workflow is deliberately short:

1. Start with the customer symptom: error rate, latency or failed synthetic probe.
2. Filter by region, service and deployment version.
3. Use a metric exemplar or trace ID to open a Tempo trace.
4. Jump from the slow span to related Loki logs.
5. Compare the timestamp with node, network, power and dependency signals.

This correlation is where Grafana earns its place: metrics, logs and traces can be navigated as one investigation rather than three disconnected tools.

Make load-shedding events visible without paging on every outage

Load-shedding-aware monitoring requires more than adding “stage” to a dashboard. The operational question is whether the estate is entering a degraded mode and whether that mode is controlled.

Record the expected power state as an event or gauge, then correlate it with UPS runtime, generator status, node availability, replication lag and traffic-routing changes. A power transition should not page the on-call engineer if redundancy is healthy. It should page when the transition removes the redundancy that protects a critical service.

Alert on consequences, not merely conditions. For example:

• Page when customer availability falls below the agreed objective.
• Page when UPS runtime is below the time required to start or stabilise backup generation.
• Warn when capacity has been reduced and headroom is below the scaling threshold.
• Annotate deployments, power transitions and failovers on the same timeline as service metrics.

Prometheus alert rules should include a useful summary and enough labels for routing. An alert saying “node down” is less actionable than “checkout capacity reduced in Johannesburg; remaining pods are at 82% CPU and eu-west failover is not enabled”.

Control ZAR cost and data exposure at the collection edge

Real-time analytics can become an expensive data-hoarding exercise. Remote-writing every debug log, retaining raw traces indefinitely and keeping one-second infrastructure metrics for years is rarely defensible, particularly when cloud spend is paid in ZAR against foreign-currency pricing.

Use tiers deliberately:

• Keep high-resolution metrics for the period needed to diagnose incidents and capacity changes.
• Downsample or retain recording rules for long-term trends.
• Sample traces intelligently, while retaining errors, slow requests and critical business transactions.
• Drop noisy logs before they cross a regional or cloud boundary.
• Measure telemetry volume by team, service and environment.

Grafana Alloy or another OpenTelemetry Collector deployment can perform filtering, batching and routing near the source. Send operational metrics to Mimir, logs to Loki and traces to Tempo according to retention and locality requirements.

POPIA is not a reason to abandon central observability, but it is a reason to classify telemetry. Request bodies, email addresses, access tokens and customer identifiers do not belong in ordinary logs. Redact at the application or collection edge, restrict access by tenant and team, and keep sensitive telemetry in an approved jurisdiction where required. South African public-sector policy also places specific emphasis on local infrastructure for certain sovereign or national-security data, so the storage decision should involve legal and security owners rather than being hidden in a collector configuration.

Turn dashboards into operating decisions

A reliable health dashboard answers three questions within seconds:

• Is a customer-impacting objective currently failing?
• Which region, service or dependency is responsible?
• What changed immediately before the failure?

Build one dashboard for the service owner and another for the platform team. The service view should focus on availability, latency, traffic, errors and dependency time. The platform view should expose saturation, scheduling failures, storage, network paths, telemetry pipeline health and capacity.

Include a visible “unknown” state. Missing telemetry is not the same as healthy telemetry. A stopped exporter, broken remote-write path or overloaded Loki distributor can make an estate look quiet precisely when visibility has failed.

Finally, test the analytics itself. During a controlled fail