Stop Treating “Healthy” as a Single Number

Real-Time Infrastructure Health Analytics: From Noisy Dashboards to Faster Incident Decisions

At 02:17, a payment platform’s Johannesburg API did not fail cleanly. CPU stayed below 50%, pods remained healthy, and the load balancer returned mostly successful health checks. Yet customers experienced timeouts because a saturated inter-region link was delaying calls to a service in eu-west. The first useful clue was not a red dashboard panel. It was the combination of rising request latency, an increase in retry logs, and traces showing that the remote dependency consumed most of the request budget.

That is the practical purpose of Real-Time Infrastructure Health Analytics: joining signals quickly enough to explain what users are experiencing, rather than merely reporting whether individual components are “up”. Prometheus, Loki, Tempo, Mimir and Grafana can provide that view, but only when telemetry is designed around decisions, failure modes and cost.

Stop Treating “Healthy” as a Single Number

Infrastructure health is multidimensional. A node can be available while its disk queue is growing. A Kubernetes deployment can have the desired replica count while requests wait on a database connection pool. A cloud region can be operational while packet loss makes it unusable for customers on a specific last-mile provider.

A useful health model starts with four questions:

• Can customers complete the critical operation?
• Is the service meeting its latency and error objectives?
• Which dependency is consuming the failure budget?
• Is the platform approaching a capacity, connectivity or cost limit?

Prometheus is well suited to the first two questions. Exporters and application instrumentation expose time series such as request rate, error rate, saturation and latency. Mimir provides durable, horizontally scalable storage when a single Prometheus server is not enough, particularly across multiple clusters or regions.

The mistake is to fill dashboards with every available metric. High-cardinality labels such as request IDs, full URLs and user identifiers make queries expensive and can overwhelm the metrics backend. Keep labels bounded: service, route template, status class, region and environment are usually more useful than raw request paths.

Build the Signal Path Around a Customer Transaction

Begin with one important transaction, such as card authorisation, parcel tracking or a mobile airtime purchase. Define its service-level indicators before adding panels:

• Availability: the proportion of requests that complete successfully.
• Latency: a percentile such as p95 or p99, measured at the user-facing boundary.
• Freshness: how recently a required data source was updated.
• Dependency success: the outcome of calls to databases, queues and external providers.

A burn-rate alert is more meaningful than a generic “CPU above 80%” alert because it relates symptoms to an objective. A basic request error ratio might look like this:

sum(rate(http_requests_total{
  service="payments-api",
  status=~"5.."
}[5m]))
/
sum(rate(http_requests_total{
  service="payments-api"
}[5m]))

Pair the ratio with a longer window to distinguish a short-lived spike from sustained customer impact. The alert should include the service, route, region, current ratio and a link to the relevant dashboard or runbook. Avoid sending an engineer a notification that says only “high error rate”.

For load-shedding-aware operations, add power and locality as dimensions where they affect the service. Track UPS state, generator runtime, facility temperature and the health of the local network path, but do not page on every expected transition. A planned power event should change routing and capacity policy; an unplanned loss of redundancy should page the owner.

Use Metrics, Logs and Traces as One Investigation

Metrics tell the team that a problem is occurring. Logs often explain the rejected operation. Traces show where time was spent. The value appears when these signals share correlation identifiers and consistent service metadata.

Loki can retain structured application and infrastructure logs without forcing every field into a high-cardinality index. Keep indexed labels small and put searchable context in the log body. A structured event should contain fields such as trace ID, span ID, service, region, dependency, outcome and a safe error classification.

Tempo completes the path from an alert to a specific request. Instrument the API gateway, asynchronous workers and important database or HTTP clients with OpenTelemetry. When a p99 latency panel turns red, a trace should reveal whether the delay came from DNS, a queue, a database lock, a cross-region request or application code.

In Real-Time Infrastructure Health Analytics, correlation matters more than having three disconnected products. Configure exemplars so a Prometheus latency series can open representative traces. Add trace IDs to logs. Then an engineer can move from an aggregate symptom to one failing transaction without manually guessing timestamps.

Grafana provides the investigation surface: dashboards for operational trends, Explore views for ad hoc PromQL and LogQL, and trace navigation for request-level analysis. The dashboard should answer “what changed?” before it presents a wall of infrastructure graphs.

Design for South African Network and Power Conditions

Latency to eu-west is not a theoretical concern for African platforms. It can vary by provider, route and time of day, and a healthy cloud service may still be slow from a particular customer network. Measure synthetic transactions from more than one South African location, where possible, and separate application latency from network latency.

Useful dimensions include:

• Probe location, such as Johannesburg, Cape Town or a regional branch.
• ISP or network segment, where collecting it is lawful and operationally justified.
• Destination region and availability zone.
• IPv4 or IPv6 path.
• DNS, TLS, connect and server-response timings.

Hybrid estates need the same discipline. An on-premises database in Johannesburg and a Kubernetes workload in a public cloud may look healthy independently while the link between them is degrading. Black-box probes and synthetic transactions expose this class of failure earlier than host dashboards do.

Load shedding requires state-aware alerting. If a site is running on generator power, a low battery or fuel estimate can be more urgent than a brief rise in CPU. Conversely, repeated pages during a scheduled event create fatigue. Suppress alerts that are expected consequences of a controlled shutdown, but continue paging on customer-facing SLO violations and loss of redundancy.

Make Cost and Compliance Part of Health

Observability is itself a production workload. ZAR cloud-cost pressure makes uncontrolled log volume and long retention difficult to justify. Measure telemetry ingestion by team and service, identify the most expensive labels, and sample noisy debug events after incidents. Metrics need long-term trends; traces often need intelligent sampling; logs need a clear retention purpose.

Recent observability practice has moved towards combining Prometheus with OpenTelemetry rather than treating them as competing choices. Grafana Labs’ 2024 survey reported that 89% of respondents were investing in Prometheus and 85% in OpenTelemetry, while its 2025 findings described continued growth in both technologies.[1][2] That direction is useful for hybrid platforms: instrument once with OpenTelemetry, then route metrics, logs and traces to the backends that match retention, query and sovereignty requirements.

POPIA considerations belong in the telemetry design, not in a later audit. Do not place identity numbers, payment details, access tokens or unnecessary customer data in labels, log messages or trace attributes. Mask payloads at the source. Define which telemetry can leave South Africa, who can query it and how long it must be retained. A dashboard that exposes sensitive fields is not a healthy operational system.

Turn the Dashboard into an Operating Decision

A practical landing page should show service-level status first, followed by the strongest evidence for the current condition:

• Request rate, error ratio and p95 or p99 latency for critical services.
• Active incidents and SLO burn rate.
• Dependency health, including database, queue and external-provider outcomes.
• Regional synthetic latency and packet-loss trends.
• Capacity risks such as memory pressure, disk growth and connection saturation.
• Telemetry pipeline health, including scrape gaps and ingestion delay.

Use recording rules for expensive, frequently viewed calculations. Keep alert expressions stable and test them against historical incidents. A good alert should identify the affected service, likely customer impact and the next investigation path. If the responder still has to search three dashboards to discover whether the problem is regional, the analytics design is unfinished.

The strongest implementation is not the one with the most charts. It is the one that reduces the distance between “customers are timing out” and “the Johannesburg-to-eu-west dependency is breaching its latency budget; route reads locally and investigate the saturated link”. That is the standard Real-Time Infrastructure Health Analytics should meet: timely evidence, joined signals, controlled cost and a decision an engineer can act on.