Why hybrid failures rarely announce themselves cleanly

Predictive Failure Detection for Hybrid Clouds: Catching the Fault Before the Pager

At 17:42 on a Tuesday, the warehouse API did not fail. Its error rate stayed below one percent, dashboards were green, and the on-call engineer was dealing with an unrelated certificate renewal. The first useful signal was a slow rise in database connection wait time on a Johannesburg virtual machine. Twenty minutes later, a load-shedding event switched the site to battery power, the WAN link began retransmitting packets, and the cloud-hosted order service started timing out against the on-premise database.

That is the operational gap Predictive Failure Detection for Hybrid Clouds must close: identifying a deteriorating system before it becomes a customer-visible outage. It is not a matter of adding “AI” to a dashboard. It is the disciplined combination of historical baselines, cross-signal correlation, dependency context and alerts that describe a trend rather than merely reporting a threshold breach.

Why hybrid failures rarely announce themselves cleanly

A single cloud workload can often be diagnosed from its own telemetry. A hybrid estate is less cooperative. The application may run in AWS, its database may remain in a Cape Town data centre, identity may depend on Microsoft Entra ID, and users may reach the service through a variable last-mile connection. Each component can appear healthy while the path between them degrades.

Common precursors include:

• Increasing TCP retransmissions between a cloud region and an on-premise subnet.
• Gradually rising query latency while CPU and memory remain normal.
• Short power interruptions that reset network equipment without taking servers fully offline.
• Disk latency increasing on a virtualisation cluster before capacity is exhausted.
• Queue depth growing during a period when application traffic is flat.
• Cloud egress costs increasing because a retry loop is moving data across regions.

Traditional alerting treats these as separate events. Predictive detection treats them as evidence in one developing failure. A rising p95 latency, an increase in retry logs and a trace showing longer database spans are more significant together than any one of those signals alone.

The 2025 observability landscape reflects this direction: Prometheus remains widely used in production, while OpenTelemetry adoption continues to grow and teams increasingly combine metrics, logs and traces rather than managing each signal in isolation. The practical implication is straightforward: instrument the path, not just the process.

Build the evidence layer before choosing an algorithm

Start with consistent labels. Every metric, log stream and trace should carry enough context to answer four questions: which service, which environment, which location and which dependency?

service="orders-api"
environment="production"
region="za-jhb"
deployment="blue"
dependency="warehouse-db"

Prometheus can scrape Kubernetes workloads, node exporters and legacy virtual machines. Remote-write metrics to Mimir when retention, tenancy or cross-site querying exceeds what individual Prometheus servers can comfortably provide. Mimir is particularly useful when Johannesburg, Cape Town and cloud workloads need to be queried as one estate without pretending they share the same failure domain.

Use Loki for application and infrastructure logs, but keep labels restrained. A label such as service is useful; a request ID as a label can create an expensive stream explosion. Put high-cardinality values in the log body or structured metadata instead.

{service="orders-api", environment="production"}
|= "timeout"
| json
| line_format "{{.timestamp}} {{.dependency}} retry={{.retry_count}} error={{.error}}"

Tempo supplies the missing dependency detail. A trace can show that user-facing latency is caused not by the API process, but by a 900-millisecond call across an unstable link to a database in another site. Correlating a trace ID from the application log with a Tempo trace is often more valuable than adding another CPU panel.

Grafana provides the investigation layer: operators can move from a forecasted metric to the associated logs and traces without manually searching three systems. The dashboard should expose the relationship between signals, not simply display a wall of green and red gauges.

Turn baselines into failure forecasts

Forecasting is most reliable when applied to a service-level indicator or resource trend with a known operating rhythm. A database filesystem may have a predictable growth curve. Internet latency may vary by hour. A binary “above 80 percent” alert is inadequate if the disk will reach 95 percent during the next backup window.

PromQL’s forecasting functions provide a useful first step:

predict_linear(
  node_filesystem_avail_bytes{
    job="node-exporter",
    mountpoint="/var/lib/postgresql"
  }[6h],
  24 * 3600
) < 0

This asks whether the recent trend predicts exhaustion within 24 hours. It is not machine learning, and that is a strength: the result is explainable, testable and easy to tune. Exclude planned maintenance, sudden restores and known batch jobs from the training window, otherwise the forecast learns the wrong lesson.

For latency, compare the current rolling percentile with a seasonal baseline. A simple approach is to record hourly p95 latency and alert when the current value is materially above the same period on recent comparable days. More advanced teams can use anomaly detection outside Prometheus, but the alert still needs a clear operational interpretation: “cross-site database latency is 2.4 times its expected level” is actionable; “model score 0.87” is not.

Use multiple conditions to reduce false positives:

(
  histogram_quantile(
    0.95,
    sum by (le) (
      rate(http_request_duration_seconds_bucket{
        service="orders-api",
        route="/orders"
      }[10m])
    )
  ) > 0.8
)
and
(
  sum(rate(http_requests_total{
    service="orders-api",
    status=~"5.."
  }[10m]))
/
sum(rate(http_requests_total{
  service="orders-api"
}[10m])) > 0.02
)

Latency alone may reflect a slow but successful dependency. Latency plus rising server errors is stronger evidence of imminent customer impact. Add dependency health, queue growth or retry volume where those signals are available.

Make load-shedding and connectivity part of the model

South African estates need operational context that a generic cloud alert rarely includes. A site running on UPS power may remain reachable while cooling, storage or network redundancy has already been reduced. During load-shedding, a short-lived packet-loss increase should not automatically page the database team, but it should change the interpretation of other signals.

Record power state as a metric or event. For example, an exporter can expose facility_power_source{site="cpt-1",source="ups"} 1. Alert routing can then distinguish between a predicted UPS transition and an unexpected power change. The purpose is not to suppress alerts blindly; it is to avoid treating a known operating mode as a mysterious application fault.

Likewise, measure the network path from the application’s point of execution, not from an engineer’s laptop. Track round-trip time, packet loss, TCP retransmissions and DNS resolution time between cloud workloads and on-premise services. A healthy probe from Johannesburg does not prove that a pod in eu-west-1 can reach the same endpoint reliably.

Cross-region latency also deserves a cost dimension. Retries and failover traffic can push ZAR-denominated cloud spend higher before availability visibly declines. Include egress, request counts and retry volume in the same service view. The cheapest path is not always the most resilient, but an unexplained cost slope is often an early indication that the system is fighting the network.

Use traces and logs to prove the forecast

A predictive alert should lead to a short investigation, not a research project. Suppose the forecast says the orders API will breach its latency objective within 30 minutes. The responder should be able to answer:

1. Is the degradation isolated to one region, deployment or tenant?
2. Which dependency contributes most to the trace duration?
3. Are retries increasing, and are they caused by timeouts or connection resets?
4. Did the change begin after a deployment, route change, power event or backup?
5. Will remediation create a POPIA or data-residency problem?

Tempo traces can expose the slow span; Loki can show the corresponding timeout and infrastructure events; Mimir can establish whether the pattern is new or recurring. This correlation is also where sampling decisions matter. Retain enough traces for failed and slow