Why hybrid failures rarely announce themselves

Predictive Failure Detection for Hybrid Clouds: Catching the Failure Before the Pager

At 02:17, the application was still returning 200 responses. At 02:24, however, the Johannesburg database replica had begun falling behind, the VPN tunnel to the cloud region was retransmitting packets, and a batch queue was quietly consuming its retry budget. Nothing had crossed a conventional outage threshold. By 02:41, the morning billing run was failing.

That is the awkward gap where Predictive Failure Detection for Hybrid Clouds earns its keep: not by guessing that “something looks odd”, but by combining weak signals across on-premises infrastructure, cloud services, networks and applications before they become an incident.

Why hybrid failures rarely announce themselves

In a single cloud environment, an engineer can often correlate a node, a load balancer and an application service in one control plane. Hybrid estates are less cooperative. A South African retailer might run its transactional database in a Johannesburg data centre, Kubernetes workloads in a local cloud region, backups in Europe, and observability components across both environments.

The first warning may be an increase in latency to eu-west, not a failed service. A generator test may cause a short power interruption at the colocation site. A firewall policy change may affect only return traffic over an IPsec tunnel. Last-mile congestion may make a healthy API appear unreliable to customers in smaller towns while internal probes remain green.

Traditional alerts tend to evaluate isolated thresholds:

• CPU above 85% for five minutes.
• Packet loss above 2%.
• Replication lag above 30 seconds.
• HTTP errors above 5%.

Those rules are useful, but they describe the present. Prediction requires looking at direction, rate of change, seasonality and relationships between signals. A replica whose lag rises from 2 to 18 seconds during every month-end run may be more dangerous than one sitting at 25 seconds but holding steady.

A practical signal design for South African hybrid estates

Start with failure modes rather than tools. List what can break, which signals precede it, and what action is possible. For a mixed on-premises and cloud platform, the list might include:

• Battery or generator instability during planned load-shedding or maintenance.
• WAN saturation and packet loss between an office, data centre and cloud region.
• Database replication lag that threatens the recovery point objective.
• Cloud spend spikes caused by runaway autoscaling or high-cardinality telemetry.
• Certificate expiry, storage exhaustion and queue backlogs.
• Cross-region latency that pushes synchronous calls beyond their timeout.
• Observability gaps caused by a disconnected site or a failed collector.

Prometheus should collect infrastructure and application metrics close to their source. Exporters cover node health, databases, network devices, UPS systems and cloud APIs. Remote-write selected series to Mimir when long retention and cross-environment queries are required. Keep the local Prometheus instance capable of alerting if the WAN is unavailable; a monitoring system that depends on the link it is meant to diagnose is a poor safety net.

Use Loki for logs, but avoid treating raw log volume as a predictive signal. Extract structured fields such as retry count, error class, peer region and circuit state. Use Tempo to follow a request across the on-premises boundary, the cloud service and the database. A trace that shows a steadily expanding network span is often more actionable than a dashboard showing average application latency.

Grafana provides the operational view where these signals meet. A useful dashboard should show the service-level symptom alongside its likely dependencies: request latency, tunnel packet loss, database lag, queue age, power status and recent deployments. The aim is not to display every metric, but to shorten the path from “something is drifting” to “this is the next likely failure”.

From static thresholds to leading indicators

PromQL is enough for many early-warning rules. The following example detects replication lag that is both high and accelerating, while avoiding an alert for a stable but elevated value:

(
  max by (cluster) (mysql_replication_lag_seconds)
    > 20
)
and
(
  deriv(max by (cluster) (mysql_replication_lag_seconds)[15m]) > 0.02
)

The derivative is not a prediction model in the machine-learning sense. It is a transparent indicator that operators can explain during an incident review. That matters. A black-box score that pages the team without exposing its contributing signals will be ignored after the first false positive.

For capacity, compare current usage with a forecast rather than a fixed limit:

predict_linear(
  node_filesystem_avail_bytes{mountpoint="/var/lib"}[6h],
  24 * 3600
) < 0

This asks whether the observed disk-consumption trend reaches zero within the next 24 hours. It should be paired with a minimum current-usage condition and excluded for filesystems with volatile or temporary data. Forecasts are only as sensible as the window selected; a six-hour trend during a nightly batch is not a reliable month-end prediction.

Seasonality deserves equal attention. South African platforms often have predictable peaks around salary days, month-end billing, school holidays and major retail campaigns. Record those periods as annotations and compare like with like. A model trained on ordinary weekdays will misclassify a legitimate month-end surge as an impending outage.

Correlating logs, traces and metrics without drowning in data

Predictive detection works best when telemetry shares a common vocabulary. Attach environment, region, service, cluster and tenant labels consistently. Keep cardinality under control: customer IDs and request IDs belong in logs and traces, not in every Prometheus time series.

In Loki, search for an increase in retryable failures around a particular dependency:

{service="payments", environment="prod"}
| json
| dependency="fraud-api"
| level="error"
| unwrap retry_count
| sum_over_time([10m])

That result becomes more useful when linked to a Tempo trace showing whether retries are caused by DNS, network delay, a remote 5xx response or local saturation. The operational question is not “how many errors occurred?” It is “which dependency is consuming the remaining failure budget, and is the trend getting worse?”

Mimir is valuable when the estate needs a durable, horizontally scalable metrics store across clusters and regions. It also enables longer baselines for forecasting. Be deliberate about retention, recording rules and remote-write costs, particularly when ZAR cloud budgets are under pressure. Sending every short-lived container label to a central backend can cost more than the service it monitors.

Load-shedding-aware alerting and data sovereignty

Power events require a different alert strategy. Do not page on every exporter timeout when a site is known to be entering a maintenance window. Instead, ingest the schedule as an event and change alert severity while preserving visibility. A UPS on battery, rising temperature and a loss of redundant network paths should still produce a high-priority alert; an isolated scrape failure during an expected shutdown should not wake the on-call engineer.

Connectivity deserves similar treatment. A site that cannot reach the central metrics backend must not be labelled “healthy” simply because no new data arrived. Monitor telemetry freshness locally and alert on the age of the last successful remote write. Store enough local data to investigate the outage after connectivity returns.

POPIA and the National Policy on Data and Cloud also affect architecture. The 2024 policy states that processing data collected within South Africa must comply with local data-protection and security laws, while government data relating to national security and sovereignty must be stored on infrastructure inside the country.[4] Observability pipelines should therefore classify telemetry before exporting it. Metrics are usually easier to move than logs containing account numbers, email addresses, payload fragments or trace baggage.

Redact at collection time, use tenant-aware access controls, and keep sensitive logs in a South African Loki deployment where required. Predictive detection should not become an accidental cross-border data-exfiltration project.

Make the prediction operational, not impressive

Recent observability practice is moving towards combined Prometheus and OpenTelemetry deployments, with cost remaining a major tool-selection concern in 2025.[3][5] That trend is useful only if the operating model changes with it. More telemetry does not automatically produce earlier warnings.

Every predictive alert needs four fields:

• Evidence: the signals that are drifting and their current values.
• Horizon: the estimated time before a defined limit is reached.
• Confidence: how much history supports the forecast.
• Action: the runbook step that can reduce the risk.

For example: “Database replica lag is increasing at 1.8 seconds per minute and is forecast to exceed the 60-second failover limit in 23 minutes; the month-end batch is active; throttle workers and verify WAN loss.” That is far more useful than “Replica unhealthy”.

Review predictions after every incident. Track false positives separately from missed failures. Retire rules that cannot trigger a meaningful action. Keep humans in the loop for destructive remediation such as failover, scaling databases or deleting data.

The strongest implementation is not the one with the cleverest algorithm. It is the one that notices a degrading tunnel, a slowing replica and a growing queue as one operational story—then gives the team enough time to change the ending.