Why hybrid failures rarely announce themselves

At 02:17, the application was still returning 200 responses. At 02:24, however, the Johannesburg database replica had begun falling behind, the VPN tunnel to the cloud region was retransmitting packets, and a batch queue was quietly consuming its…

The incident was not a CPU problem

At 18:07, a payment platform’s Johannesburg API started timing out. CPU stayed below 40%, memory looked healthy, and the Kubernetes dashboard showed green workloads. The useful clue was elsewhere: latency from the Cape Town application tier to a…

Reliability starts with the customer journey, not the dashboard

At 02:17 on a Tuesday morning, a payment API in Johannesburg began returning HTTP 504 responses. CPU was normal. Kubernetes pods were healthy. Prometheus showed no obvious saturation. The actual failure was less dramatic: a carrier route to…

Start with decisions, not data types

At 02:13, a payment API began timing out in Johannesburg. The first alert blamed database latency. The second blamed packet loss. Both were technically correct, but neither showed that a scheduled backup had saturated the private link between…

Correlation starts with topology, not a language model

At 02:17, a Johannesburg checkout service began timing out. Prometheus fired alerts for latency, Kubernetes restarts, database connections and node pressure. Loki filled with gateway errors. Tempo showed traces stopping at the payment provider. By 02:24, the on-call…

Why dashboards alone miss the incident

At 02:13, a payment API began timing out for customers in Gauteng. CPU was normal, pod counts looked healthy and the error-rate alert had not fired. The useful clue was buried in a trace: requests from a Cape…

The ecosystem is a data path, not a dashboard catalogue

At 18:00 on a load-shedding evening, a payment platform’s dashboards showed green while customers in Gauteng saw intermittent failures. Prometheus was collecting host metrics, application logs were arriving in Loki, and traces existed in Tempo. The problem was…

Start with recovery decisions, not dashboards

At 02:17, a Kubernetes worker in Cape Town stopped responding during a scheduled load-shedding event. The alert was technically correct: node heartbeats had disappeared. The first remediation was not. An automation job restarted workloads, increased retries, and pushed…

Start with an observability cost budget that engineers can use

The invoice jumped after a Kubernetes rollout in Cape Town. Nothing looked broken: request rates were normal, dashboards loaded, and the alert queue was quiet. The cause was a new pod_uid label added to several Prometheus metrics. Every…

Stop Treating Every Red Signal as an Incident

At 18:00, a load-shedding event should not page the on-call engineer simply because a Johannesburg node has disappeared. It should trigger a controlled change in monitoring behaviour: suppress predictable infrastructure alerts, preserve customer-facing SLOs, and confirm that workloads…

The useful role of AI-Augmented Root Cause Analysis Systems

At 02:17, a payment API started returning intermittent 503s. The first alert pointed at Kubernetes CPU. The second blamed Redis latency. A dashboard showed a healthy database, while Loki contained a stream of connection-reset errors from a service…

Why a service list is not a dependency model

The payment API was healthy, the database dashboard was green, and the Kubernetes nodes had plenty of capacity. Yet customers in Gauteng were receiving intermittent checkout failures. The missing clue sat three hops away: a shipping-rate call routed…