Correlation starts with topology, not a language model

At 02:17, a Johannesburg checkout service began timing out. Prometheus fired alerts for latency, Kubernetes restarts, database connections and node pressure. Loki filled with gateway errors. Tempo showed traces stopping at the payment provider. By 02:24, the on-call…

Why dashboards alone miss the incident

At 02:13, a payment API began timing out for customers in Gauteng. CPU was normal, pod counts looked healthy and the error-rate alert had not fired. The useful clue was buried in a trace: requests from a Cape…

The ecosystem is a data path, not a dashboard catalogue

At 18:00 on a load-shedding evening, a payment platform’s dashboards showed green while customers in Gauteng saw intermittent failures. Prometheus was collecting host metrics, application logs were arriving in Loki, and traces existed in Tempo. The problem was…

Start with recovery decisions, not dashboards

At 02:17, a Kubernetes worker in Cape Town stopped responding during a scheduled load-shedding event. The alert was technically correct: node heartbeats had disappeared. The first remediation was not. An automation job restarted workloads, increased retries, and pushed…

Start with an observability cost budget that engineers can use

The invoice jumped after a Kubernetes rollout in Cape Town. Nothing looked broken: request rates were normal, dashboards loaded, and the alert queue was quiet. The cause was a new pod_uid label added to several Prometheus metrics. Every…

Stop Treating Every Red Signal as an Incident

At 18:00, a load-shedding event should not page the on-call engineer simply because a Johannesburg node has disappeared. It should trigger a controlled change in monitoring behaviour: suppress predictable infrastructure alerts, preserve customer-facing SLOs, and confirm that workloads…

The useful role of AI-Augmented Root Cause Analysis Systems

At 02:17, a payment API started returning intermittent 503s. The first alert pointed at Kubernetes CPU. The second blamed Redis latency. A dashboard showed a healthy database, while Loki contained a stream of connection-reset errors from a service…

Why a service list is not a dependency model

The payment API was healthy, the database dashboard was green, and the Kubernetes nodes had plenty of capacity. Yet customers in Gauteng were receiving intermittent checkout failures. The missing clue sat three hops away: a shipping-rate call routed…

Start with the failure boundaries, not the product list

At 02:13 during another round of load-shedding, a Kubernetes cluster in Johannesburg started buffering telemetry. Prometheus kept scraping, but its remote-write queue grew faster than the backup power could stabilise the node pool. By the time the site…

Why advanced correlation matters in a hybrid African estate

At 02:17, a payment API in Johannesburg started returning intermittent 502 responses. The first alert blamed the ingress controller. Five minutes later, a database connection pool alert fired. Then a burst of authentication failures appeared in the logs.…

Start with a failure model, not a dashboard

At 02:17, a payment API looked healthy in Johannesburg. CPU was normal, the Kubernetes pods were ready, and the synthetic check from the local office passed. Customers routed through a European dependency, however, were timing out. The useful…

Why hybrid failures rarely announce themselves

At 02:13, a Cape Town payment service did not fail cleanly. Requests to the on-premises database still succeeded, but the application tier in the cloud began retrying across a congested private link. CPU remained ordinary. Memory looked healthy.…