At 18:07, the Johannesburg API was still returning a healthy 200 status. The incident started elsewhere: a brief utility interruption switched a rack to battery power, packet loss appeared between the data centre and a cloud database, and…
At 02:13, the checkout service was “up” in the Kubernetes sense and still useless to customers: p95 latency had tripled, error rates were oscillating, and half the traces ended at a payment gateway in eu-west while the other…
The billing email from the cloud provider landed five minutes before the incident report. R120k over budget on observability last month, and at the same time a major outage where alerts never fired because someone had quietly disabled…
At 02:13 on a Tuesday, the dashboard did not fail politely. It lit up with pod restarts, elevated p95 latency, queue backlogs, cache misses, and a fresh burst of TLS handshake errors from a single region. The on-call…
The alert fired at 18:23, ten minutes before Stage 4 load shedding in Johannesburg: “User checkout latency p95 > 3s in eu-west-1”. The odd part wasn’t the latency; everyone expects a spike when half the mobile network goes…
At 19:03 on a Tuesday, a national mobile wallet rollout in East Africa stalled. Dashboards were green, node health looked fine, and synthetic checks still passed. Yet users on older Android devices in rural Limpopo were hitting 12–15…
The third time a Cape Town edge node vanished during Stage 6 load-shedding, nobody even noticed until looking at the Grafana dashboard the next morning. The node had quietly failed, drained traffic, rescheduled pods to Johannesburg, and restored…
The Grafana dashboard looked fine. Error rates were down after the weekend deploy, latency was behaving, and the on-call rotation had survived another load-shedding cycle without a major incident. Then the email arrived from finance: “Can someone explain…
At 19:42 on a winter evening, a Johannesburg fintech’s core payments API hit a wall. Latency spiked, alerts fired, dashboards turned red – but every on-call engineer knew it wasn’t the app. Eskom had just dropped to Stage…
The load-shedding window was two hours. The incident lasted four. The root cause analysis deck said “network instability”, which was technically true and practically useless. That outage forced the team to rethink how they approached complex failure modes…
End-to-End Service Dependency Visibility Models are what separate a fast fix from a long, expensive blame-fest when a customer journey breaks somewhere between a mobile app, an API gateway, a payment processor, and a database replica in another…
Two hours into a Monday morning incident review, the pattern finally became obvious: every time the Cape Town Kubernetes cluster hit 40k active time-series, the entire metrics pipeline wobbled, alerts arrived late, and Grafana dashboards froze. The services…