At 18:07 on a winter evening in Johannesburg, the production dashboard looked perfectly calm: CPU under 40%, latency under 120 ms, error rates flat. At 18:09, Eskom flipped the Stage 4 switch, a regional edge node dropped off…
The post-incident review was oddly quiet. No shouting, no finger-pointing. Just an uncomfortable graph on the wall: request latency spiking every evening between 18:00 and 22:00, right when users in Johannesburg and Durban were trying to log in.…
The night Eskom dropped Stage 6 without much warning, the Johannesburg Kubernetes cluster did exactly what the business had asked for: auto-scaled to handle traffic spikes, pushed traces, logs, and metrics to three different stacks, and dutifully flooded…
The last time the Joburg Kubernetes cluster fell over during Stage 6 load shedding, it was not the generators that almost brought the team down. It was the sheer noise of alerts. Pages for “high latency”, “error rate…
The incident review started with a painful line in the post-mortem doc: “We had all the data, but no one saw the pattern.” A payment gateway spanning Johannesburg, Cape Town, Lagos and eu-west-1 had been flapping for 47…
The busiest traffic window on a large South African retail platform turned out not to be payday Friday, but 18:00–20:00 during Stage 4 load-shedding. Generators kick in, LTE towers wobble, users retry card payments three times, and suddenly…
The Redis primary went dark three minutes into stage‑4 load shedding. The Cape Town edge node lost its rack, failover to Johannesburg fired, and the latency SLO dashboard started bleeding red. What didn’t happen: no one got out…
The outage wasn’t the shock; the invoice was. After a winter week of aggressive load-shedding, a fintech team scaled up logging and tracing “just for safety” and turned on debug logs across three Kubernetes clusters. The incident retrospective…
The alert hit just after 18:00, right as Stage 4 load shedding kicked in across Gauteng. RDS connections from a Rustenburg branch dropped, Kafka lag spiked from Cape Town, and latency to eu-west-1 doubled as everyone’s failover plan…
The API error rate spiked just after 18:00, right as Cape Town rolled into Stage 4 load shedding. Latency to eu-west-1 jumped, Kafka consumer lag started climbing, and the on-call WhatsApp group lit up. Ten minutes later, a…
As a South African SRE working with distributed systems across Johannesburg, Cape Town, and cloud regions in Europe, one challenge keeps surfacing: we don’t just need to know that something is broken—we need to know where and why…
As a South African SRE working with large, distributed systems and Grafana at the centre of our observability stack, I’ve learned that the difference between a calm incident response and a 3 a.m. disaster often comes down to…