The last time the Joburg Kubernetes cluster fell over during Stage 6 load shedding, it was not the generators that almost brought the team down. It was the sheer noise of alerts. Pages for “high latency”, “error rate…
The incident review started with a painful line in the post-mortem doc: “We had all the data, but no one saw the pattern.” A payment gateway spanning Johannesburg, Cape Town, Lagos and eu-west-1 had been flapping for 47…
The busiest traffic window on a large South African retail platform turned out not to be payday Friday, but 18:00–20:00 during Stage 4 load-shedding. Generators kick in, LTE towers wobble, users retry card payments three times, and suddenly…
The Redis primary went dark three minutes into stage‑4 load shedding. The Cape Town edge node lost its rack, failover to Johannesburg fired, and the latency SLO dashboard started bleeding red. What didn’t happen: no one got out…
The outage wasn’t the shock; the invoice was. After a winter week of aggressive load-shedding, a fintech team scaled up logging and tracing “just for safety” and turned on debug logs across three Kubernetes clusters. The incident retrospective…
The alert hit just after 18:00, right as Stage 4 load shedding kicked in across Gauteng. RDS connections from a Rustenburg branch dropped, Kafka lag spiked from Cape Town, and latency to eu-west-1 doubled as everyone’s failover plan…
The API error rate spiked just after 18:00, right as Cape Town rolled into Stage 4 load shedding. Latency to eu-west-1 jumped, Kafka consumer lag started climbing, and the on-call WhatsApp group lit up. Ten minutes later, a…
As a South African SRE working with distributed systems across Johannesburg, Cape Town, and cloud regions in Europe, one challenge keeps surfacing: we don’t just need to know that something is broken—we need to know where and why…
As a South African SRE working with large, distributed systems and Grafana at the centre of our observability stack, I’ve learned that the difference between a calm incident response and a 3 a.m. disaster often comes down to…
As a South African SRE working with distributed systems across multiple regions and cloud providers, I’ve learned that traditional log aggregation is no longer enough. We need Advanced Log Intelligence and Correlation Systems to move from “scrolling through…
As a South African SRE working with teams spread across Johannesburg, Cape Town, and a couple of data centres in Europe, I’ve seen first-hand how fast organisations in our region are adopting multi-cloud. AWS for core services, Azure…
As a South African SRE working with teams spread across Johannesburg, Cape Town, and London, I’ve learned the hard way that hybrid cloud outages don’t care about your latency constraints, data residency laws, or change-freeze windows. Predictive Failure…