At 18:42, a load-shedding alert fired across three Kubernetes clusters. The alert was technically correct: node CPU had crossed its threshold. It was operationally useless. Half the affected workloads were already unreachable because the local site had lost…
At 02:17, a payment API in Johannesburg started timing out. Prometheus showed rising latency, Loki showed a burst of connection errors, and Tempo showed traces stretching across a dependency in eu-west. The first alert named the symptom correctly,…
At 02:17, the checkout error rate rose in Johannesburg while every Kubernetes health check remained green. The application team saw a healthy API, the database team saw normal query latency, and the network team saw no packet loss.…
At 02:17, a payment service started timing out across a Johannesburg Kubernetes cluster. The application was healthy enough to serve requests, but its Prometheus pods were restarting under memory pressure. A dashboard query had introduced customer_id into a…
At 18:07, during another evening load-shedding cycle, the checkout team saw a familiar pattern: API latency climbing, payment retries increasing, and a flood of “connection reset” messages from a Johannesburg service. The first dashboard suggested a database problem.…
At 02:17, a payment API started timing out for customers in Gauteng. The application dashboard showed healthy pods in AWS Cape Town. The database dashboard showed normal CPU. The incident channel blamed the carrier, then the DNS provider,…
At 17:42 on a Tuesday, the warehouse API did not fail. Its error rate stayed below one percent, dashboards were green, and the on-call engineer was dealing with an unrelated certificate renewal. The first useful signal was a…
At 18:07, the Johannesburg API was still returning a healthy 200 status. The incident started elsewhere: a brief utility interruption switched a rack to battery power, packet loss appeared between the data centre and a cloud database, and…
At 02:13, the checkout service was “up” in the Kubernetes sense and still useless to customers: p95 latency had tripled, error rates were oscillating, and half the traces ended at a payment gateway in eu-west while the other…
The billing email from the cloud provider landed five minutes before the incident report. R120k over budget on observability last month, and at the same time a major outage where alerts never fired because someone had quietly disabled…
At 02:13 on a Tuesday, the dashboard did not fail politely. It lit up with pod restarts, elevated p95 latency, queue backlogs, cache misses, and a fresh burst of TLS handshake errors from a single region. The on-call…
The alert fired at 18:23, ten minutes before Stage 4 load shedding in Johannesburg: “User checkout latency p95 > 3s in eu-west-1”. The odd part wasn’t the latency; everyone expects a spike when half the mobile network goes…