Distributed System Reliability Engineering: how we keep the lights on when the network, the cloud bill, and the pager all disagree

Distributed System Reliability Engineering: how we keep the lights on when the network, the cloud bill, and the pager all disagree

At 02:13, the checkout service was “up” in the Kubernetes sense and still useless to customers: p95 latency had tripled, error rates were oscillating, and half the traces ended at a payment gateway in eu-west while the other half never left Cape Town. That is the sort of mess Distributed System Reliability Engineering is actually about: making a distributed estate fail in smaller, more understandable ways, then recovering before the business notices.

The uncomfortable truth is that reliability work in distributed systems is no longer just about redundancy. In South African estates, it is also about load-shedding-aware alerting, last-mile instability, cross-region latency, POPIA constraints, and the very unglamorous reality of keeping cloud spend inside a ZAR-denominated budget. A graph that looks fine in isolation can still hide a service that is operationally brittle.

Distributed System Reliability Engineering starts with the failure modes you actually get

Too many teams still model reliability as a single uptime number. That is too blunt for a system that spans on-prem, a managed database, a queue, a public cloud cluster, and a SaaS dependency in another time zone. In practice, the failures that hurt most are usually one of these:

• Partial region loss, where one zone or node pool degrades but the service never fully dies.
• Latency amplification, where every retry adds pressure until the whole request path becomes unstable.
• Dependency skew, where a “small” shared service starts coupling unrelated workloads.
• Observability blind spots, where logs, traces, and metrics disagree because they are sampled, delayed, or missing labels.
• Human overreaction, where noisy paging causes engineers to suppress alerts that later mattered.

Grafana helps most when it is used as a correlation layer rather than a dashboard wallpaper. If the same incident view brings together Prometheus metrics, Loki logs, Tempo traces, and Mimir long-term metrics, the conversation shifts from “is it down?” to “which hop is failing, for whom, and since when?”

The South African wrinkle: reliability must include the power and the border crossing

In local environments, reliability engineering has to account for constraints that many overseas playbooks treat as edge cases. Load shedding still changes traffic patterns, capacity behaviour, and human response times. Connectivity between South African users and eu-west workloads can add enough latency that a chatty microservice design becomes self-inflicted damage. And regulated data often cannot simply be sprayed across regions without a governance review.

This is why Distributed System Reliability Engineering in South Africa should treat locality as a first-class design property. A service may be “multi-region” in marketing terms and still be a single point of failure if authentication, payment settlement, or the metadata store all live offshore. The question is not only “Can we survive a region outage?” but also “Can we stay useful when one region becomes slow, expensive, or administratively unavailable?”

That framing changes the observability stack too. Metrics are useful for rate and saturation, logs for causal detail, traces for path length and dependency mapping. But the real value appears when you use them to understand which geography, tenant, or product line is suffering. Without that, you end up paging on a continent-wide symptom while the real issue is one carrier path or one node pool in one region.

Distributed System Reliability Engineering with Grafana, Prometheus, Loki, Tempo and Mimir

A practical setup usually looks like this: Prometheus handles fast-moving service metrics, Loki stores logs with enough structure to search by request ID, Tempo captures traces so you can see where time is spent, and Mimir holds the longer retention window that lets teams compare “this outage” with “the last three almost-outages.” Grafana sits on top and turns that pile into something usable at 03:00.

The trick is to standardise the labels that matter before you standardise the dashboards. For distributed systems, those labels are usually:

• service
• region
• availability_zone
• tenant or customer segment
• dependency
• request_class

If those labels are missing or inconsistent, you will still have data, but not enough context to answer the questions that matter in an incident. In one hybrid estate, the most useful dashboard panel was not a fancy heatmap; it was a simple latency graph broken down by region and upstream provider. The pattern made it obvious that the Cape Town path was fine while the Johannesburg users were being hammered by a WAN issue outside the cluster.

Here is the sort of PromQL that is actually useful when a service goes sideways:

histogram_quantile(
  0.95,
  sum by (le, service, region) (
    rate(http_request_duration_seconds_bucket{service="checkout"}[5m])
  )
)

That query is not interesting because it is clever. It is interesting because it gives you a stable p95 by service and region, which is the minimum useful view for deciding whether to fail over, shed load, or simply wait for the dependency to recover.

For logs, Loki becomes far more valuable when the app emits structured fields that match the labels in metrics. A request ID, region tag, and tenant identifier can turn a hunt through millions of lines into a narrow, readable trail. Tempo then confirms whether the slowdown is in the app, the database, the message broker, or the external API. In a distributed estate, that sequence is often faster than jumping straight into infrastructure screens.

Alerting that understands load shedding, not just thresholds

Classic alerting fails when it mistakes a known operating condition for an incident. If your platform gets busier after the power returns, or if failover traffic routinely spikes during an outage window, a flat threshold will page you at exactly the wrong time.

Better alerts are based on symptoms, windows, and business context. A few rules work well:

1. Alert on user-visible impact first, not internal counters.
2. Use burn-rate alerts for SLOs so that short spikes do not create constant noise.
3. Suppress or route alerts differently during planned power events and maintenance windows.
4. Separate “needs human action now” from “needs attention before business opens.”

A useful pattern in a load-shedding-aware environment is to alert on error budget burn alongside dependency saturation. That gives operators a signal that the system is actually consuming reliability margin, rather than merely doing more work under stress.

For example, a Grafana alert built from Prometheus data can watch both 5xx rate and request latency over a rolling window, then route differently if the affected region is one that is already running on constrained power backup. That is not about making the alert complicated; it is about making it honest.

Failure testing beats certainty theatre

Distributed systems always surprise teams that only test happy paths. The antidote is deliberate failure injection: kill a pod, drop a queue consumer, add latency between services, or simulate a region-specific dependency outage. The goal is not drama. The goal is to see whether retries stampede, circuit breakers open correctly, and dashboards tell the truth.

A good reliability exercise in a South African hybrid estate usually includes at least one of the following:

• Disconnect one branch or POP and check whether users are gracefully routed elsewhere.
• Throttle calls to a non-critical external dependency and observe whether the primary flow still works.
• Force a database replica promotion and confirm the app does not assume the old writer survives.
• Drop tracing sampling temporarily and make sure the on-call team can still work from metrics and logs.

The point is not to prove invincibility. The point is to surface coupling. Most distributed incidents are just hidden coupling becoming visible at the worst possible moment.

Data sovereignty and cost pressure should shape the architecture, not follow it

Reliability in 2026 is not separate from governance or cost management. If sensitive data must stay within South Africa, then the architecture needs to enforce that boundary before the incident review reminds everyone it exists. Likewise, if the cloud bill is being paid in a weak currency, a design that sprays telemetry everywhere without retention discipline is a future outage of a different kind.

This is where Mimir and Loki retention policies matter. Retain enough to support incident forensics and seasonal comparisons, but not so much that you quietly turn observability into a storage tax. Compress, downsample where sensible, and keep high-cardinality labels only where they answer a real operational question.

One practical compromise is to keep high-resolution data near the hot path for a short period, then move aggregate metrics into longer retention. That gives teams evidence for incident reviews without burning budget on permanently expensive detail. In other words, treat observability storage as part of the reliability design, not an afterthought.