Reliability starts with the dependency graph, not the dashboard

Distributed System Reliability Engineering: Designing for Failure Across African Infrastructure

The payment API looked healthy until customers in Johannesburg began receiving timeouts. CPU was below 40 percent, the Kubernetes nodes were ready, and the Grafana dashboard was mostly green. The failure was further downstream: a service in eu-west had increased its response time, retries amplified the traffic, and a small connectivity wobble on the South African side turned a slow dependency into a customer-facing outage.

That is the real problem addressed by Distributed System Reliability Engineering. Reliability is not the absence of red alerts. It is the ability to keep useful service available when networks partition, dependencies slow down, power interruptions affect infrastructure, and operators have incomplete information.

Reliability starts with the dependency graph, not the dashboard

A distributed service should be modelled as a chain of user journeys rather than a collection of Kubernetes workloads. A checkout request might pass through a load balancer, API gateway, authentication service, inventory database, payment provider and notification queue. Each component can report a healthy status while the journey fails.

Write the dependency graph down and attach a reliability expectation to each edge:

• What happens when the dependency is slow?
• What happens when it returns errors?
• Can the caller degrade gracefully?
• Does the caller retry, and who owns the retry budget?
• Where is data allowed to cross a border under POPIA and internal policy?

For a South African estate, the graph must also include realities that are easy to omit from cloud architecture diagrams: the link between an on-premises data centre and a public-cloud region, the latency to eu-west, local internet exchange paths, backup power, and last-mile connectivity for branches or customers.

Define service-level objectives around user-visible outcomes. “API pods are available” is a useful infrastructure signal, but “99.9 percent of successful checkout attempts complete within 800 ms” is closer to what the business needs. The latter also exposes a design mistake: an SLO spanning a local API and a remote payment provider may be impossible to meet without a timeout, fallback or asynchronous workflow.

Instrument one request across metrics, logs and traces

Distributed System Reliability Engineering depends on joining signals around a common request. Prometheus metrics show that latency changed; Loki logs can explain the returned error; Tempo traces show which hop consumed the time. Without a shared trace ID and consistent service metadata, engineers end up correlating timestamps by hand during an incident.

OpenTelemetry adoption has accelerated, with Prometheus and OpenTelemetry among the most widely invested-in open-source observability projects in the 2024 Grafana Labs survey.[1] The practical lesson is not to replace every existing collector overnight. Standardise instrumentation at service boundaries, then export the signals to systems that operators already know.

Use low-cardinality labels for metrics. Do not put customer IDs, full URLs or trace IDs into Prometheus labels. Keep those dimensions in logs and traces instead.

histogram_quantile(
  0.95,
  sum by (le, service, route) (
    rate(http_server_request_duration_seconds_bucket{
      environment="prod",
      service="checkout"
    }[5m])
  )
)

This query gives a route-level latency view without creating a time series for every individual request. A Grafana dashboard can link a slow route to Tempo traces and then to Loki logs using the trace ID. That workflow matters more than a dashboard containing dozens of unrelated panels.

For logs, structure events rather than relying on prose:

{
  "level": "error",
  "service": "checkout",
  "environment": "prod",
  "route": "/v1/orders",
  "dependency": "payment",
  "error_type": "timeout",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "duration_ms": 1820
}

Loki works particularly well when labels identify a bounded set of streams and the detailed fields remain in the log body. Excessive labels increase operational cost and make querying less predictable.

Make load-shedding and retries explicit

Retries are often introduced as a reliability feature and then become the outage multiplier. If an API receives 1,000 requests per second and retries each failed call twice, a struggling dependency can face several times the original load. Add exponential backoff with jitter, cap the number of attempts, and enforce an overall request deadline.

Prefer a small, explicit retry policy:

• Retry only transient failures such as connection resets, selected 5xx responses and rate limits.
• Never retry non-idempotent operations unless an idempotency key protects the operation.
• Stop retries when the caller’s deadline is nearly exhausted.
• Use circuit breakers or concurrency limits to prevent a failing dependency consuming every worker.
• Return a useful degraded response where the business process allows it.

Load-shedding must be tested as a normal operating mode. During a power event, a hybrid platform may lose capacity in one site while traffic shifts to another. During a regional provider incident, sending every request across the Atlantic may preserve availability but destroy latency and increase cloud spend in ZAR terms. A reliable system chooses which requests to serve, queue or reject before the platform makes that decision accidentally.

Alert on saturation and queue age, not only error rate. A queue that is growing while consumers report no errors is already a customer-impacting failure. Grafana alert rules can combine Prometheus metrics with Loki-derived error signals, but alerts should map to an action: scale consumers, disable a feature, pause a deployment or fail over a dependency.

Design observability for partitions and power interruptions

Load-shedding-aware monitoring needs more than a single “site down” alert. Track UPS runtime, generator state, power transfer events, node availability, network reachability and the age of the last successful scrape. A scrape failure can mean a dead host, a broken route or an overloaded Prometheus server. Those cases require different responses.

Use a separate monitoring path where possible. If the same network segment carries application traffic and monitoring traffic, a partition can make the system blind precisely when it needs diagnosis. Remote probes can test the customer journey from multiple locations, including a South African vantage point and the relevant overseas region.

Prometheus federation or remote write can support a resilient topology, while Mimir provides horizontally scalable, Prometheus-compatible long-term metrics storage for larger estates. Keep local alert evaluation close to the workload when a wide-area link is unreliable; do not make an urgent page depend on a cross-region query completing successfully.

Test these cases deliberately:

• Block traffic between the local cluster and eu-west.
• Add 500 ms latency and packet loss to a dependency.
• Stop a telemetry collector while applications continue serving traffic.
• Power down one site and verify that DNS, queues and clients behave as designed.
• Fill a disk or object-storage bucket used by Loki, Tempo or Mimir.

A system that survives an application failure but loses its telemetry pipeline has only partial resilience. Observability components need resource limits, retention policies, backups and failure-mode documentation of their own.

Control telemetry cost without destroying diagnostic value

Telemetry is production data. It consumes CPU, storage, network bandwidth and engineer attention. In a cost-sensitive environment, retaining every debug log forever is not a reliability strategy. It is deferred operational debt.

Set retention by investigative value:

• Keep high-resolution metrics for fast incident analysis and downsample or compact older data.
• Retain audit and security records according to legal and organisational requirements.
• Sample successful traces, but retain error traces and high-latency outliers.
• Use structured log levels and remove noisy health-check entries from long-term storage.
• Estimate cross-region ingestion and object-storage costs before enabling verbose telemetry.

POPIA considerations belong in the instrumentation design, not in a later dashboard review. Avoid putting identity numbers, payment details or unnecessary personal information into logs and traces. Apply redaction at the source or collector, restrict access by role, and document where telemetry is stored. Data sovereignty requirements may mean that application data remains in South Africa while only selected operational metadata is exported elsewhere.

Turn incidents into reliability changes

An incident review should produce fewer assumptions, not merely a longer timeline. For the timeout scenario, useful actions might include reducing the remote dependency deadline, adding a queue-based fallback, creating a probe from Johannesburg, and alerting on dependency-induced latency rather than only HTTP 500 responses.

Track reliability work alongside feature delivery:

1. Record the customer journey and exact impact.
2. Identify the first observable symptom and the earliest missed signal.
3. Separate trigger, contributing conditions and control failures.
4. Assign an owner and a measurable completion condition for every action.
5. Replay the failure with a game day or automated fault injection.

The strongest teams do not aim for a perfectly green dashboard. They build systems that degrade predictably, expose uncertainty quickly and preserve enough evidence to make the next decision. That is the practical centre of Distributed System Reliability Engineering: bounded failure, purposeful telemetry and architecture that still behaves sensibly when the network, power or dependency does not.