Reliability starts with the customer journey, not the dashboard

Distributed System Reliability Engineering: Designing for Failure Across African Networks

At 02:17 on a Tuesday morning, a payment API in Johannesburg began returning HTTP 504 responses. CPU was normal. Kubernetes pods were healthy. Prometheus showed no obvious saturation. The actual failure was less dramatic: a carrier route to eu-west developed packet loss, retries multiplied the traffic, and a synchronous fraud check turned a regional network problem into a customer-facing outage.

That is the uncomfortable centre of Distributed System Reliability Engineering: a service can be “up” while the system is unusable. Reliability depends on boundaries between services, regions, networks, queues, identity providers and humans—not simply on whether a process has restarted successfully.

Reliability starts with the customer journey, not the dashboard

The first design decision is to define what must remain usable. For a South African retail platform, that might mean browsing products and creating an order must continue during a degraded connection to an overseas payment or fraud provider. For a mobile service, it may mean an authenticated user can still view a cached balance when the last-mile network is unstable.

Translate those journeys into service-level objectives (SLOs). A useful SLO is specific about both time and outcome:

• 99.9% of checkout requests complete successfully over a rolling 30-day window.
• 99% of catalogue reads return within 400 milliseconds at the Johannesburg edge.
• Payment-provider failures do not block order creation for more than 30 seconds.
• Telemetry remains queryable during an ingestion or regional outage.

Availability alone hides too much. A successful response containing stale data may be acceptable for a catalogue but not for a debit. A 200 response after a 20-second retry chain is technically successful and practically broken. Measure the user-visible result, latency and freshness separately.

Prometheus remains a strong foundation for these measurements. The wider observability ecosystem is also moving towards combined Prometheus and OpenTelemetry deployments; a 2025 Grafana Labs survey reported that 71% of respondents used both in some capacity.[1] That combination is useful when metrics describe system health and traces explain where a request spent its time.

Make remote dependencies boring before making them resilient

Cross-region calls deserve an explicit reliability budget. A request from Johannesburg to eu-west is not merely “another HTTP call”; it has variable round-trip time, possible submarine-cable or transit disruption, and a failure mode that may last longer than the caller’s timeout.

For each dependency, document:

• Normal and worst-case latency.
• Timeouts at every hop.
• Retry limits and backoff behaviour.
• Whether the operation is idempotent.
• Fallback data and acceptable staleness.
• The circuit-breaker opening condition.
• Whether failure should reject, queue or partially complete the user action.

Retries are not resilience when every replica retries at the same time. Use bounded exponential backoff with jitter, and never retry a non-idempotent operation unless it carries an idempotency key. A payment request should be safely replayable by reference to a unique transaction identifier—not duplicated because a client did not receive the first response.

Queues can separate the user journey from a fragile dependency. An order service can accept an order, persist its state, and publish a payment task. The user then receives a clear “processing” state rather than waiting on a remote provider. This shifts complexity into reconciliation, but reconciliation is easier to operate than an opaque timeout storm.

Distributed tracing in Tempo makes these trade-offs visible. A trace should show the ingress request, queue publish, consumer processing and remote call as one connected story. Trace IDs in application logs allow Loki searches to move from a slow span to the exact retry, error body or circuit-breaker transition.

Use the LGTM stack to join symptoms to causes

A useful Grafana dashboard does not contain every metric available. It answers a question in the order an incident responder needs to ask it: Are customers failing? Which journey? Which region? Which dependency? What changed?

For a checkout service, a burn-rate panel can expose fast SLO damage before an average latency graph becomes alarming:

sum(rate(http_requests_total{
  service="checkout",
  route="/checkout",
  status=~"5.."
}[5m]))
/
sum(rate(http_requests_total{
  service="checkout",
  route="/checkout"
}[5m])) > 0.02

Pair that alert with request volume. A percentage without traffic can create noise during quiet periods, while a request count without a denominator cannot express customer impact. Alert on symptoms first, then add dependency-specific alerts for diagnosis.

• Prometheus: scrape service, Kubernetes and infrastructure metrics close to where they are produced.
• Mimir: centralise long-term, Prometheus-compatible metrics across clusters and regions.
• Loki: keep structured application logs searchable without indexing every high-cardinality field.
• Tempo: retain distributed traces that reveal queue delay, remote latency and fan-out behaviour.
• Grafana: correlate all four signals in one investigation workflow, with links from alerts to dashboards, logs and traces.

Metric labels need discipline. Do not put customer IDs, full URLs or arbitrary exception messages into labels. Those values create uncontrolled cardinality and can make the monitoring system itself expensive or unreliable. Keep them in logs or trace attributes, where retention and sampling can be managed deliberately.

Loki needs the same restraint. Emit JSON with fields such as trace_id, service, region, dependency and error_type. Index only stable, frequently queried labels such as service and environment. Use LogQL filters for the rest:

{service="checkout", environment="prod"}
| json
| dependency="fraud-provider"
| error_type="timeout"

Design alerts for load-shedding days and broken last mile

Power interruptions change the shape of an incident. A site may lose local connectivity, restart networking equipment, or switch to backup power while cloud services remain healthy. Alerting that treats every scrape failure as a platform outage will page the team repeatedly and obscure the real condition.

Separate these signals:

• Exporter or scrape failure in one facility.
• Application failure observed from multiple probes.
• Loss of power or network telemetry at a site.
• Increased latency between South Africa and an overseas region.
• Queue growth caused by a dependency rather than by local compute.

Run black-box probes from more than one network. A probe in a Johannesburg data centre cannot prove that a customer on a mobile network in Limpopo can reach the service. Synthetic checks should cover DNS, TLS, authentication and a safe business transaction, not only /healthz.

During load shedding, preserve the monitoring path. Keep a small local Prometheus or agent able to buffer samples when the central endpoint is unreachable, then remote-write after connectivity returns. For critical services, alert on the absence of expected telemetry as a separate “monitoring blind spot” condition rather than labelling it immediately as application failure.

Keep telemetry reliable, affordable and lawful

Observability is a distributed system with its own failure budget. If every application sends unsampled traces, verbose logs and high-resolution metrics during an outage, telemetry volume rises precisely when operators need the platform most.

Apply controls before production:

• Sample routine successful traces and retain errors, slow requests and selected critical journeys at a higher rate.
• Use log levels that can be changed safely without redeploying every service.
• Set retention by operational value rather than habit.
• Estimate ZAR cost per million samples, gigabyte of logs and retained trace span.
• Test remote-write queues and backend capacity during incident-like bursts.

Mimir is useful for multi-cluster metric storage, but centralisation does not remove failure modes. Protect tenants with quotas, separate critical recording rules from exploratory queries, and ensure a noisy team cannot exhaust shared query capacity. Recent work across the Grafana ecosystem has also focused on reducing observability cost through adaptive log decisions and more efficient exploration workflows.[4]

POPIA adds another design constraint. Logs and traces can contain email addresses, tokens, identifiers and request payloads even when nobody intended to collect personal information. Redact at the instrumentation boundary, restrict access by role, and document where telemetry is stored and replicated. Cross-border processing should be reviewed with the organisation’s legal and security teams; “it is only a log” is not a reliable compliance argument.

Practise the failure modes that production will eventually provide

Reliability improves when teams rehearse specific failures rather than running a generic disaster-recovery exercise once a year.

1. Inject packet loss and latency between the application and eu-west dependency.
2. Return slow responses from the payment provider without returning errors.
3. Fill a queue and verify back-pressure, alerting and operator controls.
4. Stop telemetry export while keeping the application available.
5. Restore a Mimir, Loki or Tempo component and confirm that dashboards remain useful.
6. Simulate a site outage during a power transition and validate black-box monitoring from another network.

Each exercise should have a measurable assertion: checkout remains available, duplicate payments do not occur, queue age stays below a defined limit, or the alert fires within the agreed response window. Record the result in an engineering backlog. A game day that produces only confidence is entertainment; one that changes a timeout, runbook or architecture is reliability work.

The strongest Distributed System Reliability Engineering practice is therefore not a larger dashboard or a longer list of tools. It is a set of explicit failure decisions: what can be stale, what can be queued, what must fail closed, which signal pages a human, and how the service behaves when South Africa cannot reach the rest of the world for a while.