distributed

Distributed System Reliability Engineering when the lights don’t always stay on

The post-incident review was oddly quiet. No shouting, no finger-pointing. Just an uncomfortable graph on the wall: request latency spiking every evening between 18:00 and 22:00, right when users in Johannesburg and Durban were trying to log in.…

Opsgenie

— 4 min read
Distributed System Reliability Engineering when the lights don’t always stay on

Distributed System Reliability Engineering when the lights don’t always stay on

The post-incident review was oddly quiet. No shouting, no finger-pointing. Just an uncomfortable graph on the wall: request latency spiking every evening between 18:00 and 22:00, right when users in Johannesburg and Durban were trying to log in. The kicker? There was no code deploy, no infra change, nothing obvious. The only consistent factor was something not usually modelled in architecture diagrams: Stage 6 load shedding.

This is where “Distributed System Reliability Engineering” stops being an abstract discipline and becomes painfully concrete. Reliability is not just about five nines on a slide; it’s about how services behave when the power grid is unstable, cross-region links to eu-west are saturated, and your compliance team reminds you that half your logs are subject to POPIA.

Reliability as a contract, not a vibe

Distributed System Reliability Engineering starts with being explicit about what “reliable” means in a specific context. For a fintech API serving South African merchants, it’s often less about pure uptime and more about:

  • Latency targets under congested links to international regions
  • Graceful degradation during partial connectivity loss
  • Data residency guarantees for transactional and user-identifying data
  • Managed behaviour under planned and unplanned power interruptions

That pushes us beyond “the service must be up” into modelling:

  • Failure domains: On-prem vs local cloud region vs eu-west, ISP vs DC, power vs network.
  • Blast radius: What breaks when one failure domain is degraded, and what must stay alive.
  • Reliability budgets: Not just error budgets, but cost budgets (ZAR), latency budgets, and even regulatory risk budgets.

Practically, this shows up as SLIs and SLOs defined per path and per region. For a multi-region API with local and eu-west backends, a basic SLI we’ve used looks like:

# PromQL: 95th percentile latency for login API requests from ZA
histogram_quantile(
  0.95,
  sum by (le) (
    rate(http_request_duration_seconds_bucket{
      job="edge-gateway",
      route="/api/login",
      region="za"
    }[5m])
  )
)

Grafana dashboards built on these SLIs force the team to look at reliability as behaviour over time rather than a binary “up/down” state. That mindset shift is the foundation of any serious Distributed System Reliability Engineering effort.

Designing for load shedding and last-mile instability

Outages caused by load shedding are rarely clean. Sometimes the data centre stays up on generators, but local ISPs flap. Sometimes office VPNs die, but edge PoPs at ISPs keep serving traffic. The reliability discipline needs to bake these patterns into architecture decisions, not treat them as unfortunate surprises.

Three concrete reliability patterns that have proved useful in South African environments:

    • Tracking buffer queue depth and age via Prometheus metrics.
    • Grafana panels for “time since last successful replication” per region.
    • Alerts that differentiate between “buffer filling but healthy” and “buffer approaching data loss territory”.
  1. Graceful degradation under partial connectivityInstead of hard-failing every user action when eu-west is unavailable, UI and API flows expose a “limited functionality” mode. From a reliability perspective, the system remains “available”, but only a subset of operations are allowed.The trick is to reflect this explicitly in SLIs:Tempo traces become critical for verifying that degraded flows are behaving as intended, while Loki logs capture the feature flags and decision points that switch users into fallback modes.
    • Primary SLI: Successful completion of full experience flows.
    • Secondary SLI: Availability of degraded flows during upstream outages.
  2. Power-aware scheduling for batch jobsLarge ETL jobs that hit external APIs or databases can destabilise the system when they coincide with power or network instability. Reliability engineering in South Africa sometimes looks like “avoid heavy jobs during predictable Stage 4+ windows”.That policy can be encoded as schedules in CI/CD and Kubernetes, but it also needs observability:
    • Prometheus metrics for job start/finish, labelled by site and power status.
    • Grafana alerts when jobs repeatedly fail in specific time windows.

Local write buffers with eventual sync to eu-westWhen eu-west latency jumps because of undersea cable congestion, synchronous writes to remote databases cause cascading timeouts. Instead, local components buffer writes on a South African edge and replicate asynchronously when the link stabilises.Observability-wise, this requires:

# PromQL: buffer age SLI
max_over_time(
  write_buffer_oldest_event_age_seconds{
    job="za-ingest",
    buffer="eu-west-sync"
  }[10m]
)

Cost-aware reliability in a ZAR-constrained world

Globally, there’s been a push towards cost-aware SRE since around 2024, with platform teams asked to justify every extra replica and feature flag. Locally, currency exchange rates amplify that pressure: every extra eu-west node priced in USD or EUR hits the ZAR budget hard.

Distributed System Reliability Engineering in this context requires acknowledging that:

  • Over-provisioning in eu-west for low-latency reads may be unjustifiable.
  • Hybrid architectures (on-prem plus local cloud region plus eu-west) are often a necessity, not an anti-pattern.
  • Observability costs themselves (metrics cardinality, log volume, trace retention) need to be treated like any other reliability trade-off.

A practical approach that’s worked well:

  1. Tiered SLOs per region and per featureHigh-value payment flows might get a 99.9% availability SLO in eu-west and local regions, while low-value reporting endpoints only get 99% and longer recovery windows. That allows the team to selectively invest in redundancy where it matters.
    • Defining “golden” metrics that must be retained longer (SLOs, core infrastructure).
    • Shorter retention or sampling for verbose debug logs, especially in low-risk services.
    • Using label-based cost allocation: team, service, and environment labels to track high-volume offenders.
  2. Budget-linked reliability experimentsChaos experiments are often framed purely in technical terms. A cost-aware variant asks: “What happens to our SLOs if we drop one replica in eu-west and rely more on local caching?” Observability data, rendered in Grafana, becomes the evidence to justify or reject these trade-offs.

Metrics and logs as first-class cost itemsMimir and Loki provide knobs for retention and sampling, but those knobs are rarely owned by SREs in a deliberate way. A reliability-focused practice includes:

# LogQL: identify noisy services in Loki
sum by (service) (
  rate({env="prod"} |= "DEBUG"[5m])
)

POPIA, data sovereignty, and observability boundaries

Distributed System Reliability Engineering in South Africa cannot ignore POPIA and broader data sovereignty requirements. The observability stack itself can easily become a compliance liability if logs and traces include personal data shipped to foreign regions.

A disciplined approach treats observability data as regulated:

  • PII-sanitised logging: Loki pipelines that explicitly strip or hash identifiers before storage.
  • Region-tagged telemetry: Prometheus and Tempo labels that encode where data originated and which legal regime applies.
  • Separate clusters for sensitive telemetry: Critical user-facing systems emitting to a local Mimir/Loki cluster, with aggregated, non-sensitive telemetry replicated to eu-west for global analytics.

Read more