self

The Night the Cluster Healed Itself (Mostly)

The Redis primary went dark three minutes into stage‑4 load shedding. The Cape Town edge node lost its rack, failover to Johannesburg fired, and the latency SLO dashboard started bleeding red. What didn’t happen: no one got out…

Opsgenie

— 4 min read
The Night the Cluster Healed Itself (Mostly)

Self-Healing Infrastructure Monitoring Models: Building Systems That Fix Themselves Before Eskom Breaks Them

The Night the Cluster Healed Itself (Mostly)

The Redis primary went dark three minutes into stage‑4 load shedding. The Cape Town edge node lost its rack, failover to Johannesburg fired, and the latency SLO dashboard started bleeding red. What didn’t happen: no one got out of bed. The incident channel stayed quiet, the post‑mortem was a single line: “self‑healing worked, but we cut it close”. That night was the first time the team’s Self-Healing Infrastructure Monitoring Models behaved exactly as designed – and exposed all the places they still weren’t good enough.

This is where self‑healing stops being a buzzword and starts being a design problem. The question isn’t “can we auto‑remediate?” – it’s “what monitoring model makes auto‑remediation safe in a hybrid South African estate with shaky connectivity, POPIA constraints, and a ZAR cloud budget that keeps shrinking?”

What “Self-Healing Infrastructure Monitoring Models” Actually Need to Model

Most teams start with automation (“let’s just restart the pod”) and bolt on observability later. That’s backwards. A self‑healing model needs to formalise four things before you hook it to Runbooks‑as‑Code:

  • System intent: what “healthy” means in terms of SLOs, error budgets, and business context.
  • Reliable signals: metrics, logs, and traces that are stable enough to drive automated decisions.
  • Guardrails: what must never happen automatically (e.g. failover across borders under POPIA constraints).
  • Remediation workflows: concrete, versioned procedures the automation can execute without guessing.

In practice that means modeling:

  • Load-shedding-aware capacity on local DCs (predictable power cuts, unpredictable generator failures).
  • Cross-region latency to eu-west for cloud failover when South African regions saturate.
  • Cost pressure in ZAR and the risk of autoscaling into bankruptcy.
  • Hybrid topology across on‑prem, local cloud regions, and European fallbacks.
  • Data locality where POPIA and client contracts forbid cross‑border moves.

The monitoring stack – Prometheus/Mimir for metrics, Loki for logs, Tempo for traces, visualised and orchestrated via Grafana – becomes the shared “source of decision truth” for automation, not just dashboards for humans.

Designing Signal Layers for Safe Auto-Remediation

Self‑healing stands or falls on signal quality. If the metrics lie, the automation misfires. A workable pattern in local estates is to build three distinct signal layers.

Layer 1: Hard Health and Capacity Signals

These are low‑level Prometheus metrics that should drive simple, deterministic actions: restart, reschedule, scale a replica set. Examples:

  • Node availability: power‑loss or host‑down events in on‑prem clusters.
  • Pod readiness: stable readiness probes on critical services.
  • Resource saturation: CPU, memory, and I/O under sustained pressure.

A PromQL example for a GPU‑free, CPU‑bound app that self‑scales on sustained load:

sum by (deployment) (
  rate(container_cpu_usage_seconds_total{
    namespace="payments",
    deployment="txn-api"
  }[5m])
) >
  0.7
and
sum by (deployment) (
  kube_deployment_status_replicas_available{
    namespace="payments",
    deployment="txn-api"
  }
) <
sum by (deployment) (
  kube_deployment_spec_replicas{
    namespace="payments",
    deployment="txn-api"
  }
)

This query only fires when the deployment is CPU‑hot and not already at max replicas. The self‑healing action is safe: bump the replica count within a defined limit inside the region – no cross‑border failover yet.

Layer 2: Business-Level SLO Signals

The second layer models user experience: latency, error rates, and throughput. The automation here is more nuanced. For example, a Joburg API cluster that fails over to eu‑west‑1 when local latency breaches 300ms for more than 10 minutes and error rates spike:

histogram_quantile(
  0.95,
  sum by (le) (
    rate(http_server_request_duration_seconds_bucket{
      job="checkout-api",
      region=~"af-south-1|eu-west-1"
    }[5m])
  )
) > 0.3
and
sum(
  rate(http_server_errors_total{
    job="checkout-api",
    region="af-south-1"
  }[5m])
) / 
sum(
  rate(http_server_requests_total{
    job="checkout-api",
    region="af-south-1"
  }[5m])
) > 0.02

Here the automation is not “scale up”; it’s “shift a percentage of traffic to eu‑west‑1”. Guardrails must reconcile this with POPIA: some tenants are allowed to move; others aren’t. That logic lives in the remediation workflow, but the SLO signal is what decides when to consider the move.

Layer 3: Context Signals for South African Reality

The third layer blends external and internal context:

  • Planned load shedding schedules.
  • ISP‑level last‑mile outages in township networks.
  • Cloud provider status in local regions (af‑south‑1, cape‑town‑1, etc.).

These often land in Loki as structured logs from external scrapers and internal orchestration. A simple LogQL example might tag incoming events with Eskom stages:

{app="infra-orchestrator"} | json | stage="4"

When stage‑4 hits, self‑healing rules can pre‑emptively drain nodes in the affected DC, move stateful workloads to more stable racks, or tighten autoscaling thresholds to avoid overcommitting generator capacity.

From Dashboards to Runbooks to Automated Actions

Metrics and logs alone don’t heal anything. The gap between “we see it in Grafana” and “the cluster fixed itself” is bridged by explicit runbooks converted into automation.

Step 1: Make Runbooks Machine-Readable

Instead of a wikis full of bullet‑pointed instructions, teams define remediation workflows in YAML or similar, with clear inputs, consequences, and approvals. For example, a Kubernetes self‑healing rule:

apiVersion: infra.ops/v1
kind: RemediationRule
metadata:
  name: txn-api-latency-failover
spec:
  description: |
    Shift a portion of payment traffic to eu-west-1 when
    af-south-1 is saturated but POPIA-allowed tenants exist.
  trigger:
    promql: >
      histogram_quantile(0.95,
        sum by (le) (
          rate(http_server_request_duration_seconds_bucket{
            job="checkout-api",
            region="af-south-1"
          }[5m])
        )
      ) > 0.35
  conditions:
    - type: tenant_filter
      allowRegions: ["eu-west-1"]
      popiaCompliant: true
  actions:
    - type: traffic_shift
      service: "checkout-api"
      fromRegion: "af-south-1"
      toRegion: "eu-west-1"
      percentage: 30
  safeguards:
    maxExecutionsPerHour: 2
    requireHumanAckWhenPercentage>50: true

This isn’t a generic template; it encodes a real constraint: POPIA‑compliant tenants can move, others can’t, and we never shift more than 30% without a human.

Step 2: Wire Alerts to Automation, Not Humans First

The alerting rules in Prometheus or Mimir still exist; what changes is the destination. Instead of posting to Slack as the first consumer, the alert is delivered to an orchestration service that

Read more