self

Self-Healing Infrastructure Monitoring Models for a Load-Shedding Reality

Opsgenie

15 Sep 2026 — 4 min read

Self-Healing Infrastructure Monitoring Models for a Load-Shedding Reality

The third time a Cape Town edge node vanished during Stage 6 load-shedding, nobody even noticed until looking at the Grafana dashboard the next morning. The node had quietly failed, drained traffic, rescheduled pods to Johannesburg, and restored replication — all without a human touching a keyboard. That wasn’t luck; it was the result of deliberate Self-Healing Infrastructure Monitoring Models wired into the observability stack.

This article looks at one concrete pattern for self-healing in a South African hybrid estate: Prometheus, Loki, Tempo, Mimir and Grafana working together to detect, decide, and remediate infrastructure issues under load-shedding, tight ZAR budgets, and POPIA constraints.

Why Self-Healing Monitoring Has Teeth in South Africa

Self-healing is not magic. It is a set of monitoring models where alerts become decisions and decisions become actions. In a South African context, four pressures make this more than a nice-to-have:

Load-shedding and unstable power: Nodes disappear in clusters, sometimes repeatedly in a day. Manual response does not scale when an OPS engineer is watching three provinces and an AWS region.
ZAR-denominated cloud bills: With the rand under pressure, aggressive autoscaling or over-provisioning hurts. Self-healing should respect cost, not just uptime.
POPIA and data sovereignty: Failing over to eu-west-1 for resilience is attractive, but data movement and logging must be carefully controlled.
Spotty last-mile connectivity: Sometimes the problem is not a node failure but a partial network collapse between an ISP and a regional DC.

A useful way to think about Self-Healing Infrastructure Monitoring Models is as a feedback loop with three layers:

Signal: Metrics, logs, traces and events that describe the system state.
Inference: Rules and models that decide whether something is broken, likely to break, or misbehaving.
Actuation: Automated actions triggered by those decisions — scaling, traffic shifts, configuration changes, or even toggling feature flags.

The goal is not to remove people from the loop; it is to keep people focused on novel problems, while the observability stack reliably handles repetitive infrastructure failures.

Building the Monitoring Model: Signals and Decisions

Self-healing starts with getting the right signals into Prometheus, Loki, Tempo and Mimir, and arranging them so they can drive automated decisions. The pattern below has been tested across Kubernetes clusters in Johannesburg and Cape Town plus an AWS eu-west-1 footprint.

Step 1: Model “Healthy” and “Unhealthy” Per Region

Instead of tracking node health per machine, model health per region role: for example, job=“k8s-node” with labels region and role (edge, core, batch). In Prometheus, record rules can summarise these into coarse health signals that are easier to reason about.

groups:
- name: region-health
  rules:
  - record: region:node_ready_ratio
    expr: sum(kube_node_status_condition{condition="Ready",status="true"}) 
          by (region)
         /
         sum(kube_node_status_condition{condition="Ready"}) by (region)

  - record: region:pod_sla_violation_ratio
    expr: sum(kube_pod_status_phase{phase="Failed"}) by (region)
         /
         sum(kube_pod_status_phase) by (region)

These ratios become the core metrics for deciding whether a region is healthy enough to accept more traffic or should be drained pre-emptively before the next load-shedding slot.

Step 2: Encode SLO-Aware Alerts, Not Just Symptoms

South African teams often run a mix of latency-sensitive APIs (like mobile banking) and batch workloads (like nightly reconciliations). Self-healing only works when the monitoring model understands which SLOs matter for which workloads.

A simple latency SLO for a Johannesburg ingress might look like this in PromQL:

groups:
- name: api-slo
  rules:
  - alert: JHBApiLatencyTooHigh
    expr: histogram_quantile(
            0.95,
            sum(rate(http_request_duration_seconds_bucket{
                  region="jhb",
                  handler="public-api"
            }[5m])) by (le)
          ) > 0.3
    for: 10m
    labels:
      severity: critical
      slo: "public-api-latency"
    annotations:
      summary: "95th percentile latency in JHB > 300ms for 10m"
      runbook: "drain-traffic-to-ct-and-eu, increase replicas"

The monitoring model encodes the SLO via labels and annotations. Those values are later used by automation to choose the appropriate remediation path: drain traffic, scale out, or degrade features.

Step 3: Enrich with Logs and Traces

Metrics tell you that something is wrong; logs and traces help infer why and whether automation is safe. Loki allows quick pattern detection around failures, while Tempo provides trace-level context for multi-region incidents.

A practical pattern for self-healing is to look for a combination of signals:

A region health metric degrading (e.g. region:node_ready_ratio dropping below a threshold).
A spike in specific error logs in Loki, such as “failed to attach volume” for on-prem storage under power fluctuation.
Trace spans in Tempo showing repeated retries or timeouts between that region and AWS eu-west-1.

By combining these, the monitoring model can distinguish between “scale up” problems (CPU saturation) and “scale down and drain” problems (hardware instability or network partition).

From Alerts to Actions: Wiring Automation Responsibly

Once the monitoring model reliably detects failure patterns, the temptation is to automate everything. That usually ends badly. A better approach is to define a small set of safe, reversible actions and steadily extend them.

Action 1: Traffic Shifting Across Regions

For many South African stacks, traffic is split between local DCs and AWS eu-west-1. When Cape Town becomes unstable, the safest self-healing action is to reduce its traffic share rather than slam all traffic into Ireland and potentially violate POPIA.

A practical implementation:

Expose per-region weight controls via a configuration store (ConfigMap, Consul, or a custom CRD).
Run a controller (Argo Workflow, Shell-operator, or a simple Python job) that can adjust those weights based on alert labels.
Use Alertmanager webhook receivers to trigger the controller when JHBApiLatencyTooHigh or region health alerts fire.

For example, an Alertmanager route might send specific SLO breaches to a self-healing endpoint:

route:
  receiver: "self-healing-controller"
  matchers:
    - slo="public-api-latency"

receivers:
- name: "self-healing-controller"
  webhook_configs:
  - url: "http://self-healing-controller.default.svc.cluster.local/hook"
    send_resolved: true

The controller decodes the alert, checks current capacities (using Prometheus queries), and modifies the traffic weights. All changes are visible as config metrics, so the human operators can audit what the automation did.

Action 2: Cost-Aware Autoscaling

With cloud bills rising in ZAR terms, blindly scaling up is not sustainable. Self-healing models can incorporate cost signals from cloud billing exporters and Mimir to decide whether to scale vertically, horizontally, or degrade features.

For example:

Ingest high-level cost data (e.g. daily spend per cluster) as metrics.
Define soft limits (e.g. “eu-west-1 spend > X ZAR per day”).
When SLOs are failing and spend is beyond the limit, prefer temporary feature degradation (turn off non-essential endpoints) instead of adding more replicas in Ireland.

This is more opinionated, but it is realistic. South African teams often prioritise core banking or payments SLOs and accept slower secondary features during cost spikes.