Stop Treating Every Red Signal as an Incident

Modern SRE Monitoring Automation Frameworks: Building Resilient Observability for South African Systems

At 18:00, a load-shedding event should not page the on-call engineer simply because a Johannesburg node has disappeared. It should trigger a controlled change in monitoring behaviour: suppress predictable infrastructure alerts, preserve customer-facing SLOs, and confirm that workloads have failed over as designed. That distinction is where Modern SRE Monitoring Automation Frameworks earn their keep.

The practical shift is from dashboards that describe an incident to automation that detects, explains and safely responds to it. Prometheus, Loki, Tempo, Mimir and Grafana provide the telemetry foundation; SLOs, event-driven runbooks and policy-as-code turn that foundation into an operating system for reliability.

Stop Treating Every Red Signal as an Incident

Traditional monitoring usually begins with thresholds: CPU above 80 percent, five hundred errors in a minute, or a node down for five minutes. Those rules remain useful, but they are poor substitutes for service context. A database replica being unreachable during a planned failover is not equivalent to customers receiving failed payments.

A stronger framework starts with three questions:

• Which user journey or business capability is at risk?
• What amount of failure is acceptable within the service-level objective?
• What automated action is safe, reversible and auditable?

For a South African retailer, the meaningful signal may be checkout availability in ZAR, not the health of every Kubernetes pod. For a SaaS platform serving clients in Cape Town and London, latency to eu-west may matter more than average latency across all regions. Monitoring automation should therefore be organised around service objectives and dependencies, not around whichever exporter happens to be installed.

Current observability practice is moving in this direction. Prometheus remains widely used for infrastructure metrics, while OpenTelemetry is increasingly used as a vendor-neutral collection layer; recent industry reporting also shows growing adoption of traces, profiles and unified application and infrastructure observability.[1][2]

The Five-Layer Framework: Signals, Context, Decisions, Actions and Evidence

A reliable automation framework separates concerns. This prevents alert rules from becoming an unmaintainable mixture of queries, business logic and shell scripts.

1. Signals: Collect metrics with Prometheus or OpenTelemetry, logs with Loki, traces with Tempo, and long-term metrics with Mimir.
2. Context: Attach consistent labels such as service, team, region, environment and criticality.
3. Decisions: Evaluate SLO burn rates, dependency health, maintenance windows and known environmental conditions.
4. Actions: Route notifications, open incidents, roll back releases, scale workloads or invoke a runbook.
5. Evidence: Preserve the alert, query, decision context and action result for post-incident review.

Grafana is useful at the decision and evidence layers: a single service view can connect a Prometheus time series to Loki logs and a Tempo trace, while alert rules and contact points can be managed alongside dashboards. The operational advantage is not a prettier graph; it is reducing the number of systems an engineer must mentally join during a failure.

Use SLO Burn Rates Instead of Static Thresholds

Static thresholds page too often for noisy systems and too late for fast-moving failures. An SLO burn-rate alert asks whether the service is consuming its error budget faster than planned.

Suppose an API has a 99.9 percent availability objective. A short, high-speed burn can page immediately, while a slower burn can create a ticket for daytime investigation. The exact windows depend on the service, but a multi-window policy is generally more useful than “five errors equals an outage”.

groups:
- name: payments-slo
  rules:
  - alert: PaymentsHighErrorBudgetBurn
    expr: |
      (
        sum(rate(http_requests_total{
          service="payments",
          status=~"5.."
        }[5m]))
        /
        sum(rate(http_requests_total{
          service="payments"
        }[5m]))
      ) > 0.05
      and
      (
        sum(rate(http_requests_total{
          service="payments",
          status=~"5.."
        }[1h]))
        /
        sum(rate(http_requests_total{
          service="payments"
        }[1h]))
      ) > 0.01
    for: 2m
    labels:
      severity: page
      service: payments
    annotations:
      summary: Payments availability is burning its error budget quickly
      runbook: payments-high-burn

This example is intentionally incomplete without a formally defined SLO and recording rules, but it shows the design principle: alert on customer-impacting behaviour and include enough metadata for automation to make a sensible routing decision.

Make Load Shedding and Connectivity Explicit Inputs

Power instability and last-mile connectivity are not unusual edge cases in local infrastructure. They should be represented as signals rather than handled through tribal knowledge.

A power-management system, UPS controller or facilities feed can publish a maintenance or risk event. Alertmanager can then inhibit node-level pages while continuing to alert on customer-facing SLOs. The inhibition must be narrow: suppressing every alert during a power event merely hides a genuine failover failure.

• Suppress expected node-down alerts for the affected site.
• Continue evaluating API availability, queue age and payment success rate.
• Increase probe frequency for critical public endpoints.
• Page if traffic fails to move to the standby region within the defined window.
• Automatically remove the inhibition when the facilities event closes.

Black-box probes are especially valuable where customers use unstable networks. Probe from more than one South African location when possible, and compare Johannesburg, Cape Town and external-region results. A single central probe can mistake an ISP or POP issue for a platform outage.

For cross-region services, record latency separately rather than hiding it in a global average. A healthy local request path can mask a degraded connection to eu-west, particularly when a third-party dependency is involved.

Control Telemetry Cost Before It Controls the Architecture

ZAR cloud-cost pressure changes observability design. Sending every debug log and high-cardinality label to a remote backend is not a sustainable automation strategy. Cost controls should be part of the collection pipeline.

Use Prometheus recording rules for expensive or frequently reused queries. Keep labels bounded: customer IDs, request IDs and arbitrary URLs do not belong in metric labels. Send detailed request context to Loki or Tempo instead, with appropriate sampling and retention.

Grafana’s LGTM approach supports this separation: Mimir handles scalable Prometheus-compatible metrics, Loki stores logs, and Tempo stores traces. Recent platform developments have also focused on adaptive telemetry and cost-aware log handling, reflecting the wider industry focus on observability spend.[3]

A practical policy might retain:

• High-resolution SLO metrics for the period needed to investigate incidents.
• Longer-term, downsampled metrics for capacity and trend analysis.
• Unsampled traces for errors and a representative sample of successful requests.
• Structured logs at useful levels, with sensitive fields redacted before export.

Cost dashboards should show ingestion volume, active series, log bytes, trace sampling rate and storage growth by team. If the platform team cannot explain which service generated the spend, the framework is not yet automated enough.

Design for POPIA, Hybrid Estates and Safe Automation

Telemetry often contains more personal information than teams expect. URLs can contain identifiers, logs can include email addresses, and traces can capture request attributes. POPIA and South African data-governance requirements therefore need to influence pipeline design, not merely the final storage location. South Africa’s 2024 National Policy on Data and Cloud states that processing data collected within the country must comply with applicable South African protection and security laws, with certain government data required to remain on infrastructure within the country.[4]

Apply controls at collection time:

• Drop or hash personal identifiers before telemetry leaves the workload.
• Separate operational metadata from customer content.
• Route sensitive logs to an in-country Loki deployment where required.
• Restrict cross-border trace export to approved fields and services.
• Audit who can query raw logs, not only who can edit dashboards.

Hybrid estates need the same labels and alert semantics across on-premise and cloud. Grafana Alloy or another OpenTelemetry Collector deployment can provide a controlled edge layer, buffering telemetry during WAN disruption and forwarding only what policy allows. Collector configuration should be versioned, reviewed and promoted like application code.

Automation also needs guardrails. A remediation workflow may restart a deployment or shift traffic, but it should require bounded retries, a clear rollback, an ownership label and an expiry for temporary changes. “Automatically fix everything” is not SRE; it is an unattended change-management process with better branding.

What Good Looks Like During the Next Incident

When a release increases payment failures, the framework should detect an SLO burn, attach the affected version and region, link representative traces, show matching Loki errors and route the incident to the owning team. If the failure began immediately after deployment, an approved workflow can pause rollout or roll back. If the same symptoms appear during a power event, the system should distinguish infrastructure loss from failed customer traffic migration.

The measure of success is not the number of dashboards or alerts. It is whether the on-call engineer can answer, quickly and confidently:

• Are customers affected?
• Which region, dependency or release is responsible?
• What action is safe to take now?
• How will the system prove that the action worked?

That is the practical purpose of Modern SRE Monitoring Automation Frameworks: convert telemetry into governed decisions, while keeping reliability, cost and data sovereignty visible in the same operating model.