grafana

Start with an SLO, not a dashboard

At 18:00, the API error rate stayed below its alert threshold while customers in Gauteng reported failed payments. The real problem was not the application: a brief power interruption had caused an edge router to flap, traffic was…

Opsgenie

— 5 min read
Start with an SLO, not a dashboard

Modern SRE Monitoring Automation Frameworks: Building Resilient Observability for South African Platforms

At 18:00, the API error rate stayed below its alert threshold while customers in Gauteng reported failed payments. The real problem was not the application: a brief power interruption had caused an edge router to flap, traffic was taking a longer path to eu-west, and the payment provider’s latency had crossed the business limit without breaching the platform’s generic five-second alert.

That incident is why Modern SRE Monitoring Automation Frameworks must monitor customer impact, not merely infrastructure health. The practical design combines Prometheus and Mimir metrics, Loki logs, Tempo traces, Grafana dashboards, OpenTelemetry collection, and automation that turns telemetry into safe operational decisions.

Start with an SLO, not a dashboard

A monitoring framework becomes noisy when it begins with “What can this exporter measure?” Begin instead with the service-level objective. For a South African payments API, a useful starting point might be:

  • 99.9% of valid payment requests complete successfully over 30 days.
  • 95% of successful requests complete within 800 milliseconds for users in South Africa.
  • Requests routed to eu-west remain below a separately defined latency budget.

The first objective produces an error-budget calculation. The second exposes regional user experience. The third prevents a normal-looking national average from hiding an expensive or unreliable international route.

Prometheus records the signals needed to calculate these objectives. Mimir provides horizontally scalable, long-term metrics storage when several Kubernetes clusters, data centres, or cloud accounts must be queried together. This separation matters: local Prometheus servers can continue scraping during a network disruption, while Mimir supplies the cross-estate view once connectivity returns.

Use recording rules for expensive or frequently reused calculations. Keep alert rules focused on symptoms that require action:

groups:
  - name: payments-slo
    interval: 30s
    rules:
      - record: job:payments_requests:error_ratio5m
        expr: |
          sum(rate(http_requests_total{
            job="payments-api",
            status=~"5.."
          }[5m]))
          /
          sum(rate(http_requests_total{
            job="payments-api"
          }[5m]))

      - alert: PaymentsHighErrorBurn
        expr: |
          job:payments_requests:error_ratio5m > 0.02
        for: 10m
        labels:
          severity: page
          service: payments-api
        annotations:
          summary: "Payments error budget is burning quickly"
          runbook: "Check deployment, provider latency and regional routing"

The exact threshold should come from the SLO and traffic profile, not from a copied example. For low-volume services, combine burn rate with a minimum request count so that two failed requests do not page the on-call engineer.

Use the LGTM components for different questions

A useful framework gives each telemetry type a clear job:

  • Prometheus: short-term scraping, alert evaluation, Kubernetes metrics and service-level indicators.
  • Mimir: durable, multi-cluster metrics with tenant isolation and central querying.
  • Loki: operational logs, deployment events and structured application context without indexing every log word.
  • Tempo: distributed traces that show where latency accumulates across services, queues and external providers.
  • Grafana: dashboards, alert routing, correlations and investigation workflows across all four signals.

Grafana is most valuable when it reduces the number of context switches during an incident. A latency panel should link to the relevant Tempo traces; a trace should expose its trace ID in Loki; a log line should lead back to the service, deployment and region that produced it. This is more useful than a wall of unrelated panels.

OpenTelemetry has become a practical complement to Prometheus rather than a replacement for it. In Grafana’s 2025 observability survey, 71% of respondents reported using both Prometheus and OpenTelemetry in some capacity, while Prometheus remained in production for 67% of respondents.[1][2] Use OpenTelemetry for portable traces, logs and application metrics, then preserve Prometheus scraping where it is already reliable.

Automate collection at the estate boundary

In a hybrid environment, collectors should sit close to workloads. An OpenTelemetry Collector distribution such as Grafana Alloy can receive OTLP from applications, scrape Prometheus endpoints, add environment metadata and forward signals to the appropriate backends. The same pipeline can run in Kubernetes, on virtual machines, and near on-premises services.

Do not attach high-cardinality values such as user IDs, full URLs or payment references to metric labels. Put those values in logs or trace attributes, subject to redaction. A sensible metric label set might include service, route, method, status_class, region and environment.

For Loki, structured JSON logs are preferable to free-form strings. Include a timestamp, severity, service name, deployment version, region, request ID and trace ID. Redact identity numbers, card data, access tokens and unnecessary personal information before logs leave the workload boundary. POPIA compliance is not achieved by placing a “sensitive” label on an unredacted message.

Data residency also belongs in the architecture. Keep raw logs and traces in South African storage where policy requires it, and send only aggregated or scrubbed data to a central platform. Tenant boundaries, retention periods and access controls should be represented as code rather than configured manually after the first audit finding.

Make alert automation aware of South African failure modes

Load-shedding-aware monitoring does not mean suppressing every alert during a power interruption. It means distinguishing an expected infrastructure transition from an uncontrolled service failure.

  • Annotate planned maintenance and facility power events in the monitoring system.
  • Alert on UPS runtime, generator state, battery health and cooling alarms before hosts disappear.
  • Use a short grace period for expected node shutdowns, but page immediately when customer-facing SLOs burn.
  • Compare availability by site, availability zone and network path rather than relying on one national aggregate.

Last-mile connectivity needs similar treatment. A synthetic probe from a Johannesburg data centre may be healthy while users on a mobile network experience packet loss. Run probes from multiple South African locations and at least one external region. Track DNS time, TCP connection time, TLS negotiation, time to first byte and total duration separately.

For eu-west dependencies, alert on the latency distribution and route choice. A mean latency panel is particularly dangerous: a small number of very slow requests can damage checkout conversion while the average remains acceptable. Prometheus histograms and Tempo span duration make this visible.

Control ZAR cost with telemetry budgets

Observability can quietly become a second cloud bill. Cost pressure is especially visible when workloads scale in dollars but budgets are approved in ZAR. Grafana’s 2025 trend reporting identified cost as a leading criterion in observability-tool selection.[3]

Set budgets for samples, log volume, trace retention and cardinality. Use these controls:

  • Sample ordinary traces while retaining all error traces and slow traces.
  • Keep high-resolution metrics for recent incident analysis and downsample older data.
  • Route noisy debug logs to short retention, while preserving audit and security events longer.
  • Review top metric labels and log streams monthly.
  • Measure telemetry cost per service, team and customer transaction.

Do not solve cost by deleting the evidence needed for incident response. A better approach is tiered retention: fast local data for immediate operations, lower-cost object storage for historical analysis, and strict controls on what crosses regions.

Turn dashboards into an automated operating model

Infrastructure as code should define dashboards, alert rules, notification policies, recording rules, collectors and access permissions. Pull requests then become the review point for an alert’s owner, severity, query, runbook and expected response.

Every page should answer three questions:

  1. What customer or SLO impact is occurring?
  2. What is the most likely failure domain?
  3. What safe action can the responder take next?

Automation should handle reversible work: scaling a deployment within limits, restarting a failed consumer, shifting traffic away from an unhealthy zone, or opening a change record with correlated logs and traces. It should not automatically delete data, disable security controls or fail over across regions without tested safeguards.

After each incident, test the framework itself. Was the alert early enough? Did labels identify the affected region? Could the responder reach the relevant trace? Did load-shedding or a network partition create false silence? These questions improve the monitoring system faster than adding another dashboard.

The strongest Modern SRE Monitoring Automation Frameworks are deliberately unglamorous: explicit SLOs, bounded telemetry, portable collection, regional context and automation with guardrails. When those pieces work together, Grafana becomes the investigation surface rather than the place where unresolved noise is displayed.

Read more