observability

When the Observability Bill Is the Biggest Incident of the Month

The Grafana dashboard looked fine. Error rates were down after the weekend deploy, latency was behaving, and the on-call rotation had survived another load-shedding cycle without a major incident. Then the email arrived from finance: “Can someone explain…

Opsgenie

14 Sep 2026 — 5 min read

Observability Cost Governance Strategies for South African Teams Under Real Budget Pressure

When the Observability Bill Is the Biggest Incident of the Month

Nobody had pushed a new feature called “burn more money”. Yet that’s how observability cost creep often shows up in South African teams: not as a technical fault, but as a budget incident. This is where Observability Cost Governance Strategies stop being an abstract topic and become a survival skill.

This article takes a hands-on, practice-first look at how DevOps engineers and SREs in South Africa can govern observability costs across Prometheus, Loki, Tempo, Mimir and Grafana, without breaking visibility — or the ZAR budget.

Step 1: Make Observability Spend Visible Like Any Other Metric

The first governance mistake is treating observability cost as a finance-only concern. If it’s not on a dashboard, it doesn’t exist. The teams that avoid cost shocks make cost a first-class signal, alongside CPU and latency.

A practical pattern:

Expose ingestion and storage metrics from Prometheus, Loki, Tempo and Mimir.
Model approximate cost per unit (per million samples, per GiB stored, per query) in ZAR.
Create a “Cost SLO” — a soft, but visible target for observability spend per service.

For Prometheus/Mimir, the building blocks are usually:

scrape_samples_total (per job / per instance).
Remote-write bytes or samples, if using Mimir or another remote store.
Retention configuration (how long data is stored per tenant).

A simple first cut PromQL query to monitor sample volume per tenant could look like:

sum by (tenant) (
  rate(scrape_samples_total[5m])
)

If the cost model says a million samples per day maps to R100 of storage and processing, this query can be multiplied by a constant to show an approximate ZAR spend per tenant. It won’t be perfect, but it makes trends visible to engineering in the same place they watch error budgets.

The same pattern applies to Loki: ingest volume, index size and query counts per team or service. When an engineer can open Grafana and see that “team-payments” is burning 3× the ingest budget, governance conversations move from abstract arguments to specific charts.

Step 2: Align Retention and Fidelity with Real Risk (Not Fear)

The fastest way to waste money in observability is over-retention. Keeping everything “just in case” is a natural reaction after a nasty incident, but in a South African context with ZAR pressure, POPIA considerations and hybrid estates, it’s rarely sustainable.

A practical approach is to define retention tiers driven by risk and regulatory context:

Tier 1 – Regulatory / POPIA-sensitive data: Logs and traces that include personal information or data subject identifiers. Store them in-region (e.g. Johannesburg or Cape Town), with strict access controls and POPIA-compliant retention. Often shorter retention is better: minimise risk and cost.
Tier 2 – High-risk production services: Payments, core APIs, services impacted by load-shedding. Longer metrics retention (90–180 days), logs for 14–30 days, traces for 7–14 days. Enough history to debug complex failures and capacity trends.
Tier 3 – Low-risk / internal services: Short retention; e.g. metrics for 30–60 days, logs for 7 days, traces for 3–7 days. Focus on current health, not long-term forensics.

With Mimir, this often translates to per-tenant retention policies based on the service’s tier. For Loki, separate tenants or labels can map to different retention buckets, sometimes even different storage backends (cheap object storage for Tier 3, more controlled storage for Tier 1).

Tempo tracing retention is particularly important for cost governance. High cardinality spans and long trace retention can explode storage and query costs. A sensible pattern is:

Keep full traces for a short window (e.g. 3–7 days).
Retain aggregated trace metrics (latency histograms, error rates per span) for longer windows in Prometheus/Mimir.

That way, the expensive, high-fidelity trace data is available where it matters most — around incidents and new releases — but long-term visibility is maintained through cheaper numeric aggregates.

Step 3: Attack Cardinality and Noise Before You Attack the Budget

South African teams running mixed on-prem and cloud estates often discover that their biggest observability cost line item is not storage, but chaos: label cardinality, verbose logs and overly chatty alerts.

Prometheus and Mimir costs are tightly linked to label cardinality. Every unique label combination becomes a series, and every series has a storage and query cost. Common cardinality explosions include:

Embedding user IDs, session IDs or random request IDs as labels.
Using high-cardinality tags as part of metric names instead of as values.
Creating per-pod or per-container metrics with ephemeral labels that change constantly.

Cardinality governance is not glamorous, but it’s effective. Some practical rules:

Ban user-specific identifiers from metric labels. Keep them in logs if absolutely needed.
Limit label sets per metric to a small, stable core (environment, service, region, instance role).
Use metric relabelling in Prometheus to drop or normalise noisy labels at scrape time.

A simple Prometheus scrape config applying relabeling might look like:

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Drop ephemeral pod UID from labels
      - source_labels: ['__meta_kubernetes_pod_uid']
        regex: '.*'
        action: 'drop'
      # Normalise environment label
      - source_labels: ['__meta_kubernetes_namespace']
        target_label: 'env'
        regex: '(prod|staging|dev).*'
        replacement: '$1'

Loki noise control follows similar logic, but with a focus on log volume:

Move very verbose debug logs behind dynamic sampling, especially on high-traffic services.
Filter or drop health-check logs that don’t add value.
Avoid logging full payloads for every request unless there’s a clear, audited need.

LogQL can be used to enforce patterns and to monitor noisy streams. For example, tracking the top log streams by volume:

topk(10,
  sum by (job, level) (
    rate({cluster="prod"} |= ""[5m])
  )
)

Running this in Grafana and reviewing with teams weekly can surface “noise offenders” and lead to disciplined log hygiene — which often saves more money than any negotiated discount.

Step 4: Design for Load-Shedding and Connectivity Rather than Over-Provisioning

South African infrastructure has some unique stressors that impact observability design and cost. Eskom’s load-shedding is the obvious one; intermittent last-mile connectivity in some regions and cross-region latency to eu-west are others.

The naive reaction is to over-provision: build heavy HA setups in multiple regions, duplicate observability stacks across Johannesburg, Cape Town, and eu-west, and stream everything everywhere. Governance strategies need a more nuanced approach.

For Prometheus/Mimir:

Scrape locally, store centrally where justified: Use local Prometheus instances near the workloads (e.g. on-prem clusters or edge sites) with remote write into Mimir clusters in stable DC regions.
Buffer for load-shedding: Configure local Prometheus and Loki to tolerate short outages (e.g. using persistent disks and sensible retention) so metrics and logs survive brief power and network blips without requiring over-duplicated infrastructure.
Segment tenants by