When the Observability Bill Is the Biggest Surprise in the Retrospective

Observability Cost Governance Strategies for South African Teams Under Real Budget Pressure

When the Observability Bill Is the Biggest Surprise in the Retrospective

The outage wasn’t the shock; the invoice was. After a winter week of aggressive load-shedding, a fintech team scaled up logging and tracing “just for safety” and turned on debug logs across three Kubernetes clusters. The incident retrospective ended with a R480k surprise: observability and logging costs for a single month, mostly from ingestion and cross-region data transfer. That’s the scenario where “Observability Cost Governance Strategies” stops being a theory and becomes a survival skill.

South African DevOps teams are juggling hybrid estates, POPIA constraints, ZAR-dollar exchange-rate pain, and cross-region latency to eu-west. In that mix, observability costs can quietly become the third-largest line item after compute and databases. This article takes a hands-on approach: how to design observability cost governance in practice, with Prometheus, Loki, Mimir, Tempo and Grafana sitting at the centre of the story.

Cost-Aware Observability Architecture for Hybrid South African Estates

Start with architecture, not budgets. Most cost grief stems from an observability topology that ignores where data lives and how it moves.

A common South African pattern:

• One or more on-prem DCs (Johannesburg, Cape Town), sometimes hosted at Teraco.
• Primary cloud workloads in AWS eu-west-1 or Azure West Europe because local regions still lag in services or pricing.
• Latency-sensitive components (payments, trading) running closer to users, with caching and local failover.

Without cost governance, the instinct is to centralise everything into a single “big” observability cluster in eu-west-1. That’s usually a mistake. You pay for:

• Cross-region egress on every log line and trace span leaving South Africa.
• Larger, more complex Grafana stacks with over-provisioned storage.
• Higher cardinality due to unbounded labels from multiple environments.

A more cost-aware architecture:

• Local collection and tiered retention: Run Prometheus and Loki agents locally in each DC/cluster. Use short retention (e.g. 7–14 days) on hot storage and ship only aggregates or sampled data to a central Mimir and Tempo setup.
• Data sovereignty-aware routing: Traffic and logs containing personal data subject to POPIA stay in-country, with anonymised metrics sent to cloud for cross-environment dashboards.
• Split write/read paths: Writes go to cheaper local storage; reads for long-term trend analysis hit cloud Mimir/Tempo with lower resolution metrics and sampled traces.

In practice, this means accepting that not all data is equal. You don’t need full-fidelity traces for every HTTP request from your mobile app during peak load-shedding chaos. You need enough sampling to understand systemic behaviour, not every stack trace.

Label Discipline: The Cheapest Observability Cost Governance Strategy

Most DevOps engineers learn cardinality pain the hard way. Prometheus and Mimir don’t charge per “nice dashboard”; they charge in CPU, RAM, and storage for every unique time series. In Loki, the same applies to indexed labels. Cost governance starts with label governance.

Three practical rules that make a difference:

• No user identifiers in labels: Avoid labels like user_id, session_id, request_id. Use these only in log bodies and restrict index labels to bounded sets like service, env, region.
• Bounded cardinality per environment: Every environment should have a known, capped number of instances per service. Monitor it. Use max_series-style safeguards where possible.
• Drop noisy labels at ingestion: Configure relabeling to remove labels that explode cardinality, especially from auto-generated tags in service meshes and cloud SDKs.

Here’s a concrete example from a South African payments platform using Prometheus for Kubernetes metrics. The following snippet drops pod-specific labels in a cost-sensitive environment, keeping the focus on service-level metrics:

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Keep only service, namespace, environment
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: service
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_label_env]
        target_label: env

      # Drop per-pod unique IDs to reduce cardinality
      - source_labels: [__meta_kubernetes_pod_name]
        action: drop

This looks like a small tweak, but it changes cost curves significantly. Mimir and Grafana respond better when dashboards are built around service-level metrics, not per-pod obsessive detail, especially when clusters scale up aggressively during load-shedding-related traffic spikes.

Tiered Retention and Storage: Hot vs Cold vs “We Don’t Actually Need This”

Observability data retention is where most cost governance conversations become hand-wavy. A few concrete policies help.

Think in three tiers:

• Hot data (minutes to days): Used for real-time alerting, debugging active incidents, and SLO tracking. This is your Prometheus metrics plus recent logs in Loki and recent traces in Tempo.
• Warm data (weeks to a few months): Used for performance trend analysis, capacity planning, and financial reporting on SLOs. Typically stored in Mimir with downsampled metrics and sampled traces.
• Cold or archived data (months to years): Required for compliance, audits, or rare investigations. This may live in object storage with reduced index capabilities and slower query paths.

In a South African context with strict POPIA requirements and rising cloud storage prices, it’s often cheaper to keep some cold data on-prem in S3-compatible storage or tape-backed solutions, with Grafana querying through data sources that know the trade-off—slower queries, but lower long-term costs.

Loki configuration is a good place to encode these decisions. For example, you can set different retention by tenant or log stream. A minimal, cost-conscious Loki config might look like:

schema_config:
  configs:
    - from: 2024-01-01
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 168h  # 7 days for full logs
  per_stream_rate_limit: 2MB
  per_stream_rate_limit_burst: 4MB

For POPIA-sensitive services, retention might be lower and anonymisation more aggressive, while infrastructure logs with no personal data may be kept longer in cheaper storage.

Load-Shedding, Connectivity, and Cost-Aware Alerting

South African infrastructure has unique constraints: load-shedding windows, spotty last-mile connectivity, and cross-region latency to eu-west. Observability cost governance must respect that reality in alerting design.

Two practical strategies:

• Adaptive alerting during load-shedding: During scheduled outages and generator failover, it’s common to see transient spikes and noise. Instead of pushing every metric to central Grafana and Mimir at full resolution, reduce scrape frequency or build conditional alerting rules that adjust thresholds during known risk periods.
• Local buffering and backpressure: If connectivity to eu-west drops or becomes unstable, agents should buffer locally and prioritise metrics over verbose logs. Lost debug logs are cheaper than losing high-level metrics when connectivity returns.

A PromQL example for cost-focused alerting on observability infrastructure itself:

# Alert when Mimir is ingesting too many time series, indicating a cost risk
sum(mimir_ingester_active_series) > 1e6

This isn’t just an operational alert; it’s a cost governance tool. When that alert fires, someone should ask: which teams or services added labels or enabled new scraping that pushed active series above a safe threshold?

Similarly, Loki can be used to enforce ingestion discipline. For example, use LogQL to identify noisy services:

{env="prod"} | stats count() by service | sort desc | limit 10

This query quickly shows which services are pumping out the