Start with an observability cost budget that engineers can use

Observability Cost Governance Strategies: Control Telemetry Spend Without Blinding Your On-Call Team

The invoice jumped after a Kubernetes rollout in Cape Town. Nothing looked broken: request rates were normal, dashboards loaded, and the alert queue was quiet. The cause was a new pod_uid label added to several Prometheus metrics. Every restart created another set of time series, while Loki retained verbose container logs and Tempo sampled every trace from a noisy checkout endpoint.

That is the uncomfortable truth behind Observability Cost Governance Strategies: most cost explosions begin as perfectly reasonable engineering changes. A label added for debugging, a longer retention period requested for compliance, or a “capture everything” trace policy can quietly turn into a permanent operating cost.

Start with an observability cost budget that engineers can use

A single platform-wide budget is too blunt. Split spend into the signals and teams that create it: metrics, logs, traces, profiles, storage, egress and query infrastructure. Then assign ownership to the service or platform team responsible for the telemetry.

• Prometheus and Mimir: track active series, samples per second, remote-write volume and long-term retention.
• Loki: track ingested bytes, log streams, index size, retention and expensive queries.
• Tempo: track spans received, sampled spans, storage bytes and trace retention.
• Grafana: track dashboard queries, alert evaluations and data-source load.

Express the budget in engineering terms rather than only in rand. For example, a team may have a monthly ZAR allocation, a maximum active-series growth rate, and a log-ingestion limit per production service. The finance number tells the organisation when to act; the technical limits tell engineers what to change.

Tag cloud resources, object-storage buckets and managed observability tenants by environment, product and team. For a hybrid estate, include the cost of moving telemetry from on-premises clusters to a cloud-based Mimir, Loki or Tempo deployment. Cross-region traffic to eu-west can be more expensive than the storage itself, particularly when every log line and trace is exported synchronously.

Make cardinality a governed design decision

Metrics become expensive when labels multiply. A metric with 20 methods, 10 status codes, 12 regions and 50,000 request paths does not have “a few useful dimensions”; it has a potentially enormous series space.

Reject unbounded labels by default. Request paths, user IDs, email addresses, session IDs, container IDs and raw exception messages generally belong in logs or exemplars, not in metric labels. Prefer bounded dimensions such as service, route template, HTTP method, status class, availability zone and workload.

Measure before deleting. Prometheus exposes useful signals about the shape of the problem:

topk(20,
  sum by (__name__) (
    count by (__name__, job, instance) ({__name__=~".+"})
  )
)

In practice, the query used to identify high-cardinality metrics should be adapted to the Prometheus or Mimir version and the available metadata APIs. The more useful governance pattern is a recurring report that ranks metrics by active series, ingestion rate and whether they appear in dashboards, recording rules or alerts.

Mimir is valuable when a South African organisation needs horizontally scalable, multi-tenant metrics with centralised retention. It does not remove the need for discipline. A badly labelled metric remains expensive whether it is stored locally, remote-written or aggregated in Mimir.

Grafana can turn cardinality reviews into a visible engineering workflow: publish a dashboard showing top metrics, owners, environments and week-on-week growth, then make the owning team responsible for exceptions. Grafana Cloud’s Adaptive Metrics is designed to recommend aggregation or removal of underused series; Grafana reported an average 35% metrics-cost reduction across more than 1,200 organisations in 2024.[1]

Use retention tiers instead of keeping everything forever

Retention should follow investigative value. A common mistake is applying the same period to every signal because the storage policy is easy to configure.

• Keep high-resolution Prometheus or Mimir metrics for the period needed for incident response and capacity analysis.
• Use recording rules for durable service-level trends, such as hourly error rate, saturation and latency percentiles.
• Retain Loki logs longer only when they support audit, security investigation or contractual requirements.
• Keep detailed Tempo traces briefly, then retain sampled traces or service-level aggregates for longer-term analysis.

For example, detailed application logs might remain searchable for 14 days, while security logs go to a separate controlled archive. A 30-day operational metric view can coexist with a 13-month monthly capacity series. This is cheaper and often more useful than storing every raw sample indefinitely.

POPIA changes the governance question. Log retention must account for personal information, access controls and deletion requirements, not just cost. Avoid placing identity numbers, access tokens or full request bodies in Loki. Redaction at the collector is safer than relying on every downstream query author to remember what must not be displayed.

Reduce noise at the source, not only in the backend

Loki is not a dumping ground for every line emitted by a container. Set structured logging standards and separate event severity from developer debugging output. Keep labels low-cardinality: cluster, namespace, workload and level are usually safer than request IDs or arbitrary logger fields.

Drop predictable noise before ingestion. Health checks, successful polling requests and repeated connection messages may be useful during development but provide little production value. Use pipeline stages to parse and redact, while retaining enough context to investigate failures.

pipeline_stages:
  - json:
      expressions:
        level: level
        message: message
  - drop:
      expression: "level == \"debug\""
  - labeldrop:
      - request_id
      - user_id

The exact configuration depends on the collector and deployment model, but the policy is portable: parse fields, remove sensitive data, avoid high-cardinality stream labels and discard known noise before it reaches Loki.

Tempo needs the same treatment. Tail-based sampling can retain slow, failed or unusual traces while reducing routine successful traffic. Preserve traces containing server errors, high latency or important business operations. Do not sample randomly and assume the remaining data represents incidents.

Design alerts for unstable power and imperfect networks

Load-shedding-aware alerting is a cost control measure as much as an availability measure. During a planned power event, a local data centre may lose capacity, exporters may stop scraping and alert evaluations may fail in batches. If every missing target produces a page, the organisation pays in human attention and incident fatigue.

Use recording rules and multi-window alerts to distinguish a genuine service failure from a telemetry gap. Alert on the monitoring system’s own health, but route lower-confidence scrape failures to a ticket or maintenance notification when a scheduled outage explains them.

For branches and POPs with unreliable last-mile connectivity, avoid exporting every raw event continuously. Buffer logs locally, apply rate limits, and prioritise security events and service-level metrics. A regional agent can continue collecting while the link is down, then forward within a bounded window. Set explicit limits so a prolonged outage does not create an unbounded queue or a costly recovery burst.

For services used from South Africa but hosted in eu-west, measure both user-facing latency and telemetry transport latency. A dashboard query that repeatedly scans months of remote logs is not an operational strategy. Precompute common views, use narrower time ranges and investigate whether a local read replica or regional aggregation is justified.

Put cost changes through the same path as production changes

Telemetry configuration deserves code review. Store scrape configurations, recording rules, Loki pipelines, Tempo sampling policies and retention settings in version control. Require every change to state its expected impact on series count, bytes per second, storage and diagnostic coverage.

1. Measure the current ingestion and active-series baseline.
2. Apply the change in a non-production or limited tenant.
3. Compare cost indicators and incident queries before and after.
4. Define exemptions for security, compliance and critical service-level signals.
5. Roll out gradually and review the result after one billing cycle.

Set automated guardrails in CI. A pull request that introduces a label sourced from an unbounded field should fail or require an explicit approval. A new dashboard should not be allowed to add a query scanning an entire year of raw Loki data without an owner and a reason.

Monthly reviews should focus on exceptions, not ceremony. Ask which metrics no alert or dashboard used, which log streams grew fastest, which traces helped during incidents, and whether teams can still answer their defined service-level questions. The target is not the smallest telemetry bill. It is the lowest cost that preserves fast, trustworthy decisions during failure.

The best Observability Cost Governance Strategies make that trade-off visible. Prometheus and Mimir provide the measurements, Loki and Tempo preserve the investigative detail, and Grafana gives teams a shared place to see both operational value and financial impact. When cost is treated as a property of telemetry design, the next unexpected invoice becomes a reviewable change rather than a surprise.