Start with a cost map, not a retention argument

The invoice arrived two weeks after a seemingly harmless deployment: a new HTTP route label had multiplied the payments service into tens of thousands of time series. Nothing was broken. Prometheus was healthy, dashboards loaded, and alerts remained…

Start with a cost map, not a retention argument

Observability Cost Governance Strategies: Stop Paying for Telemetry Nobody Uses

The invoice arrived two weeks after a seemingly harmless deployment: a new HTTP route label had multiplied the payments service into tens of thousands of time series. Nothing was broken. Prometheus was healthy, dashboards loaded, and alerts remained green. Yet the monthly observability bill had climbed while the engineering team gained almost no useful signal.

That is the uncomfortable reality behind Observability Cost Governance Strategies: telemetry is not expensive merely because there is a lot of it. It becomes expensive when nobody owns its production, retention, cardinality or destination.

Start with a cost map, not a retention argument

Before deleting logs or reducing trace sampling, create a cost map that connects telemetry to a service, team and operational purpose. In a hybrid South African estate, that usually means accounting for Prometheus collectors in a data centre, Mimir storage in the cloud, Loki log ingestion, Tempo traces and the Grafana dashboards and alert rules that query them.

Tag every workload with a small, stable ownership vocabulary:

  • team: the accountable engineering group.
  • service: the production service or platform component.
  • environment: production, staging or development.
  • region: for example, Johannesburg, Cape Town or eu-west.
  • cost_centre: the financial owner in ZAR reporting.

Do not use arbitrary pod names or deployment hashes as ownership labels. Those values change frequently and create new series. A cost report should answer “who owns this?” without requiring an engineer to decode Kubernetes metadata.

In Grafana, usage and cardinality dashboards can expose which teams generate the most active series and which labels drive growth. The same review should include Loki bytes ingested, Tempo span volume, Mimir active series and query volume. A team producing modest ingestion but running expensive, wide-range queries may still be consuming a significant share of the platform.

Put a cardinality budget around every service

Metrics cardinality is usually the first governance control because one metric multiplied by several unbounded labels can become thousands of time series. Request paths, customer IDs, email addresses, container IDs and full URLs are common offenders.

For example, this metric is dangerous:

http_requests_total{method="GET", path="/customers/839271/orders", status="200"}

The path contains an identifier. A safer instrumentation pattern normalises it:

http_requests_total{method="GET", route="/customers/:id/orders", status="200"}

Set a service-level cardinality budget and review exceptions through pull requests. The budget does not need to be identical everywhere. A payment gateway may justify more dimensions than a low-value internal batch job, but the owner must explain why.

Prometheus recording rules can preserve useful aggregations while allowing raw dimensions to be dropped from long-term storage. In Mimir, keep high-resolution data for the operational window and record lower-cardinality, longer-lived series for capacity and business reporting.

Cost governance should also include a “series per deployment” check. A deployment that increases active series by 40% should trigger review even if application tests pass. This catches telemetry regressions before they reach the cloud bill.

Separate telemetry by operational value

Not every signal deserves the same retention period or destination. Keeping every debug log and unsampled trace for months is convenient, but convenience is a poor storage policy.

  • Metrics: retain high-resolution data for incident response, then downsample or record aggregated series for long-term trends.
  • Logs: keep structured security and audit events longer than routine request logs; expire noisy informational events quickly.
  • Traces: sample ordinary successful requests, while retaining errors, slow requests and important business flows.
  • Profiles: collect during performance investigations or scheduled windows rather than continuously across every workload.

For Loki, define retention by tenant, stream or policy rather than applying one long default to the entire estate. Kubernetes logs from a disposable development namespace should not have the same retention as authentication or payment audit events. Drop health-check noise at the agent or gateway before it reaches Loki; deleting it later still incurs processing and ingestion cost.

Tempo sampling should be deliberate. Tail-based sampling can retain traces with errors, high latency or selected route names while reducing the volume of successful, unremarkable requests. A trace policy that keeps every request to investigate a once-a-month incident is usually poor economics.

Recent observability practice increasingly uses OpenTelemetry as a common collection layer, making it easier to route and transform metrics, logs and traces before storage. That matters in a hybrid environment: sensitive attributes can be removed in the Collector, high-value data can go to the primary platform, and lower-value data can go to cheaper storage.

Make load-shedding and connectivity part of the design

South African infrastructure adds constraints that generic cost playbooks often miss. A prolonged load-shedding event can restart agents, delay remote writes and produce a flood of recovery logs once connectivity returns. A cost policy that only works during stable network conditions is not a governance policy; it is a best-effort suggestion.

Use local buffering with explicit limits. When a Johannesburg site cannot reach cloud storage, agents should queue only the signals worth preserving and discard low-value debug traffic once the queue is full. Unbounded buffers merely move the failure from the network to disk or memory.

Remote-write configurations should protect the application estate from an observability outage. Limit retries, cap queue sizes and monitor dropped samples separately from ingestion volume. If Prometheus continues retrying indefinitely during a eu-west connectivity problem, the resulting backlog can consume resources precisely when the platform needs them for recovery.

Cross-region latency also affects cost. Sending every local log and trace to eu-west may increase transfer charges and make dashboards sluggish. Keep operationally local data near the workload where possible, while applying POPIA and data-residency requirements to decide which records may leave South Africa. Data sovereignty is not only a compliance question; it is also a routing and storage-cost decision.

Turn governance into deployment controls

Manual monthly reviews are too slow. Add telemetry checks to the same delivery pipeline that deploys application code.

  1. Reject metric labels that match known unbounded fields such as IDs, UUIDs, full URLs or timestamps.
  2. Require an owner and retention class for every new log stream.
  3. Compare expected series growth with the service’s approved budget.
  4. Run a canary deployment and measure active series, log bytes and trace volume before full rollout.
  5. Alert the owning team when ingestion exceeds its budget for a sustained period.

A basic PromQL alert can detect a sudden increase in active series:

(
  sum by (service) (prometheus_tsdb_head_series)
  /
  sum by (service) (prometheus_tsdb_head_series offset 1h)
) > 1.25

The exact expression depends on how Prometheus instances are labelled, but the principle is more important than the query: alert on rate of change, not just an absolute threshold. A small service can suffer a serious regression without approaching the platform-wide limit.

For logs, alert on bytes and event rates by service rather than merely counting lines. A stack trace repeated thousands of times during a database outage is both operationally useless and costly. Rate limiting, deduplication and structured severity fields make it possible to control that failure mode without hiding genuine errors.

Give engineers a safe path to richer visibility

Cost governance fails when it is experienced as “delete telemetry”. Engineers then bypass the policy, disable sampling during incidents or create unofficial monitoring systems. The better approach is to provide approved ways to increase visibility temporarily.

  • Allow a time-limited high-detail trace policy for a named service.
  • Provide a debug log level with automatic expiry.
  • Offer a recording-rule template for common RED and USE metrics.
  • Expose team-level usage and cost dashboards in Grafana.
  • Require an owner and expiry date for every cost exception.

Every exception should answer three questions: what decision will this data support, how long is it needed, and who will remove it? During an incident, that small amount of structure prevents a temporary diagnostic setting from becoming permanent platform debt.

Adaptive telemetry features can help identify unused or partially used metrics and recommend lower-cardinality versions. They should support engineering judgement rather than replace it: an apparently unused metric may be essential during a rare regulatory investigation or a once-a-quarter batch failure.

Measure value, not just volume

A useful governance review combines financial and operational measures. Track ZAR cost per service, active series per service, log gigabytes per transaction, trace volume per successful request and the percentage of telemetry linked to an alert, dashboard or investigation.

Then review the exceptions. If a service consumes substantial Loki storage but has no dashboard, alert or documented operational use, the owner has a clear optimisation opportunity. If a small volume of Tempo traces repeatedly shortens incident resolution, retain it even if it looks expensive on a raw-volume chart.

The strongest Observability Cost Governance Strategies do not pursue the lowest ingestion number. They make expensive telemetry explainable, owned and proportional to the decisions it enables. That means controlling cardinality before storage, routing data according to value, respecting local connectivity and sovereignty constraints, and giving teams automation rather than a blunt deletion mandate.

When the next invoice arrives, the useful question is not “why did observability cost more?” It is “which service made that decision, what operational value did it create, and was the choice still valid?”

Grafana can provide the shared view needed to make those decisions visible across Prometheus, Mimir, Loki and Tempo, provided the platform is governed as part of the production system rather than treated as an afterthought.