Start with a telemetry budget, not a surprise invoice

Observability Cost Governance Strategies: Stop Paying for Telemetry Nobody Uses

The bill arrived after a harmless-looking configuration change: a Kubernetes team reduced a Prometheus scrape interval, added pod and request labels, and enabled debug logging for a release. Nothing failed. Dashboards remained green. Yet the month-end invoice was several thousand rand higher because the estate was collecting millions of additional samples and log lines that nobody queried.

That is the central problem behind Observability Cost Governance Strategies. Cost control is not about deleting telemetry until incidents become invisible. It is about deciding which signals deserve high resolution, which need short retention, which can be sampled, and which should never leave the workload.

Start with a telemetry budget, not a surprise invoice

Observability spend should be treated like a reliability budget. Assign ownership and limits before teams deploy collectors, exporters and new dashboards. A useful starting point is a monthly budget split across metrics, logs, traces and profiles, with separate allocations for production, non-production and shared platform services.

In a South African environment, express the budget in both the billing currency and ZAR. Exchange-rate movement can make a flat US-dollar allowance feel very different to a finance team paying local invoices. Track ingestion, retained volume, active series, query load and egress—not only the provider’s final number.

• Allocate costs to a product, team, environment and region.
• Set a warning threshold at 70–80% of the monthly allowance.
• Require an owner for every new exporter, log pipeline and high-cardinality metric.
• Review cost per service alongside error rate, latency and availability.
• Record exceptions, such as a temporary incident investigation or migration.

Grafana dashboards make this practical by putting usage and service health in the same operational view. A team can see that a new request label increased active series at the same time that it improved a latency investigation—or that it added cost without changing a single alert.

Recent Grafana guidance describes usage attribution, billing dashboards, usage alerts and cardinality analysis as core cost-management capabilities rather than optional finance reporting.[1] That distinction matters: engineers need feedback while changing telemetry, not six weeks later.

Put cardinality controls at the collection edge

Prometheus and Mimir costs are usually driven less by the number of metric names than by the number of active time series. Labels such as pod, container_id, request_id, full URLs and unbounded user input can create a series for every short-lived value.

Keep labels bounded and useful. Prefer route="/payments/{id}" to a raw path containing every transaction identifier. Use status classes or controlled error codes rather than exception messages. Do not put customer references, email addresses or trace IDs into metric labels.

Enforce the rule in code review and in the collector. For example, a Prometheus scrape configuration can drop known-dangerous labels before remote write:

scrape_configs:
  - job_name: payments
    kubernetes_sd_configs:
      - role: pod
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'http_request_duration_seconds_bucket'
        action: keep
      - source_labels: [request_id, user_id, session_id]
        action: labeldrop

This is not a licence to drop indiscriminately. Before removing a label, check whether it powers an alert, recording rule, SLO or incident workflow. If a label is needed only for a dashboard, consider aggregating it into a recording rule and retaining the lower-cardinality result in Mimir.

Scrape frequency deserves the same scrutiny. A 15-second interval is sensible for a critical payment path, but excessive for a rarely changing certificate-expiry metric. Grafana has noted that moving from six data points per minute to one can reduce that metric’s volume by a factor of six, where the lower resolution still answers the operational question.[2]

Make Loki retention reflect investigative value

Logs are where cost governance often fails first. Developers enable debug output during a production incident, the setting remains in a Helm values file, and Loki receives a permanent stream of stack traces, health checks and successful request messages.

Filter at the edge whenever possible. Dropping a log before it reaches Loki saves network, indexing and storage costs. Keep structured fields that support investigations, but avoid indexing every field as a Loki label. A label such as namespace or service is generally useful; a label containing trace_id, customer ID or URL is usually expensive and difficult to aggregate.

• Retain security and audit events for the period required by policy.
• Keep application errors and sampled slow requests longer than routine access logs.
• Send repetitive health checks and successful polling messages to short retention—or discard them.
• Use structured JSON so engineers can query fields without turning every field into an index label.
• Make debug logging an expiring change with an automatic rollback.

For POPIA-sensitive workloads, redact personal information before logs cross a trust boundary. A telemetry platform hosted outside the required jurisdiction may create a data-residency and processor-management problem even when the invoice looks attractive. Keep sensitive streams in an approved South African or organisational environment, and export only the minimum diagnostic data needed for a central view.

Grafana’s 2024 Adaptive Logs announcement described sampling recommendations based on how frequently log patterns are queried, illustrating a useful principle: retention should follow investigative value rather than habit.[3] The same principle can be implemented with open-source pipelines and explicit Loki retention policies.

Sample traces for signal, not convenience

Tempo makes distributed tracing valuable across services, but storing every trace from every request is rarely justified. A sensible policy keeps all traces with errors, high latency or unusual attributes, while sampling the healthy baseline.

Tail sampling is preferable when the decision depends on the completed trace. For example, retain 100% of traces containing a server error, traces exceeding a two-second threshold, and traces from a critical payment operation; retain a smaller percentage of ordinary successful requests.

Do not sample away the evidence needed for load-shedding and connectivity investigations. A mobile banking service may need more traces for requests arriving through a degraded last-mile network, while a stable internal batch job can use a much lower rate. Sampling rules should reflect business risk, not merely traffic volume.

Keep trace attributes bounded. A raw SQL statement, request body or user-generated search term can make spans both expensive and unsafe. Record operation names, service boundaries and controlled outcome fields; attach detailed payloads only under an approved, short-lived diagnostic mode.

Design for load-shedding, regional latency and outages

South African systems often operate across a hybrid estate: workloads in local data centres or cloud regions, users on variable last-mile links, and dependencies in eu-west. During load-shedding or a WAN fault, observability itself must degrade predictably rather than competing with customer traffic.

Use local buffering and bounded queues in Grafana Alloy or OpenTelemetry collectors. Prioritise alerts, service-level metrics and error traces over verbose logs. Apply backpressure and drop low-value telemetry when disk, CPU or network limits are reached. Never allow an observability exporter to exhaust resources required by the application.

Measure the cost of cross-region transport explicitly. A local Prometheus or Mimir gateway can aggregate data before sending it to a central system. A regional Loki deployment can retain routine logs locally while forwarding only security events and high-value incidents. Tempo collectors can make sampling decisions close to the workload, avoiding unnecessary egress.

Alert on telemetry pipeline health as well as application health. Useful signals include remote-write queue depth, dropped samples, collector memory, Loki ingestion errors and Tempo export latency. During an outage, an honest “telemetry degraded” indicator is safer than dashboards that appear normal because the data path has failed.

Turn governance into an engineering control loop

Monthly cost reviews are too slow for modern deployment frequency. Add telemetry checks to pull requests and CI pipelines. Reject new metrics with unbounded labels, flag scrape intervals below an agreed threshold, and require a retention decision for new log streams. A small catalogue should record each signal’s owner, purpose, resolution, retention and downstream consumers.

Run a quarterly “unused telemetry” exercise. Identify metrics never used in dashboards, alerts or recording rules; log patterns never queried; and traces retained beyond their investigative window. Remove them in stages, starting with non-production. Keep a rollback path, because an apparently unused signal may become important during a rare failure.

For teams using managed Grafana services, cost attribution and usage alerts can shorten this feedback loop. For self-managed Prometheus, Loki, Tempo and Mimir, the same discipline can be built with usage dashboards, object-storage reports and Git-managed policy. The platform choice changes the mechanics, not the governance principle.

The strongest Observability Cost Governance Strategies make the economical path the safe path: bounded labels by default, sampled healthy traffic, short-lived debug modes, local aggregation, explicit POPIA controls and budgets visible to the people changing production. Spend then becomes an operational signal—one that reveals waste before it becomes an incident.