Why correlation fails when every field becomes a label
At 18:07, a payment API began returning intermittent 502 responses. The first alert pointed at ingress. Five minutes later, application logs implicated Redis. A dashboard showed rising latency in Johannesburg, while the database looked healthy in Cape Town.…
Advanced Log Intelligence and Correlation Systems: From Noisy Alerts to Defensible Incident Evidence
At 18:07, a payment API began returning intermittent 502 responses. The first alert pointed at ingress. Five minutes later, application logs implicated Redis. A dashboard showed rising latency in Johannesburg, while the database looked healthy in Cape Town. The actual fault was a failing network path between an on-premises service and a cloud workload in eu-west. The evidence existed, but it was scattered across logs, metrics, traces and infrastructure events.
This is the operational problem that Advanced Log Intelligence and Correlation Systems must solve: not merely collecting more events, but connecting the right events quickly enough to explain customer impact. For South African platforms, that means accounting for hybrid estates, regional latency, load-shedding, constrained links and strict control over personal information.
Why correlation fails when every field becomes a label
Loki is deliberately different from a conventional full-text search engine. It indexes stream labels and stores log content cheaply, so the label strategy determines both query performance and cost. Putting request IDs, user IDs or trace IDs into labels creates a new stream for every value. That can produce a spectacularly expensive cardinality problem without making investigation meaningfully faster.
Keep stream labels stable and operationally useful:
service_name,namespace,clusterandenvironment- Deployment or region where the value changes infrequently
- Severity only when there is a clear operational reason to index it
Keep high-cardinality values in the log body, parsed fields or structured metadata. Loki’s current OpenTelemetry path is designed for this model: OTLP attributes that are not promoted to index labels can remain structured metadata, avoiding an index explosion while retaining queryable context.[1]
The distinction matters in a cost-sensitive environment. A platform team paying cloud storage and egress in ZAR should not index every customer reference merely because a developer finds it convenient. The better question is: “Will this field narrow most investigations?” If not, leave it out of the stream selector.
Build one event context across Loki, Tempo and Prometheus
Correlation starts before an incident. Every request crossing a service boundary should carry a trace ID, and every service should emit that ID consistently. A useful log event includes:
- Timestamp in UTC, with the local timezone available in presentation layers
- Service, version, environment and deployment region
- Trace ID and span ID
- HTTP method, route template and status code
- Duration, upstream dependency and retry count
- A privacy-safe tenant or customer reference where needed
Do not log raw identity numbers, access tokens or full request bodies. POPIA compliance is not achieved by hiding a field in a dashboard; it requires controlling collection, retention, access and downstream copies. Redaction belongs in the collector pipeline, before data is shipped across a region or stored in a central tenant.
Grafana provides the navigation layer between these signals. A trace can lead to matching Loki entries, while an error log can link back to its Tempo trace and the Prometheus time series showing saturation. Grafana’s correlation features support links from traces to logs, metrics, profiles and external systems, with high-cardinality values such as trace IDs better handled as pipeline filters or structured metadata rather than stream labels.[2]
A practical OpenTelemetry Collector pipeline might look like this:
processors:
attributes/redact:
actions:
- key: http.request.header.authorization
action: delete
- key: user.email
action: hash
- key: user.id
action: delete
batch:
timeout: 5s
send_batch_size: 1024
service:
pipelines:
logs:
receivers: [otlp]
processors: [attributes/redact, batch]
exporters: [otlphttp/loki]
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/tempo]The exact exporter configuration varies by deployment, but the design principle is stable: normalise and remove sensitive data at the edge, then preserve correlation identifiers consistently.
Use metrics to find the failure window, not to replace the evidence
Prometheus and Mimir are excellent at answering “when did this get worse?” Logs and traces answer “which requests and dependencies were involved?” Combining the two avoids the common mistake of opening Loki first and searching an entire day of traffic.
For example, an API error-rate panel can identify a five-minute investigation window:
sum(rate(http_server_requests_total{
service="payments",
status=~"5.."
}[5m]))
/
sum(rate(http_server_requests_total{
service="payments"
}[5m]))From there, a derived Loki query can focus on the affected service and interval:
{service_name="payments", environment="prod"}
| json
| status >= 500
| trace_id != ""Prometheus-compatible Mimir is particularly useful when metrics must be retained across clusters and regions. Keep metric labels bounded: route templates are safer than raw URLs, and outcome classes are safer than exception messages. Native histograms can retain higher-resolution latency information without relying solely on large sets of classic buckets; Mimir supports storing and visualising Prometheus native histograms.[3]
That resolution helps distinguish a genuine application regression from a network tail-latency problem. A median may remain healthy while the 99th percentile collapses during congestion between Johannesburg and eu-west. Correlation becomes much more convincing when a trace shows the same remote call consuming the tail of the request budget.
Design for outages that do not look like outages
Load-shedding creates a particularly awkward observability problem. A node can disappear because its facility lost power, because a UPS exhausted its runtime, or because a site-level network device rebooted. Treating every missing scrape as an application failure produces noisy pages at exactly the time engineers need signal.
Use separate alerts for:
- Service failure with healthy infrastructure power and network telemetry
- Infrastructure unavailability affecting multiple services at one site
- Observability pipeline failure, where logs or metrics stop arriving
- Remote-region degradation, especially for dependencies in eu-west
Alert on customer symptoms first, then attach likely causes. A useful incident dashboard should show request success rate, latency by region, scrape health, log ingestion delay, trace sampling health and recent deployment markers together. Mimir can provide the regional time series, Loki the deployment and error evidence, and Tempo the cross-service path.
Do not silently interpret “no logs” as “no errors”. During a last-mile outage, the application may be failing while the collector queue is merely unable to export. Monitor collector queue depth, dropped records and exporter retry counts. The observability system requires its own service-level objectives.
Turn correlation into an investigation workflow
The most effective teams standardise the path from alert to evidence rather than relying on an engineer’s memory. A sensible workflow is:
- Start with the customer-facing symptom and establish the affected regions, services and time window.
- Use Prometheus or Mimir to compare error rate, latency and traffic against the previous deployment or normal period.
- Open a representative Tempo trace and inspect the slowest span, retries and remote calls.
- Follow the trace ID into Loki, filtering high-cardinality fields after selecting low-cardinality streams.
- Compare matching events across availability zones, clusters and network paths.
- Record the confirmed causal chain, not just the loudest error message.
For example, “Redis timeout” may be a symptom rather than the root cause. If traces show the timeout only for requests routed through one cluster, and metrics show packet loss on that cluster’s link, the remediation belongs in networking. Log intelligence is valuable precisely because it prevents the team from stopping at the first plausible exception.
Make the system cheaper, safer and more useful over time
Review the correlation design after every serious incident. Remove labels that were not useful, promote fields that repeatedly narrowed investigations, and shorten retention for verbose debug streams. Loki’s native OTLP ingestion, introduced in the Loki 3.x era, reduces the need for a separate Loki exporter and aligns log ingestion more closely with OpenTelemetry conventions.[4]
Keep longer retention for aggregated metrics and selected traces, while retaining raw logs according to operational and legal requirements. Sampling should preserve errors, high-latency requests and representative traffic from each region rather than randomly discarding the very requests needed to explain an incident.
The goal of Advanced Log Intelligence and Correlation Systems is not an impressive wall of dashboards. It is a short, defensible path from “customers in Gauteng are seeing failures” to “these requests crossed this dependency, acquired this latency, generated these errors and were affected by this infrastructure condition”. With disciplined labels, consistent OpenTelemetry context and deliberate Grafana workflows, Loki, Prometheus, Tempo and Mimir become one investigative system instead of four disconnected products.