Correlation starts with evidence, not an LLM

AI-Powered Incident Correlation Frameworks for South African SRE Teams

At 02:17, a payment API started timing out in Johannesburg. Prometheus reported rising latency, Loki showed a burst of connection errors, and Tempo traced the slow requests to a service calling a database in Cape Town. Five minutes later, the alert stream had grown to 46 notifications. The actual trigger was simpler: a site-to-site link failed over during a load-shedding event, sending traffic across a congested route.

This is the problem AI-Powered Incident Correlation Frameworks should solve. Not “put a chatbot in front of Grafana”, but connect metrics, logs, traces, topology, deployments and operational context well enough to reduce dozens of symptoms to one defensible incident hypothesis.

Correlation starts with evidence, not an LLM

Artificial intelligence cannot reliably correlate telemetry that has no shared identity. Before introducing a model, standardise the fields that allow signals to be joined:

• service.name, environment and namespace
• Cluster, node, availability zone and region
• Release version, deployment identifier and Git commit
• Trace ID and request ID
• Customer-facing journey or business transaction
• Owner, escalation policy and dependency

Prometheus and Mimir should carry stable labels without turning every user ID, URL or error message into a high-cardinality time bomb. Loki should use structured labels for fields used during filtering, while keeping volatile values in the log body. Tempo should preserve trace context across HTTP, messaging and database calls.

A useful rule is that every investigation must be able to move in both directions: from an alert to its traces and logs, and from a suspicious trace to the surrounding infrastructure metrics. Grafana provides the practical investigation surface for those pivots, with dashboards, alerting, Explore and links between telemetry sources. OpenTelemetry’s 2025 work on AI-agent observability also reflects the wider move towards common semantic conventions for new types of workloads.[3][9]

A practical signal path for Prometheus, Loki, Tempo and Mimir

Consider a hybrid retailer running workloads in a Johannesburg data centre, AWS Cape Town and a European region. Mimir stores long-term Prometheus metrics, Loki receives application and platform logs, and Tempo stores distributed traces. Grafana presents the operational view; the correlation service consumes selected evidence through controlled queries and APIs.

The framework should not continuously send the whole observability estate to a model. That is expensive, slow and difficult to govern. Instead, an alert opens a bounded investigation window:

1. Identify the affected service, region and alert fingerprint.
2. Query baseline and current values for latency, error rate, saturation and traffic.
3. Find related alerts within a defined time range.
4. Search Loki for matching error patterns and deployment changes.
5. Inspect Tempo spans for the slowest or most frequently failing dependency.
6. Compare the result with topology, maintenance windows and recent changes.
7. Return ranked hypotheses with supporting evidence and confidence.

A PromQL query for a basic service-level symptom might look like this:

histogram_quantile(
  0.95,
  sum by (le, service, region) (
    rate(http_request_duration_seconds_bucket{
      namespace="payments",
      service="checkout"
    }[5m])
  )
)

The model should receive the result, not blindly improvise the query. Query templates, time limits and maximum series counts protect Mimir from an enthusiastic investigator generating a regional denial of service.

For logs, the equivalent Loki search might be intentionally narrow:

{namespace="payments", service="checkout", region="jhb"}
|= "upstream timeout"
| json
| line_format "{{.timestamp}} {{.dependency}} {{.error}}"

The output needs provenance. “The database is broken” is not an incident finding. “Checkout p95 rose from 420 ms to 3.8 s after route failover; 71% of slow traces contain calls to the Cape Town inventory service; no application deployment occurred in the preceding hour” is evidence an engineer can challenge.

Designing correlation for South African failure modes

Global examples often assume stable connectivity and a single cloud region. Local estates have different edges.

Load-shedding and power events

Power interruptions can restart network equipment, reduce site capacity or trigger failover. Treat facility and network events as first-class telemetry. Export UPS state, generator mode, battery runtime and link changes as metrics. Add event annotations to Grafana dashboards so an AI system can distinguish “application regression” from “the branch router rebooted at 02:16”.

Latency to Europe

Some organisations route specialised services to eu-west. A correlation engine should compare Johannesburg-to-Europe latency with local service latency, packet loss and queue depth. If only the cross-region dependency deteriorates, the probable cause is very different from a CPU saturation event in Cape Town.

Last-mile instability

Mobile, branch and rural connections can produce intermittent failures rather than a clean outage. Track retries, disconnects, DNS failures and synthetic transactions by access network. A national average may look healthy while one provider or province experiences severe degradation.

ZAR cloud-cost pressure

Correlation should include cost signals. An automatic recommendation to increase retention, replicas or sampling may restore reliability while creating an unacceptable bill. Mimir retention, Loki ingestion and Tempo trace sampling should be visible alongside usage and budget metrics. Costs should be normalised in rand for the people approving the change, even when the underlying provider reports in US dollars.

Use AI to rank hypotheses, not to declare root cause

The most useful pattern is a hypothesis-ranking pipeline. Rules and statistical detection identify anomalies; the model explains relationships and proposes the next check.

• Symptom: checkout error rate is above its service-level objective.
• Temporal link: failures began 90 seconds after a network route change.
• Trace link: failed requests share a dependency span.
• Log link: the dependency reports connection resets.
• Change link: no checkout deployment occurred, but a network policy changed.
• Next action: compare route health and revert the policy only if the change is confirmed.

This approach limits hallucination because the system must attach each claim to a query result, dashboard annotation, trace group or change record. It should also expose uncertainty. A 0.82 confidence score is meaningless unless the team knows whether it represents calibrated historical accuracy or merely the model’s language preference.

Grafana’s 2025 announcements show where the market is moving: context-aware assistants and investigation workflows are being positioned to explore metrics, logs, traces and profiles, generate findings and suggest remediation.[4][5] That is valuable, but autonomous remediation should remain behind explicit approvals until the evidence chain and rollback path are trustworthy.

POPIA changes what enters the correlation context

Incident data frequently contains personal information: email addresses, phone numbers, account identifiers, IP addresses and request payloads. Sending raw logs or traces to an external AI service can create a compliance and data-sovereignty problem. POPIA regulates cross-border transfers of personal information and requires an appropriate legal and protection basis for those transfers.[1][7]

Build privacy controls into the telemetry path:

• Redact tokens, payment details and personal identifiers before ingestion.
• Hash identifiers only when correlation needs continuity, not visibility.
• Use allow-listed fields for AI investigation context.
• Keep raw Loki and Tempo data in approved regions where required.
• Record which queries, evidence and model outputs were used.
• Apply retention limits to prompts, responses and incident transcripts.

A strong design keeps the model’s context small and operational: label sets, aggregated metrics, exception fingerprints, trace metadata and selected log lines. Engineers can retrieve sensitive detail through controlled access when necessary, rather than making it part of every automated request.

Measure whether the framework actually helps

Alert volume is not enough. Measure the quality of investigation:

• Time from first alert to a validated hypothesis
• Percentage of incidents with a correct top-ranked hypothesis
• Number of telemetry pivots required before diagnosis
• False-correlation rate
• Queries that exceed cost or latency budgets
• Suggested actions rejected by responders
• Change failure rate after AI-assisted remediation

Run the system in advisory mode first. Replay resolved incidents, compare its ranking with the post-incident diagnosis, and deliberately test missing telemetry, clock skew, duplicate alerts and misleading symptoms. If it cannot explain why it believes two signals are related, it is not ready to operate unattended.

The winning AI-Powered Incident Correlation Frameworks will not be the ones with the most impressive chat interface. They will be the ones that preserve telemetry quality, respect POPIA, understand regional failure modes and show their working. In a mixed Johannesburg, Cape Town and European estate, that evidence-first discipline is what turns an alert storm into a manageable incident.