Why a service list is not a dependency model

End-to-End Service Dependency Visibility Models: From Blind Spots to Actionable Graphs

The payment API was healthy, the database dashboard was green, and the Kubernetes nodes had plenty of capacity. Yet customers in Gauteng were receiving intermittent checkout failures. The missing clue sat three hops away: a shipping-rate call routed from Johannesburg to a service in eu-west, where a brief latency increase pushed the payment workflow beyond its timeout.

This is the problem that End-to-End Service Dependency Visibility Models must solve. A dependency map is useful only when it connects architecture to live behaviour: which service called which dependency, across which region, with what latency, error rate and business impact.

Why a service list is not a dependency model

A catalogue of Kubernetes Deployments, VM names and cloud resources describes ownership. It does not describe runtime dependencies. A service may depend on Redis, Kafka, an external identity provider, a DNS resolver, a load balancer, a payment gateway and a third-party fraud API without declaring all of those relationships in infrastructure code.

Static maps also miss conditional paths. The checkout service might call the local stock service for most orders, but fall back to a European inventory endpoint when a warehouse is unavailable. During an incident, that fallback path is often the path that matters.

A practical model needs at least four dimensions:

• Topology: which components communicate.
• Traffic: request volume, direction and protocol.
• Health: errors, saturation, latency and availability.
• Context: region, environment, tenant, release and data sensitivity.

OpenTelemetry is increasingly used as the common instrumentation layer across languages and telemetry types. Grafana Labs’ 2025 observability survey reported Prometheus production use at 67%, while 41% of respondents reported OpenTelemetry in production; the latter also showed strong momentum among teams still evaluating it.[7] That combination suits hybrid estates: existing Prometheus metrics can remain in place while traces gradually expose the real request paths.

Model one: the static ownership map

The first model is a manually maintained service catalogue. It normally comes from Helm values, Terraform, Backstage metadata or a CMDB. Nodes represent services and edges represent intended relationships.

This model is cheap and valuable for ownership. It answers questions such as “who owns the customer-profile API?” and “which team supports the Kafka cluster?” It also supports change review: a proposed network policy or database migration can be assessed against declared consumers.

Its weakness is drift. A new feature flag, emergency integration or sidecar can create a dependency without updating the catalogue. Static maps should therefore be treated as the control-plane view, not the source of truth for runtime behaviour.

Model two: telemetry-derived service graphs

The stronger operational model derives edges from traces. Each request carries a trace ID through the application, gateway and downstream services. Tempo can inspect parent-child span relationships and generate Prometheus-compatible service-graph metrics such as request totals, failed requests and server-side duration.[3][4]

The resulting graph can be rendered in Grafana, with Prometheus or Mimir storing the generated metrics and Tempo retaining the detailed traces. This is more than a visual exercise: the edge between checkout-api and fraud-service can show request rate, failure ratio and latency, while a selected edge can lead directly to representative traces.

A typical Tempo metrics-generator configuration looks like this:

metrics_generator:
  storage:
    path: /var/tempo/generator/wal
  remote_write:
    - url: http://mimir-gateway/api/v1/push
      send_exemplars: true

overrides:
  defaults:
    metrics_generator:
      processors:
        - span-metrics
        - service-graphs

The exact deployment differs between Tempo, Grafana Alloy and managed platforms, but the design is consistent: traces provide relationship evidence; the metrics backend provides efficient aggregation; the dashboard provides navigation.

Exemplars close the gap between a high-level alert and an individual request. A latency point in a Prometheus or Mimir panel can carry a trace ID, allowing an engineer to jump from a RED metric to the corresponding Tempo trace.[5] Without that link, the graph says that an edge is slow. With it, the trace can reveal a DNS delay, a queue wait, a TLS handshake or a remote database call.

Model three: critical-path and business-transaction graphs

Not every dependency deserves equal visual weight. A service graph containing every exporter, cache and Kubernetes control-plane call quickly becomes unreadable. The useful question is not “what calls what?” but “which dependencies can prevent this transaction from completing?”

Build a second graph around business transactions:

1. Define journeys such as login, checkout, claims submission or airtime purchase.
2. Identify the entry span and mark the customer-visible outcome.
3. Calculate the critical path from trace timing, not from architecture diagrams.
4. Classify dependencies as blocking, optional, asynchronous or fallback.
5. Assign an owner and service-level objective to each blocking edge.

This model handles graceful degradation properly. A recommendation service may be slow without blocking checkout, while an identity provider with the same latency can make the entire journey fail. Both appear in the topology; only one belongs on the checkout critical path.

For South African platforms, add geography to the edge. A call from Cape Town to an on-premises Johannesburg cluster, or from a local application to eu-west, should expose round-trip latency and failure behaviour separately. A single global average can hide a last-mile problem affecting customers on mobile networks or a regional routing issue affecting only one ISP.

Model four: dependency health and failure-domain overlays

Topology alone does not explain correlated failures. Add overlays for failure domains: availability zone, data centre, cloud provider, region, connectivity provider and power dependency.

Load-shedding-aware operations are a good example. A UPS-backed data centre may remain online while a branch network, access circuit or local edge device fails. If the dependency graph shows only “API calls database”, it hides the shared power and network assumptions underneath. Tag spans and metrics with dimensions such as cloud.region, deployment.environment, site and network.provider, while keeping cardinality under control.

Use PromQL to identify a failing edge rather than alerting on every node:

sum by (client, server) (
  rate(traces_service_graph_request_failed_total[5m])
)
/
sum by (client, server) (
  rate(traces_service_graph_request_total[5m])
)
> 0.02

Pair that ratio with request volume. A two-percent failure rate on three requests is not equivalent to two percent of a high-volume payment edge. Alert labels should include environment, region and the owning team, but avoid putting customer IDs, full URLs or unbounded error text into metric labels.

Keeping the graph useful under cost and compliance pressure

Retaining every trace at full fidelity is rarely sensible when cloud spend is measured in ZAR and traffic spikes during month-end or load-shedding recovery. Sample normal traffic, retain error traces and preserve complete traces for a controlled set of critical transactions. Keep service-graph metrics longer than raw traces because they are compact and useful for trend analysis.

Mimir provides scalable Prometheus-compatible storage for long-lived metrics, while Loki can correlate application logs with trace IDs. Store only the labels needed for investigation. A log line containing a trace ID, service name, region and deployment version is usually more useful than one containing a copied request payload.

POPIA adds another design constraint. Trace attributes and logs can accidentally capture identity numbers, phone numbers, addresses or payment details. Redact at instrumentation or collection time, restrict tenant access, and decide where telemetry is allowed to reside before choosing a hosted backend. Data sovereignty is not solved by drawing a South African region on a dashboard; it requires documented retention, access and transfer controls.

A model that works during the next incident

The best implementation combines the four views rather than choosing one:

• Use a static catalogue for ownership and intended architecture.
• Use Tempo-derived graphs for observed runtime relationships.
• Use business-transaction graphs for customer impact and critical paths.
• Use failure-domain overlays for regional, network, power and provider correlations.

Test the model with controlled failure: add latency to the eu-west call, block a DNS path, stop a Kafka consumer, or simulate a Mimir write outage. The graph should show the affected edge, the critical transaction should show its degraded outcome, and the alert should link to a trace that explains the failure.

That is the standard for End-to-End Service Dependency Visibility Models: not a polished node diagram, but an evidence-backed view that tells an on-call engineer what changed, who is affected, where the dependency lives and which action is safe to take next.