Why a service map is not yet dependency visibility

End-to-End Service Dependency Visibility Models: From Blind Spots to Actionable Service Maps

The payment API looked healthy: CPU was normal, Prometheus showed no obvious saturation, and the checkout dashboard stayed mostly green. Customers in Johannesburg still experienced failed transactions because the API was waiting on a database connection in Cape Town, which was waiting on an identity service hosted in eu-west. The dependency was real, but no single dashboard showed it.

That is the problem solved by End-to-End Service Dependency Visibility Models. A useful model does more than draw lines between services. It explains which dependencies exist, how often they are used, where latency accumulates, what happens during a regional outage, and which team owns the resulting risk.

Why a service map is not yet dependency visibility

A topology graph is only a starting point. Kubernetes labels can show that a workload exists, while a trace can show that a request actually travelled through checkout, fraud scoring, Redis, PostgreSQL and an external payment gateway. Those are different kinds of truth.

Four questions separate a decorative map from an operational model:

• Reachability: Can service A call service B?
• Usage: Does service A actually call service B in production?
• Criticality: Does failure of B block a customer journey, or merely remove an optional feature?
• Change impact: Which services, dashboards and SLOs are affected when B changes?

Tempo’s Service Graph uses trace-derived relationships and can display request rate, error rate and duration between services.[5] That makes it considerably more valuable than an inventory generated only from deployment manifests. Grafana can then provide the operational entry point: a team can move from a red edge to the related trace, logs in Loki and metrics in Prometheus or Mimir.

The distinction matters in a hybrid estate. A service running on-premises may call a cloud queue through a private link, while a monitoring agent exports telemetry through a separate egress path. Treating both as ordinary “dependencies” hides the fact that one affects application availability and the other affects observability itself.

Five models for representing the same estate

1. The inventory model

This is the familiar catalogue of applications, hosts, namespaces, databases and queues. It is cheap to create and useful for ownership, but it describes what should exist rather than what is happening. CMDB data, Kubernetes objects and Terraform state are good inputs.

Use this model for:

• Finding unowned services and unsupported versions.
• Recording data classification and POPIA-related handling requirements.
• Showing deployment regions, environments and recovery tiers.

Do not use it as the only source for incident diagnosis. Inventory graphs commonly retain stale edges and miss undocumented calls made by libraries, feature flags or scheduled jobs.

2. The traffic model

The traffic model is built from observed calls. Distributed traces provide the strongest evidence because they retain parent-child relationships, duration and status. Metrics generated from spans can provide a lower-cost aggregate view, while service-mesh telemetry or eBPF can cover workloads that are difficult to instrument.

For teams adopting OpenTelemetry, standard resource attributes such as service name, namespace, deployment environment and cloud region are more important than producing a visually impressive graph. If one service reports checkout-api and another reports checkout, the graph will split one dependency into two fictional systems.

Prometheus remains effective for RED metrics—request rate, errors and duration—while Mimir provides a scalable long-term metrics backend. Loki adds the event detail needed when a trace points to a failed retry, a rejected certificate or a rate-limit response. Tempo links these signals without requiring every investigation to begin with a log search.

3. The critical-path model

Not every edge deserves the same alert. A recommendation service might be slow without preventing checkout; an authorisation service might add only 80 milliseconds but stop every transaction when unavailable.

Assign each dependency a role:

• Blocking: The request cannot complete without it.
• Degrading: The request completes with reduced functionality.
• Asynchronous: The dependency affects later processing rather than the user response.
• Observability-only: Failure affects telemetry delivery, not the product path.

Represent that role as metadata rather than relying on a person to infer it from line thickness. A blocking dependency should influence the parent service’s SLO and incident priority. An asynchronous dependency may need queue-age alerting instead of request-latency alerting.

4. The failure-domain model

South African environments make failure domains especially practical. Load-shedding can affect an office, an edge site or a data centre even when the cloud region remains healthy. A last-mile provider can introduce packet loss for customers while internal synthetic checks remain green. A dependency in eu-west can be reachable but slow enough to break a Johannesburg user journey.

Model each edge with location and transport attributes:

• Source and destination region or availability zone.
• Network path, such as MPLS, VPN, internet or private interconnect.
• Expected latency budget.
• Fallback behaviour and timeout.
• Data residency or cross-border processing constraints.

For example, an authentication call from Cape Town to eu-west should not be compared with an in-cluster Redis call. The map should make the geographic penalty visible and should expose whether a local token-validation path exists.

5. The ownership and change model

A dependency is operationally useful only when someone can act on it. Attach a service owner, escalation route, repository, runbook and SLO to each node. Attach a consumer owner to each edge where possible. The database team may own PostgreSQL, but the checkout team owns the consequence of a slow query during a sale.

This model also supports change review. Before migrating a queue or rotating certificates, query consumers by service name and environment. A graph that cannot answer “who will break if this endpoint changes?” is incomplete, regardless of how polished it looks.

Build the model from correlated telemetry

Start with context propagation. HTTP, gRPC, messaging and database spans should carry trace identifiers consistently. Where source changes are difficult, eBPF-based instrumentation can help discover network relationships and provide initial coverage. Recent observability practice increasingly combines OpenTelemetry with eBPF and profiling to connect application behaviour with system-level evidence.[1]

Use an OpenTelemetry Collector distribution such as Grafana Alloy to normalise attributes, sample traces and route telemetry. Keep high-value traces—errors, slow requests and selected business journeys—while reducing routine volume. Sampling is not merely a storage decision: aggressive head sampling can erase rare dependency failures before they reach Tempo.

A basic Prometheus query can expose edges whose error rate is rising, provided span-derived metrics use stable labels:

sum by (client, server) (
  rate(traces_service_graph_request_failed_total[5m])
)
/
sum by (client, server) (
  rate(traces_service_graph_request_total[5m])
) > 0.02

Keep cardinality under control. Never add customer ID, full URL, request ID or arbitrary exception text to dependency labels. Put those values in logs or trace attributes, where they can be searched without multiplying every time series.

For each important edge, create a drill-down path:

1. Metric identifies the affected client and server pair.
2. Trace shows the critical span, retries and timeout location.
3. Loki reveals the corresponding application or infrastructure event.
4. Deployment metadata identifies the release, region and owner.

Make regional and cost constraints visible

Cross-region calls often survive functional testing and fail under real latency. Set separate budgets for local and remote dependencies. A 300-millisecond eu-west call might be acceptable for a back-office export but unacceptable inside an interactive payment request.

Cloud cost also belongs in the model. Long retention in Mimir, high-volume logs in Loki and unsampled traces in Tempo can create a material ZAR-denominated bill. Record telemetry volume per service and use that information during ownership reviews. The cheapest dependency graph is not the one with the least telemetry; it is the one that preserves enough evidence to explain expensive failures.

POPIA adds another dimension. Trace attributes can accidentally contain email addresses, identity numbers or payment details. Redact sensitive fields at instrumentation or collector pipelines, and keep tenant, region and retention metadata explicit. A service map should help answer where data travels—not quietly become another place where personal information is copied.

Alert on broken journeys, not colourful graphs

Dependency visibility becomes operational when it drives decisions. Alert on symptoms at the business or service boundary, then use the graph to identify likely causes. A sensible alert might combine checkout error rate, dependency edge failure rate and the affected region:

(
  sum(rate(http_server_request_duration_seconds_count{
    service_name="checkout-api",
    http_response_status_code=~"5.."
  }[5m]))
  /
  sum(rate(http_server_request_duration_seconds_count{
    service_name="checkout-api"
  }[5m]))
) > 0.03

During load-shedding, avoid paging on every exporter outage. Separate application availability from telemetry availability, and use longer evaluation windows for power-sensitive sites where agents may restart as UPS systems transfer load. For spotty last-mile connectivity, compare regional synthetics with internal service metrics before declaring a platform fault.

The strongest End-to-End Service Dependency Visibility Models combine inventory, observed traffic, critical paths, failure domains and ownership. Start with one customer journey—payments, policy quotes or mobile sign-in—then prove that an engineer can move from symptom to dependency to evidence within one incident. Expand only after that path is trustworthy.

That approach produces fewer diagrams and better decisions: which call needs a timeout, which region needs a fallback, which team owns the risk, and which telemetry is worth retaining when the next outage begins.