Why a single service map fails in a hybrid African estate

At 14:07, the checkout API looked healthy: CPU was below 40%, pods were ready, and the error-rate alert had not fired. Customers in Johannesburg still saw payment timeouts. The missing dependency was a fraud-scoring service in eu-west, reached…

Why a single service map fails in a hybrid African estate

End-to-End Service Dependency Visibility Models: From Blind Spots to Actionable Service Maps

At 14:07, the checkout API looked healthy: CPU was below 40%, pods were ready, and the error-rate alert had not fired. Customers in Johannesburg still saw payment timeouts. The missing dependency was a fraud-scoring service in eu-west, reached through a congested international link. Nothing in the service dashboard showed that checkout depended on it.

That is the practical problem behind End-to-End Service Dependency Visibility Models. A service map is not merely a collection of boxes and arrows. It is a model of how requests, failures, latency and operational ownership move through a system. If the model omits queues, DNS, third-party APIs, database replicas or regional network paths, the resulting dashboard can be attractive and operationally useless.

Why a single service map fails in a hybrid African estate

Most organisations start with an infrastructure view: Kubernetes workloads, virtual machines, load balancers and databases. This is useful for capacity planning, but it does not answer the incident question: “Which dependency made this customer transaction slow?”

A second model usually comes from application traces. It shows the path of an instrumented request, such as web to orders to payments. That is far more useful, but it still has blind spots:

  • Uninstrumented legacy services running in an on-premises data centre.
  • Asynchronous Kafka or RabbitMQ flows where the downstream work happens minutes later.
  • External providers such as banks, identity services and SMS gateways.
  • Network and DNS failures that occur before an application span is created.
  • Last-mile problems affecting customers but not internal synthetic checks.
  • Power interruptions that restart edge equipment or reduce available capacity during load-shedding.

The answer is not to force every concern into one diagram. Use several complementary visibility models, then connect them through consistent service names, environment labels, trace IDs and ownership metadata.

Model one: the request-path graph

The request-path graph is the default model for customer-facing incidents. It represents synchronous calls and attaches RED signals to each edge: request rate, error rate and duration. Tempo can derive service-graph metrics from trace parent-child relationships and write those metrics to a Prometheus-compatible backend. Grafana then renders the relationships and allows engineers to move from an unhealthy edge to the traces behind it.[1]

For a South African retailer, a useful graph might show:

  • mobile-app to api-gateway.
  • api-gateway to cart-service in Cape Town.
  • cart-service to pricing-service in Johannesburg.
  • pricing-service to a tax provider in eu-west.
  • checkout-service to a local payment switch and an international fraud provider.

The important detail is that the international edge must remain visible even when the provider is outside the organisation’s control. A dependency labelled only “external API” hides both latency and commercial risk. Label it with the provider, region, route and business capability where policy permits.

Prometheus can alert on service-graph metrics rather than relying exclusively on application-level alerts:

sum by (client, server) (
  rate(traces_service_graph_request_failed_total{
    environment="production",
    client="checkout-service"
  }[5m])
)
/
sum by (client, server) (
  rate(traces_service_graph_request_total{
    environment="production",
    client="checkout-service"
  }[5m])
) > 0.05

This catches a failing dependency edge, but it should not be the only signal. A trace-derived graph depends on correct instrumentation and representative traffic. During a quiet period, the absence of an edge can mean “no requests happened”, not “the dependency disappeared”.

Model two: the transaction and message-flow graph

Synchronous maps are poor at representing work that crosses a queue. An order may be accepted successfully, placed on a topic, consumed by a warehouse service, enriched by an ERP adapter and eventually sent to a courier. The customer-facing request ends before most of that work begins.

Use a transaction graph for the business flow and a message-flow graph for the technical path. Correlate them with trace context in message headers, a business transaction identifier and durable event attributes. Avoid putting customer names, identity numbers or full payloads into telemetry. POPIA obligations still apply to logs and traces, and cross-border storage can introduce data-sovereignty concerns.

With Loki, log correlation should answer three questions quickly:

  • Which consumer processed the event?
  • How long did it wait before processing?
  • Was it retried, dead-lettered or rejected?

A practical Loki query might isolate failures for a single order flow:

{service_name="warehouse-consumer", environment="production"}
| json
| transaction_id="tx-8f31"
| level="error"

Do not confuse consumer health with business progress. A consumer can report a normal process rate while the queue age grows because producers have accelerated. Add queue depth, oldest-message age, consumer lag and dead-letter volume to the dependency model. These are dependency edges with time attached.

Model three: the resource and failure-domain graph

The resource graph answers a different question: what can fail together? It maps services to nodes, availability zones, clusters, circuits, storage systems and power domains. This matters when the application graph looks normal but a shared platform dependency is degrading.

For example, several workloads may run in separate Kubernetes namespaces but share one on-premises SAN. A single storage controller can therefore affect payments, reporting and identity at once. Likewise, two cloud regions may appear independent while sharing a transit provider or a single DNS service.

Prometheus and Mimir are well suited to long-term infrastructure and dependency metrics. Mimir provides a scalable Prometheus-compatible store for retaining historical series, while Grafana dashboards can compare error rates with node reboots, interface drops, storage latency and power events. The point is not to place every metric on one screen. It is to preserve enough history to distinguish a one-off pod issue from a recurring failure domain.

Load-shedding-aware alerting belongs in this model. During planned outages, alerts for node availability may be expected, but customer-impact signals must remain active. Suppress known infrastructure noise by maintenance window, not by disabling all alerts. Continue monitoring payment success, queue age, synthetic journeys and regional latency.

Model four: the user and network-path graph

Internal telemetry often says “healthy” while users on a particular ISP, mobile network or rural last-mile link experience failure. A user-path model adds vantage points and network conditions to the dependency chain:

  • Customer region or POP.
  • ISP or mobile network.
  • CDN and edge firewall.
  • Public DNS and certificate chain.
  • Cloud or data-centre ingress.
  • Application service and downstream dependencies.

Run synthetic checks from South African regions and from important African markets where customers actually operate. Compare Johannesburg, Cape Town, Durban and international points rather than relying on a single probe. Track DNS duration, TLS negotiation, time to first byte and complete journey time separately.

For a dependency in eu-west, record both application latency and network RTT. A rising p95 on the fraud provider may be caused by the provider, the route, a firewall inspection policy or packet loss. The graph should make those hypotheses visible without pretending to identify the cause automatically.

Build one operational view from four models

A practical implementation starts with naming discipline. Use stable OpenTelemetry resource attributes such as service.name, deployment.environment.name, cloud.region and service.namespace. Keep cardinality under control: customer IDs and arbitrary URLs do not belong in metric labels.

Instrument applications with OpenTelemetry, collect infrastructure signals with Grafana Alloy or existing exporters, send traces to Tempo, logs to Loki and metrics to Prometheus or Mimir. Tempo’s service graph processing supports direct service calls and messaging relationships when spans carry the expected semantic information.[2]

In the dashboard, provide linked perspectives rather than one overloaded canvas:

  1. Start with the customer journey and its critical dependency edges.
  2. Open the RED metrics for the failing edge.
  3. Jump to representative Tempo traces.
  4. Correlate the trace ID with Loki logs.
  5. Inspect Mimir or Prometheus metrics for the relevant resource and failure domain.
  6. Check the user-path panel for regional or ISP-specific impact.

Use Grafana for the navigation layer, not as a substitute for modelling. A useful dashboard makes the next investigation step obvious: edge to trace, trace to log, log to infrastructure, and infrastructure back to customer impact.

Measure whether visibility is actually improving

Track the quality of the dependency model itself. Useful measures include the percentage of critical services emitting traces, the percentage of external calls carrying destination attributes, the number of unknown edges, queue flows with propagated context and dashboards with named owners.

Review the map after every major architecture change. Service graphs provide a historical view of changing topology, which helps identify when an unexpected dependency first appeared.[3] That history is particularly valuable in fast-moving platform teams, where eBPF and platform-level telemetry are increasingly being used to cover workloads that application teams have not instrumented.[4]

The best End-to-End Service Dependency Visibility Models do not promise perfect knowledge. They expose uncertainty, distinguish observed from inferred relationships and preserve the path from a customer symptom to a responsible engineering team. In an estate spanning local data centres, public cloud, international APIs and unreliable connectivity, that is the difference between “the dashboard is green” and knowing why the business is losing transactions.