grafana

Start with a failure model, not a dashboard

At 02:17, a payment API looked healthy in Johannesburg. CPU was normal, the Kubernetes pods were ready, and the synthetic check from the local office passed. Customers routed through a European dependency, however, were timing out. The useful…

Opsgenie

— 5 min read
Start with a failure model, not a dashboard

Unified Monitoring for Multi-Cloud Ecosystems: A Practical Observability Blueprint for African Platforms

At 02:17, a payment API looked healthy in Johannesburg. CPU was normal, the Kubernetes pods were ready, and the synthetic check from the local office passed. Customers routed through a European dependency, however, were timing out. The useful clue was not in a single dashboard: it sat across an AWS metric, an Azure application log, a trace sampled in Frankfurt, and a latency change caused by a failing undersea route.

That is the operational case for Unified Monitoring for Multi-Cloud Ecosystems. The aim is not to force every provider into an identical interface. It is to create one trustworthy operating view across cloud accounts, regions, clusters, networks and on-premises systems, while retaining enough provider-specific detail to troubleshoot failures properly.

Start with a failure model, not a dashboard

Multi-cloud monitoring often fails because teams begin by importing every available metric. The result is a large Grafana installation with inconsistent labels, duplicated alerts and no clear answer to the question: “Which customer journey is broken?”

Define the failure model first. For a South African service, that may include:

  • A Johannesburg application cluster losing access to an eu-west dependency.
  • A cloud region becoming unreachable during a network or power event.
  • Load-shedding affecting an on-premises database, edge appliance or observability collector.
  • Last-mile packet loss making a healthy service appear unavailable to a branch or customer.
  • A ZAR-denominated cloud bill rising because telemetry retention or cardinality has escaped control.
  • Personally identifiable information entering logs that are stored outside the permitted jurisdiction.

Each scenario needs a signal, an owner, a threshold and an action. “Monitor AWS, Azure and Google Cloud” is not an objective. “Detect when checkout errors exceed the SLO for customers entering through South Africa, then identify whether the fault is application, dependency or network related” is.

Build a common telemetry contract across providers

Prometheus remains a sensible metrics standard for heterogeneous estates. Exporters and cloud integrations can collect service health, Kubernetes state, node resources, load balancer performance and managed-database metrics. The important design choice is the label contract.

Every metric should carry a small, stable set of dimensions such as environment, cloud, region, cluster, service and team. Avoid embedding volatile identifiers, request IDs or customer values in metric labels. High cardinality increases storage, query and operational costs without necessarily improving diagnosis.

groups:
  - name: checkout-slo
    rules:
      - alert: CheckoutHighErrorRate
        expr: |
          (
            sum by (service, region) (
              rate(http_requests_total{
                service="checkout",
                status=~"5.."
              }[5m])
            )
          /
            sum by (service, region) (
              rate(http_requests_total{
                service="checkout"
              }[5m])
            )
          ) > 0.02
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "Checkout errors exceed 2% in {{ $labels.region }}"
          runbook: "checkout-error-rate"

Prometheus can remain close to workloads for fast local evaluation, while Mimir provides horizontally scalable, long-term Prometheus-compatible storage. That separation is useful when a cluster loses external connectivity: local alerting can continue, and metrics can be forwarded when the link recovers.

Grafana should present the same service-level dashboard regardless of whether a workload runs on AWS in Cape Town, Azure in Johannesburg or a private VMware estate. Provider-specific panels still have a place, but they should sit behind a shared service view rather than define it. [Grafana](https://grafana.co.za) is particularly useful here because dashboards can correlate multiple data sources without requiring engineers to learn a different visual workflow for each cloud.

Use logs and traces to explain the metric

A red error-rate panel is an alert, not a diagnosis. Loki gives teams a cost-conscious way to aggregate logs while indexing labels rather than the entire log body. Use labels for bounded fields such as namespace, application and region; keep request IDs, user agents and exception text inside the log payload.

{app="checkout", environment="prod", region="jhb"} 
|= "timeout"
| json
| duration_ms > 5000

Do not ship every log indiscriminately across borders. Redact tokens, identity numbers, email addresses and payment information before collection. Route sensitive streams to an approved in-country store, and retain only the fields needed for incident response, audit or security investigations. POPIA compliance is not achieved by adding a “popia=true” label after the data has already crossed a boundary.

Tempo completes the investigation when the request crosses several services and clouds. Instrument applications with OpenTelemetry where practical, then propagate trace context through HTTP, messaging and asynchronous jobs. A trace can show that the South African checkout service spent 40 milliseconds locally, 900 milliseconds waiting for a European fraud service and another 300 milliseconds retrying a database call.

Tempo integrates with Grafana, Mimir, Prometheus and Loki, allowing an engineer to move from a latency metric to a trace, then from a slow span to the relevant logs. Prometheus exemplars are valuable for this path because a metric point can link directly to a representative trace rather than leaving the investigator to guess which request was affected.[1]

Make regional and power events visible without paging on noise

South African infrastructure needs more than a generic “host down” alert. During load-shedding, a site may disappear because its UPS is exhausted, while the cloud service remains healthy. If the monitoring system pages for every unreachable endpoint, the response team will eventually ignore the alerts.

Separate infrastructure availability from customer impact. A collector heartbeat can identify that telemetry from a site has stopped. A synthetic transaction from an independent network can establish whether customers are affected. Battery runtime, generator state, WAN packet loss and power-quality telemetry can provide useful context without becoming page-worthy on their own.

  • Page when a customer-facing SLO is breached and an independent probe confirms impact.
  • Ticket when one telemetry collector is stale but service health remains normal.
  • Annotate dashboards with planned outages, load-shedding schedules and maintenance windows.
  • Use multi-region probes to distinguish a Johannesburg access problem from a global service failure.

Latency to eu-west deserves its own baseline. A fixed threshold that is reasonable for local traffic may be useless for an international dependency. Alert on deviation from a measured baseline, and track packet loss separately from round-trip time. A route can remain reachable while retransmissions make application requests fail.

Control telemetry cost in ZAR, not just time series

Observability cost is a reliability concern. Mimir retention, Loki ingestion, Tempo trace volume and cross-region egress all affect the monthly bill. Cost pressure becomes more visible when budgets are reviewed in ZAR, especially where teams have accumulated separate tools for every provider.

Useful controls include:

  • Record only the labels needed for aggregation and ownership.
  • Use recording rules for expensive, frequently viewed queries.
  • Keep high-resolution metrics for a short period and downsample or retain aggregates longer.
  • Sample routine traces while retaining error traces and slow traces at a higher rate.
  • Drop debug logs at the collector unless an incident requires temporary capture.
  • Tag telemetry storage and egress by platform team, environment and service.

Do not reduce telemetry blindly. A cheap system that cannot explain a payment failure is not economical. The better question is whether each signal changes an operational decision.

Operate the stack as a platform product

A unified monitoring platform needs ownership rules. The central team should provide collectors, authentication, retention tiers, naming conventions, dashboards and alert-routing patterns. Application teams should own service-level indicators, runbooks and the alerts that describe customer impact.

Use infrastructure as code to provision data sources, folders, contact points and dashboards. Test alert rules in CI. Treat dashboard changes like code: review them, version them and remove panels that no longer support a decision.

Recent observability practice is moving towards combined infrastructure and application telemetry rather than isolated monitoring products. A 2025 Grafana Labs survey reported that 71% of respondents used Prometheus and OpenTelemetry in some capacity, while 67% used Prometheus in production in some capacity.[2] That does not make OpenTelemetry mandatory everywhere, but it does reinforce the value of a vendor-neutral collection layer alongside Prometheus-compatible metrics.

The strongest implementation is deliberately boring at the edges: consistent labels, predictable routing, documented retention and tested failure modes. The centre can then be sophisticated, linking Mimir metrics, Loki logs and Tempo traces into one investigation path.

When the next outage crosses a cloud boundary, the team should not ask which provider dashboard to open first. It should see the affected service, the failing region, the relevant customer path and the evidence needed to restore it—without losing control of data sovereignty or the ZAR cost of keeping that evidence.

Read more