real

Real-Time Infrastructure Health Analytics: A South African SRE’s Playbook with Grafana

For DevOps engineers and SREs, Real-Time Infrastructure Health Analytics is no longer a nice-to-have – it is the difference between a quiet Saturday braai and a 3 a.m. incident call. As a South African SRE working with distributed…

Opsgenie

— 4 min read
Real-Time Infrastructure Health Analytics: A South African SRE’s Playbook with Grafana

Real-Time Infrastructure Health Analytics: A South African SRE’s Playbook with Grafana

For DevOps engineers and SREs, Real-Time Infrastructure Health Analytics is no longer a nice-to-have – it is the difference between a quiet Saturday braai and a 3 a.m. incident call. As a South African SRE working with distributed teams and latency-sensitive services, I’ve learned that seeing infrastructure issues as they happen, not after the fact, is critical to maintaining SLAs and keeping costs under control.

In this article, we’ll walk through how to design and implement a practical Real-Time Infrastructure Health Analytics stack using Grafana, with concrete examples, dashboards, and alerting patterns you can copy into your own environment.

Why Real-Time Infrastructure Health Analytics Matters

Traditional monitoring focuses on static thresholds and periodic checks. Real-Time Infrastructure Health Analytics goes further by:

  • Streaming telemetry (metrics, logs, traces, events) as it happens.
  • Correlating signals across infrastructure, apps, and users.
  • Surfacing anomalies early enough to prevent customer impact.
  • Feeding data back into capacity planning and cost optimization.

For South African teams, there are additional realities: regional latency to global clouds, undersea cable incidents, and sometimes flaky local connectivity. That makes end-to-end, real-time visibility across regions even more important.

Architecture for Real-Time Infrastructure Health Analytics

A practical architecture for Real-Time Infrastructure Health Analytics with Grafana typically includes:

  • Metrics pipeline: Prometheus, Grafana Mimir, or cloud metrics (CloudWatch, Azure Monitor, GCP Monitoring).
  • Logs pipeline: Loki, Elasticsearch, or a managed log service.
  • Traces: Tempo or an OpenTelemetry-compatible backend.
  • Visualization and alerting: Grafana as the central observability layer.

You want your metrics scraped or pushed in near real time (every 15–30 seconds for core infrastructure). Logs should stream via agents like promtail or fluent-bit, and traces sampled according to criticality.

Instrumenting Linux Servers for Real-Time Health

Let’s start with the basics: CPU, memory, disk, and network. A common pattern is to use node_exporter on each VM or bare-metal host and have Prometheus scrape it.

Example docker-compose.yml snippet for node_exporter:

version: "3.8"
services:
  node_exporter:
    image: prom/node-exporter:v1.8.1
    container_name: node_exporter
    restart: unless-stopped
    network_mode: host
    pid: host
    volumes:
      - /:/host:ro,rslave
    command:
      - '--path.rootfs=/host'

Prometheus scrape config (in prometheus.yml):

scrape_configs:
  - job_name: 'node'
    scrape_interval: 15s
    static_configs:
      - targets:
        - 'server-1.example.local:9100'
        - 'server-2.example.local:9100'

With this in place, you can build Grafana panels for Real-Time Infrastructure Health Analytics that answer questions like:

  • Which nodes are CPU or memory saturated right now?
  • Are we seeing packet drops or high network latency between regions?
  • Is disk IO a bottleneck on any critical service?

Example Grafana Panels for Node Health

Here are a few PromQL queries you can use directly in Grafana panels.

1. CPU Utilization per Node

100 - (avg by (instance) (
  irate(node_cpu_seconds_total{mode="idle"}[5m])
) * 100)

This panel shows real-time CPU usage per instance. Use a heatmap or time series with thresholds to highlight nodes above, say, 80%.

2. Memory Pressure

(node_memory_MemTotal_bytes
 - node_memory_MemAvailable_bytes)
 / node_memory_MemTotal_bytes * 100

This query gives you the percentage of used memory. In Grafana, set alerts if it exceeds a threshold for more than a few minutes.

3. Disk Space Usage

100 - (
  node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}
  / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}
) * 100

Filter by mountpoint to focus on critical volumes like /var/lib/docker or database data directories.

Alerting Strategy for Real-Time Infrastructure Health Analytics

Dashboards without alerts are just pretty graphs. A mature Real-Time Infrastructure Health Analytics setup uses multi-level alerting:

  • Warning: Early signals (e.g., CPU > 70% for 10 minutes).
  • Critical: Customer-impacting or imminent risk (e.g., disk > 90% and increasing).
  • Silent diagnostics: Signals that are recorded but do not page (used for post-incident analysis).

An example Prometheus alert rule (in alerts.yml) for high CPU:

groups:
  - name: infrastructure-health
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by (instance) (
                irate(node_cpu_seconds_total{mode="idle"}[5m])
              ) * 100) > 80
        for: 10m
        labels:
          severity: warning
          team: sre
          environment: production
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: |
            CPU usage has been above 80% for more than 10 minutes
            on {{ $labels.instance }} in {{ $labels.environment }}.

In Grafana, configure Alerting to send these to your preferred channel: Slack, Microsoft Teams, PagerDuty, or even a South African SMS gateway for when mobile data is down.

Bringing Logs into the Picture with Loki

Metrics tell you what is wrong; logs often tell you why. For effective Real-Time Infrastructure Health Analytics, you should be able to jump from a red CPU panel straight into relevant logs for that node and timeframe.

A minimal promtail config for pushing system logs into Loki:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

clients:
  - url: http://loki:3100/loki/api/v1/push

positions:
  filename: /var/log/positions.yaml

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*.log

Once logs are in Loki, you can create a Grafana panel to detect error spikes in near real time:

sum by (job) (
  rate({job="varlogs", level="error"}[5m])
)

Link this panel to your infrastructure metrics dashboard using Grafana’s dashboard variables and data links. When an alert fires on a node, you should be able to click through to filtered logs for that instance within seconds.

Tracing Critical Requests with Tempo and OpenTelemetry

In multi-region setups (e.g., South Africa to eu-west-1), latency can degrade without obvious infrastructure issues. Adding distributed tracing to your Real-Time Infrastructure Health Analytics pipeline helps you see where time is actually being spent.

Example: Instrumenting a Go service with OpenTelemetry to export traces to Tempo and metrics to Prometheus:

import (
  "go.opentelemetry.io/otel"
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
  "go.opente

Read more