Start by separating collection, transport and storage
At 02:13 on a Tuesday morning, a load-shedding event dropped two edge sites onto generator power. When connectivity returned, every Prometheus instance attempted to flush its backlog at once. The monitoring platform did not lose the data, but…
Scalable Metrics Processing Architectures: Designing for Growth Without Blowing the ZAR Budget
At 02:13 on a Tuesday morning, a load-shedding event dropped two edge sites onto generator power. When connectivity returned, every Prometheus instance attempted to flush its backlog at once. The monitoring platform did not lose the data, but ingestion latency climbed, dashboards timed out, and the alert that mattered most arrived after the service had already recovered.
That incident exposed the real problem with Scalable Metrics Processing Architectures: scale is not simply a matter of adding storage. A useful design must absorb bursts, isolate noisy tenants, keep queries responsive, and continue making sensible alerting decisions when links between South African sites and cloud regions are unreliable.
Start by separating collection, transport and storage
A small estate can run Prometheus, Grafana, Loki and Tempo on one Kubernetes cluster. That arrangement is attractive because it is simple, but it couples four very different workloads:
- Prometheus scraping and local rule evaluation.
- Remote-write transport and retry queues.
- Long-term metrics storage and query execution.
- Logs, traces and dashboard traffic.
Once the estate includes Johannesburg and Cape Town facilities, cloud workloads in eu-west, and hundreds of exporters, those workloads should be separated. The usual pattern is to keep collection close to the targets, move durable storage into a horizontally scalable backend, and place query capacity behind an independent frontend.
Prometheus remains an excellent scraper and local troubleshooting tool. Its local TSDB provides fast access during an incident, while remote write forwards selected data to Mimir. Prometheus Agent mode is also useful at constrained sites: it retains Prometheus discovery and scrape configuration but removes local querying, alerting and ordinary TSDB functions in favour of scraping and remote writing.[3]
A practical flow looks like this:
exporters and Kubernetes
|
v
Prometheus or Prometheus Agent
|
| remote_write
v
regional gateway / distributor
|
v
Mimir ingesters and object storage
|
+--> Grafana dashboards and PromQL
+--> ruler and alertmanager
The important boundary is between the scraper and the long-term backend. If a cloud link fails, the scraper should queue data and apply a defined retention policy rather than allowing a remote storage outage to consume all available memory.
Use a regional ingestion layer for burst control
Directly sending every site to a single Mimir endpoint is easy to deploy and difficult to operate. It creates a long network path, makes eu-west latency part of every write, and turns a central endpoint into the default failure domain.
A better South African layout places a write path in the same broad network region as the workloads. Johannesburg-based systems can write to a Johannesburg Mimir cluster or gateway; Cape Town systems can use a local endpoint; cloud workloads can use the closest regional path. Replication and object storage policy then determine how much data is kept locally and how much is copied elsewhere.
Mimir is built from horizontally scalable components rather than one monolithic server. Distributors receive writes, validate samples and enforce tenant limits, while query frontends handle incoming reads.[1][2] This separation allows ingestion and query capacity to scale independently. It also makes it possible to protect a critical dashboard from a flood of low-value metrics.
Tenant limits should be treated as reliability controls, not administrative decoration. Set limits for:
- Maximum active series per team or environment.
- Ingestion rate and burst size.
- Query time range and parallelism.
- Label length and label count.
- Rules that create excessive recording-series output.
When a new deployment adds a unique pod UID, request ID or customer token to a metric label, the resulting cardinality increase can be more damaging than a CPU spike. Rejecting or throttling that stream is preferable to allowing it to degrade every tenant.
Make the Prometheus edge tolerant of bad links
Load-shedding is not the only connectivity risk. A rural branch may have a healthy application and an unreliable last-mile link. A Prometheus server that scrapes locally and remote-writes centrally should therefore be configured for temporary failure, not perfect connectivity.
remote_write:
- url: https://metrics.example.net/api/v1/push
queue_config:
capacity: 12000
max_shards: 20
min_shards: 2
max_samples_per_send: 2000
batch_send_deadline: 5s
metadata_config:
send: true
The exact queue values depend on sample rate, available disk and outage duration. Capacity should be calculated from an explicit recovery objective: if a site produces 3,000 samples per second and must survive 20 minutes without upstream connectivity, the buffer needs to accommodate at least 3.6 million samples, with headroom.
Do not confuse buffering with indefinite durability. Prometheus WAL and remote-write queues are recovery mechanisms, not an archival system. If a branch can be offline for hours, deploy a local gateway or regional store and define which metrics are worth retaining. Keep high-value service indicators, capacity signals and security metrics; avoid forwarding every experimental series.
Alerting also needs locality. A central alert that fires because the monitoring link failed is not equivalent to an alert that proves the application failed. Evaluate link, power and service alerts at the site where possible, then route notifications centrally. Grafana can present both views in one dashboard: the service state and the telemetry-path state.
Control cost before cardinality controls you
Metrics storage costs are driven less by the number of dashboards than by samples, retention and series churn. ZAR cloud-cost pressure makes “store everything forever” a poor default, especially when object storage, cross-region traffic and query egress are included.
Use a tiered retention model:
- Keep high-resolution data for a short operational window.
- Downsample or record longer-term service-level indicators.
- Retain low-cardinality business and capacity metrics for trend analysis.
- Keep raw samples only when they support a defined investigation or compliance need.
Recording rules can reduce repeated expensive queries, but they can also create another large set of time series. A rule such as a per-route, per-customer aggregation may be more expensive than the original data. Review the output cardinality before deploying it.
PromQL should aggregate early and avoid unconstrained dashboard queries:
sum by (cluster, namespace) (
rate(container_cpu_usage_seconds_total{container!="POD"}[5m])
)
This query produces a useful operational view without carrying pod names into the final result. Dashboards should use bounded time ranges, sensible step sizes and variables that cannot expand to thousands of simultaneous series. Grafana makes those limits visible to operators, but the underlying query design still determines whether the system remains responsive.
Keep metrics, logs and traces on related but distinct paths
Metrics, logs and traces support one investigation, but they do not have identical scaling characteristics. Loki is usually dominated by log volume and retention. Tempo is shaped by trace ingestion, sampling and object storage. Mimir is sensitive to active series, sample rate and query fan-out.
Running all three on one pool can make an incident worse. A log surge from a failing application may consume the resources needed by metrics ingestion. Separate node pools, resource quotas and object-storage policies reduce that blast radius.
Use labels consistently across signals: service, environment, cluster, namespace and region are generally more valuable than request-specific identifiers. A trace ID can be included in logs, while exemplars can connect a metric point to a trace without adding the trace ID to every metric label. Grafana then becomes the investigation surface rather than the place where data is unnecessarily duplicated.
POPIA and data-sovereignty requirements should shape the topology early. Decide whether logs contain personal information, whether traces may leave the country, and which teams can query each tenant. Redaction belongs as close to ingestion as practical. Access control in Grafana is necessary, but it should not be the only barrier protecting sensitive telemetry.
Test the architecture with the failures it is meant to survive
A scalable design is credible only after failure testing. Run controlled exercises for:
- A 30-minute loss of the WAN link from a branch.
- A sudden reconnect where buffered samples arrive in a burst.
- A single tenant exceeding its active-series limit.
- A Mimir ingester or distributor becoming unavailable.
- A Loki log-volume spike during an application incident.
- High query concurrency during a regional outage.
Measure ingestion lag, dropped samples, queue depth, active series, distributor rejection rates, query latency and alert-delivery delay. Track these as first-class service-level indicators for the observability platform itself.
The strongest Scalable Metrics Processing Architectures are deliberately boring at the edges: local scraping, bounded queues, explicit limits and clear failure signals. The sophistication belongs in the separation of responsibilities. Prometheus collects, Mimir stores and scales the metrics path, Loki and Tempo handle their own workload profiles, and Grafana ties the evidence together without becoming the system’s single point of failure.