Kubernetes Observability Platform
End-to-end observability for a Kubernetes fleet — metrics, logs and alerting unified into SLO-driven dashboards so failures surface before users feel them.
- Role
- Design, build & operations
- Timeline
- 2025
- Stack
- Kubernetes · Prometheus · Grafana · ELK · Alertmanager · Helm
The problem
Services ran across a multi-tenant Kubernetes fleet with no shared view of health. Each team kept its own dashboards and its own idea of "broken", so incidents were usually reported by users rather than detected by monitoring. Debugging meant SSH-ing toward whichever node looked suspicious and reading container logs by hand — and because pods are ephemeral, the evidence was often already gone by the time anyone looked.
Constraints
- Multi-tenant cluster — one team's noisy metrics must not degrade another's.
- Pods are ephemeral; logs and metrics have to outlive the workload that produced them.
- Alerting had to be adoptable by teams who did not write it, so conventions mattered more than cleverness.
- Retention and cardinality had to fit a fixed storage budget.
Architecture
- 01
Collection
Prometheus scrapes application and infrastructure targets through ServiceMonitor definitions, so onboarding a service is a manifest change rather than a config edit. Node and cluster-level metrics come from node-exporter and kube-state-metrics.
- 02
Log pipeline
A node-level agent ships container logs into Elasticsearch with Kubernetes metadata (namespace, pod, labels) attached at ingest, so a log line can be traced back to the workload and deploy that produced it after the pod is gone.
- 03
SLO & alert layer
Recording rules turn raw counters into per-service availability and latency indicators, and alert rules fire against error-budget burn rather than instantaneous spikes. Alertmanager handles routing, grouping and inhibition so one failing dependency produces one page, not fifty.
- 04
Presentation
Grafana dashboards are templated by service and environment — one dashboard definition serves every team instead of a hand-built copy per service. The whole stack ships as versioned Helm releases, so the platform is reproducible and reviewable like any other code.
Decisions & tradeoffs
Alert on error-budget burn rate, not raw thresholds.
why Static thresholds either page constantly on healthy noise or stay silent through a slow degradation. Burn rate ties every page to user-visible impact, which is also what makes an alert arguable in a review.
cost Requires defining an SLO per service up front — real work, and a conversation with each team before their alerts mean anything.
Templated dashboards over per-service dashboards.
why A single parameterised dashboard means a fix or improvement lands everywhere at once, and every service is read the same way during an incident.
cost Less room for service-specific panels; teams with genuinely unusual workloads needed an escape hatch.
Metadata enrichment at ingest rather than query time.
why Attaching pod and namespace labels while the context still exists makes logs searchable after the pod is deleted — the exact moment you need them.
cost Larger documents and a heavier ingest path, paid for with tighter retention on low-value log streams.
Outcomes
[verify]
MTTR reduction
Measure median time-to-resolution across incidents before and after the platform landed.
[verify]
Services onboarded
Count of workloads reporting metrics and logs through the platform.
[verify]
Detection coverage
Share of incidents caught by alerting rather than reported by a user — the metric that best justifies the work.
What I'd do differently
- Dashboards are not observability. The step that changed on-call was defining what "healthy" meant per service; the graphs were the easy part.
- Alert fatigue is a design failure, not a people problem — grouping and inhibition mattered more than adding rules.
- Cardinality is a budget. A single unbounded label on a busy metric can cost more than an entire service's monitoring.
next case study
Infrastructure Automation Pipeline