Kubernetes Observability: Prometheus & Grafana Production Guide 📊☸️

A production engineering guide to building an enterprise observability stack on Kubernetes. Deploy kube-prometheus-stack, configure ServiceMonitors, write PromQL alerts, and visualize cluster health.

1. The Cloud-Native Observability Architecture

Operating Kubernetes at enterprise scale requires real-time telemetry across three distinct layers: the Kubernetes Control Plane, the underlying Node hardware, and application Pod runtimes.

The production standard is the Prometheus Operator (kube-prometheus-stack), which packages:

Prometheus: Time-series database engine utilizing a pull-based HTTP scraping model.
Node Exporter: DaemonSet collecting hardware-level metrics (CPU, memory, disk, network) from physical nodes.
kube-state-metrics: Ingests Kubernetes API objects (Pod statuses, replica counts, deployment rollouts).
Alertmanager: Deduplicates, groups, and routes critical alarms to Slack, PagerDuty, or Webhooks.
Grafana: Analytics and interactive visualization dashboards.

2. Deploying kube-prometheus-stack via Helm

Deploy the operator into a dedicated monitoring namespace with persistent volume storage:

# 1. Add Prometheus Community Helm Repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# 2. Deploy with customized storage values
helm install k8s-monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.retention=30d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi \
  --set alertmanager.enabled=true \
  --set grafana.adminPassword="ProductionSecurePassword123!"

3. Exposing Custom Application Metrics with ServiceMonitors

Instead of manually editing Prometheus YAML configuration files, the Prometheus Operator introduces the ServiceMonitor Custom Resource Definition (CRD) to discover scrape targets automatically:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: order-service-monitor
  namespace: monitoring
  labels:
    release: k8s-monitoring
spec:
  selector:
    matchLabels:
      app: order-service
  namespaceSelector:
    matchNames:
      - production
  endpoints:
  - port: metrics
    interval: 15s
    path: /actuator/prometheus

4. Production PromQL Queries & Alert Rules

Master these production PromQL queries to monitor service level indicators (SLIs):

Pod CPU Saturation Rate:

sum(rate(container_cpu_usage_seconds_total{namespace="production", container!=""}[5m])) by (pod)

Pod Memory Working Set (OOM Risk):

sum(container_memory_working_set_bytes{namespace="production", container!=""}) by (pod)

HTTP 5xx Error Rate (%):

sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100