Observability Stack Notes #

A) Metrics #

• Use a Helm chart to install the kube-prometheus-stack.
• This stack includes:
  • Prometheus Operator — initializes & manages the Prometheus instance; gives you CRDs → ServiceMonitor
  • Prometheus — time-series database, queried using PromQL
  • Alertmanager — service that sends alerts based on PrometheusRule (e.g. up == 0)
  • Node Exporter — DaemonSet, runs on every node, exposes hardware/OS state
  • kube-state-metrics — watches the Kubernetes API itself (how many pods are pending, does replica count = desired count?)
  • Grafana — GUI tool

Flow:

api/metrics               ┐
kube-state-metrics         ├─► scraped every 30s by Prometheus ─► Grafana
node-exporter (per node)  ┘

• Operator constantly watches ServiceMonitor for configuration changes
• Alertmanager evaluates PrometheusRules & notifies users

ServiceMonitor: defines a set of targets for Prometheus to target & scrape

• Allows you to avoid touching Prometheus configs directly
• A ServiceMonitor will reference a Service anywhere in your application

B) Logging: Loki + Grafana Alloy #

Alloy (DaemonSet, 1 pod/node) ──► Loki ──► Grafana

• Alloy tails the log files written by kubelet
• Loki = database — stores + indexes it

Only difference between Loki & Prometheus:

• Loki only indexes the labels
• Prometheus indexes everything

Alloy pushes to Loki, Prometheus pulls data.