# Observability Stack Notes

## A) Metrics

- Use a Helm chart to install the `kube-prometheus-stack`.
- This stack includes:
  - **Prometheus Operator** — initializes & manages the Prometheus instance; gives you CRDs → `ServiceMonitor`
  - **Prometheus** — time-series database, queried using `PromQL`
  - **Alertmanager** — service that sends alerts based on `PrometheusRule` (e.g. `up == 0`)
  - **Node Exporter** — DaemonSet, runs on every node, exposes hardware/OS state
  - **kube-state-metrics** — watches the Kubernetes API itself (how many pods are pending, does replica count = desired count?)
  - **Grafana** — GUI tool

**Flow:**

```
api/metrics               ┐
kube-state-metrics         ├─► scraped every 30s by Prometheus ─► Grafana
node-exporter (per node)  ┘
```

- Operator constantly watches `ServiceMonitor` for configuration changes
- Alertmanager evaluates `PrometheusRule`s & notifies users

**ServiceMonitor:** defines a set of targets for Prometheus to target & scrape
- Allows you to avoid touching Prometheus configs directly
- A `ServiceMonitor` will reference a Service anywhere in your application

---

## B) Logging: Loki + Grafana Alloy

```
Alloy (DaemonSet, 1 pod/node) ──► Loki ──► Grafana
```

- Alloy tails the log files written by kubelet
- Loki = database — stores + indexes it

**Only difference between Loki & Prometheus:**
- Loki only indexes the labels
- Prometheus indexes everything

**Alloy pushes to Loki, Prometheus pulls data.**