zat.dev / stream
jetstream v2 in zig stream.waow.tech
Star
Fork 0
Watch
stream COSTS.md
2.7 kB
Markdown
at zig-0.17

COSTS.md #

Mandate. Tracks the infrastructure cost attributed to stream (stream.waow.tech) and what we're doing about it. Numbers drift, so this file names the levers and the instruments, never a hardcoded figure.

current cost — fetch it live, never hardcode #

The daily snapshot (collected by my-prefect-server, surfaced at https://hub.waow.tech) attributes this repo's resources to the stream project:

curl -s https://hub.waow.tech/api/costs.json | jq '{
  as_of: .generatedAt,
  stream_monthly_usd: (
    [ .lineItems[] | select(.project == "stream") ]
    | (map(.amount) | add // 0) / 100
  ),
  lines: [ .lineItems[] | select(.project == "stream")
           | {service, provider, usd: (.amount/100), estimated} ]
}'

Resources: the stream-cx43 hetzner box (hel1) + its 3 TB volume, plus any fly apps whose names contain stream (e.g. the loadgen). Attribution lives in my-prefect-server packages/mps/src/mps/costs/projects.py; fix mappings there, not numbers here.

the shape of the spend (measured 2026-08-09) #

  • the volume is the cost. block storage bills per provisioned GB (€0.0767/GB·mo gross); the sealed archive (~1.75 TB) plus grow-only headroom dwarfs the server line. treemap snapshot of the full breakdown: operator session artifact "stream · where the money goes".
  • memory is not a lever. ~3 GB RSS on 16 GB; the rest is page cache over a network-attached volume, which is what makes cold serving tolerable. breakdown: stream_manifest_resident_bytes + tail + tombstone gauges (grafana → "Memory contributors").
  • format density is not a lever. recompression of real blocks measured ≤ 11% headroom (per-block l19 / whole-file l19 / trained dict) — not worth a 1.7 TB rewrite, and upstream owns the format.

levers, in order #

  1. compaction deletes-fold reclaiming segment bytes (throttled workers).
  2. right-sizing the volume — hetzner volumes grow but never shrink, so this is a copy-to-smaller-volume migration, sized after growth settles.
  3. object-storage tiering for cold segments. go/no-go is empirical: stream_compaction_rewritten_age_seconds reports whether steady-state rewrites cluster in recent segments (tiering works) or scatter across deep history (it doesn't). decide from the histogram, not from vibes, and not while upstream format work is in flight.

changelog #

  • 2026-08-09 — initial notice. hetzner cost connector taught to price volumes (my-prefect-server 665b2dd); stream project mapping added (5c3f8d7). pending: HCLOUD token for the stream hetzner project in the hetzner-tokens secret block — until it exists, snapshots show no hetzner lines for this repo.