Hydrant-backed Wisp intake #

Hydrant owns upstream relay connections, repository discovery, verification, backfill, and a persistent record stream. Wisp retains one elected writer, bounded admission, per-site ordering, PDS-authoritative reconciliation, durable retry handoff, and the completed-prefix checkpoint. This is not a broadcast queue with consumer acknowledgements: keep Wisp leadership enabled for replicas.

Current deployment and scope #

On 2026-10-02, only Hydrant on valefar was upgraded/reset. The fleet Wisp firehose containers remain on their existing relay images/configuration; the checkout integration is not a fleet release or completed cutover.

• Upstream revision: 3a1fbec1d94100ff4c11b9c82f1716982ae4471e.
• Image: hydrant:wisp-3a1fbec1d941 (revision label checked).
• Image ID: sha256:dd97217823dad8fac643e7249cae8844dca26d6f1dfed23aeadbf2058a7f0776.
• Database epoch: wisp-20261002t185051z.
• New volume: wisp-hydrant-data-20261002t185051z.
• Preserved old volume: wisp-hydrant_hydrant_data; never reused for the reset.
• Preserved old image: hydrant:rollback-20261002t185051z.
• Remote evidence/rollback compose/archive: /home/regent/docker/hydrant/records/20261002T185051Z.
• Offline old-volume archive SHA-256: 8be1b24cdbcec67f9813bba2625a200d4b93c3435f61d2ed116131d0d70f459e.
• Local verification evidence: /Users/regent/fleet/records/2026-10-02-hydrant-wisp-upgrade.

compose.yaml is the service-only source of truth. Its image and external volume are required variables, supplied by /home/regent/docker/hydrant/.env. Pass an explicit --env-file .env to Docker Compose when using sudo; sudo may discard shell-provided variables. Preserve rollback files and the old volume.

Management APIs have no authentication. Bind only the mesh/tailnet addresses, never 0.0.0.0 host ports. Public forwarding/reverse proxying is not authorized. Wisp's HTTP exception is confined to the exact root endpoint http://valefar.mesh.wisp.place:3010 or http://10.88.0.10:3010 with HYDRANT_ALLOW_MESH_HTTP=true; otherwise production requires HTTPS. No generic PDS/SSRF rule is relaxed. The configured Hydrant service is a trusted operator boundary, not an untrusted record provider whose JSON signatures Wisp verifies.

Upstream configuration #

The by-collection crawler rejects wildcard signals. Supply the five concrete signals in compose (fs, subfs, settings, v2.domains, v2.wh), but retain place.wisp.* for indexed collections, including historical namespaces. Both relay sources, the collection crawler, and backfill are enabled. Permanent history, complete record bodies, full signature/CID/MST verification are enabled; no history/body TTL is configured. Monitor disk growth: permanent history is intentionally not bounded by Wisp's intake queue.

Changing filters does not rebuild an existing database. A true filter reset uses a distinct empty persistent volume with an independently recorded epoch. Never delete live Fjall files, mutate only selected database files, or reuse the only copy of the old volume. No upstream source edits were made.

Wisp opt-in, not automatic fallback #

FIREHOSE_SOURCE_MODE=hydrant
HYDRANT_URL=http://valefar.mesh.wisp.place:3010
HYDRANT_ALLOW_MESH_HTTP=true
HYDRANT_EPOCH=wisp-20261002t185051z
BACKFILL=false

Remove FIREHOSE_SERVICE and FIREHOSE_SERVICE_SECONDARY from the effective configuration rather than leaving a primary with an empty value. Explicit Hydrant mode rejects manually configured relay endpoints, rejects DB-fill-only, requires Bun ping/pong support, and disables Wisp's own startup backfill, relay failover, traffic-stall watchdog, standby probing, and time-based cursor seeking. Setting HYDRANT_URL alone does not select this mode: default remains relay. Keep production Redis, S3, independent leadership supervisor and write fences; Hydrant does not replace those side-effect safety boundaries. Single-instance mode with Redis also reads and saves the epoch-specific completed checkpoint.

The checkpoint is scoped to endpoint and database epoch, not the relay host or relay sequence. Every database reset, restore, replacement, or operator recovery that may rewind/reuse stream IDs needs a new epoch on all replicas. Never copy a raw-relay cursor into this namespace or reuse the former database epoch. Hydrant IDs are now progress hints, not event identities: Wisp replays retained history from zero through its PostgreSQL CID ledger, including after a head rewind.

Replay, readiness, and limits #

/stream?cursor=N uses JSON and an inclusive record-stream cursor. Every Wisp Hydrant connection subscribes from 0, never an omitted cursor or a saved numeric ID: a source can reuse IDs and catch up past the old head before Wisp reconnects. The hydrant_record_ledger table stores one latest accepted CID/repo revision per record, with delete tombstones. Marking follows writer completion or the existing durable repair handoff; failed ledger queries cannot certify a checkpoint. The ledger survives Wisp restarts and uses the existing PostgreSQL pool; first Hydrant startup requires permission to create its table/index. Raw-relay mode is unchanged.

Replay is streaming with bounded admission, but each connection costs O(retained records) traversal/database lookups. It cannot recover records already evicted from upstream history. Completed older revisions and current CIDs are skipped; delete tombstones become eligible for cleanup after 30 days in indexed batches of 256 at startup and every 256 accepts, so older deletion replays still depend on authoritative PDS reconciliation. No unbounded event-history or in-memory seen-CID set is kept. IDs may have holes; exact safe integer wire tokens are required, not rounded JSON numbers. Identity/account notifications are live-only and do not advance Wisp's record checkpoint or certify readiness.

Each connection captures /stream/head; ready means Wisp's completed prefix has reached that snapshot. A genuinely empty snapshot may be ready while idle; pongs prove only connection liveness. Neither status certifies crawler/global backfill completion or the integrity of every historical upstream row. Permanent history can still contain missing bodies, and upstream can skip malformed/uninflatable stored rows. Wisp treats missing bodies as reconciliation hints, never as a successful empty filesystem. Settings failures require authoritative reconciliation or a durable retry handoff before checkpoint completion.

Source admission is 64 retained frames and 32 MiB conservative reservation, including the current callback. Raw frame cap is 16 MiB, but reservation factor makes the effective single-JSON cap roughly 1 MiB; it is not a measured heap cap. At overload or malformed input, close and replay through the durable ledger; never discard a frame and continue checkpointing later records. Repeated oversized records can require operator recovery. PDS SubFS expansion, writer scheduling, revalidation and cache fencing retain their own limits. Foreign callbacks and fetch/body/socket finalizers join actual settlement; an uncooperative dependency can block teardown rather than falsely certify release.

Cutover gates: required before a fleet release #

1. Build/test the Wisp image on the supported pinned Bun runtime. Verify replica leadership, epoch-specific checkpoint persistence, retry replay, byte parity, settings updates, and PDS-confirmed deletions through actual application writers.
2. Audit/reconcile all existing Wisp projections against authoritative PDS state, including old cached sites/settings absent from the fresh Hydrant DB. A fresh database replays discovered/current records but cannot recreate prior deletion events. Hydrant inventory absence and crawler readiness are not delete authority. Use bounded pages and existing site-write/revalidation fences; transient/auth/cap failures must preserve data and block cutover. The existing absent-site sweeper does not itself enumerate all unmarked old projections.
3. Establish upstream durability/recovery policy. The pinned main has no signal-driven worker drain. init: true forwards TERM and avoids PID1 ignoring it; it does not implement graceful application shutdown. Index batches are atomic, but publish is not gated by synchronous per-event durable persistence. Periodic persist defaults to Buffer every 3 seconds; even the optional __persist_sync_all path is not a per-publish barrier. Wisp checkpoints can outlive recently published upstream events after an OS/power crash. Head regression checks and operator epochs are not a complete solution. Before claiming lossless crash recovery, require an upstream durable-publish contract or a tested recovery/reconciliation procedure that changes epoch when necessary.
4. Obtain separate approval for the fleet app release; do not change every app merely because the Hydrant service reset completed.

Observed process-crash failure #

The isolated 2026-10-02 gate reproduced loss of an already published live record in 6/6 immediate SIGKILL trials against the pinned image: three at the default 3-second persistence interval and three at 1 hour. Each genuine PDS settings record's stream CID and body matched the upload. Same-database reopen had all three ingestion flags false; the stored head regressed, the target returned 404 RecordNotFound, and its replay row was absent. A default-interval control that waited 4,505 ms retained its exact ID, CID, body and replay row.

This is observed process-crash loss, not just a hypothetical OS/power-loss window. Neither delayed persistence nor the head guard constitutes lossless recovery. That gate did not test ID reuse or OS/power crashes. Raw commands, full rows, disabled reopen checks and scripts are archived under ~/fleet/records/2026-10-02-hydrant-battle-4_lzp2dt/crash/. evidence.json contains the six loss trials; positive-evidence.json contains the surviving delayed control. The archive root includes MANIFEST.json and verified SHA256SUMS; it does not depend on temporary-directory retention. The local PLC used a namespace-only public-literal loopback alias and transparent forwarding; genuine DID documents and full signature/CID/MST checks were unchanged. Native localhost-PDS backfill stayed guarded and is not certified by this gate. No production mutations or upstream source edits were needed. Fleet intake stays on relay; durable publish or tested authoritative recovery remains a cutover blocker.

Earlier writer battle gate (before CID-ledger recovery) #

The isolated Bun 1.4.2 gate used the exact pinned image, genuine devnet PDS/PLC records, a fresh PostgreSQL database, initially empty Redis database 15, a unique MinIO prefix, and the real firehose/hosting applications. Leadership was disabled for this single-writer test; it is not replica/production certification.

• Rust 2.0.0 uploaded 258 files and a SubFS manifest; all served bytes matched.
• Deltas uploaded/reused 2/256 then 1/257 files; each 258-file byte check passed. The removed file returned 404; settings produced the requested response header.
• A byte-preserving TCP splice dropped the actual stream with a real MinIO PUT held. The retained writer remained active across reconnect, replay readiness was false until physical release, and completion/replay restored exact bytes.
• A whole Hydrant API outage instead exhausted the default three-attempt recovery budget and ended in a nonclean drain (exit 1). The completed checkpoint did not certify the pending manifest; application restart replayed it and recovered exact bytes. This is not a passing graceful-outage/restart gate.
• An announced empty-DB reset with the old checkpoint made no stream subscription. A new epoch subscribed from zero and became ready. The old process required the harness's later TERM to exit; its autonomous shutdown was not certified. This does not prove silent reset or reused-ID recovery.
• Actual deletion of the manifest, settings and all three owned SubFS records propagated to hosting 404. Owned apps/containers, database and Redis namespace were removed; MinIO listing confirmed the unique prefix empty. Original devinfra remains running, with no shared-service restart or production mutation.

Receipts, byte hashes, causal TCP trace, outage/reset observations and disposable scripts: ~/fleet/records/2026-10-02-hydrant-battle-4_lzp2dt/pipeline/. Start with verification.json; the other JSONs retain individual observations. Archive hashes were verified and credential values checked before preservation; no credential redactions were needed. Native localhost-PDS backfill, live replica leadership and the all-old-projection PDS audit remain separate gates. The process-crash loss above still blocks cutover.

CID-ledger continuation gate (2026-10-02) #

The earlier ingestion-enabled crash test reproduced a temporary head452→450; Hydrant reconstructed the original events and emitted new ones, but retained Wisp intake stayed stopped for nearly five minutes. Fresh hosting200 came from storage miss repair, not firehose recovery. That failing receipt remains at ~/fleet/records/2026-10-02-hydrant-resume-e7b4ca27/.

With CID-ledger replay, the final isolated gate passed three actual child SIGKILL restarts, with ingestion on and the same mounted DB and Wisp process/checkpoint. Heads484→483, 488→483, and492→489 reopened after kills182–234ms after publication. Each round independently confirmed PostgreSQL acceptance, fresh/update exact hosting bytes200, deletion tombstones and hosting404, and final ready health. Source IDs were not reset by the harness and no retry thresholds/backoff constants were changed. A real-socket regression test also covers reused IDs when the head has already caught up beyond the old checkpoint, and failed SQL markers cannot advance it. Evidence and cleanup receipts: ~/fleet/records/2026-10-02-hydrant-cid-e2e1254b/.

This verifies local single-writer continuation, not upstream power-loss durability, complete old-projection recovery, replica leadership or production release. Fleet remains on relay. The full retained-history replay cost still needs production sizing.

Intentional restart and rollback #

For an intentional restart, first PATCH /ingestion with crawler/firehose/backfill all false. Confirm in-flight work has settled and head/stats are stable, allow the periodic cursor/persistence checkpoint, then stop with the matching timeout:

cd /home/regent/docker/hydrant
sudo docker compose --env-file .env -f compose.yaml stop -t 120 hydrant
sudo docker inspect wisp-hydrant --format '{{json .State}}'
# Only after stopped; do not manipulate its active volume.
sudo docker compose --env-file .env -f compose.yaml up -d --no-build hydrant

The configured ENABLE flags can re-enable ingestion immediately on restart; check /ingestion, then verify image/volume/private bindings and persistent head/representative historical records. PATCH ingestion back to all true if needed. These pause/stability steps reduce intentional-stop risk; they do not create a shutdown fsync/drain contract. Exit 143 with init demonstrates signal termination, not an upstream joined drain. The pre-upgrade container exited 137 after its stop timeout; its archive was made only after stopped, not presented as a clean shutdown.

For rollback, stop and verify the new container first. Use the archived compose.rollback.yaml, explicitly set its image to the retained rollback tag (the old file's unpinned image name alone is unsafe), and point it at the preserved old volume. Inspect rendered compose before starting it. Keep both volumes; do not restore a backup over an active database. If Wisp has ever been cut over, stop/fence those writers and assign a new epoch for the restored DB; returning to raw relay intake must use its original relay cursor namespace, not a Hydrant ID. No rollback-to-old-service proof or power-loss certification is claimed.