diff --git a/docs/PORTING.md b/docs/PORTING.md index 20866abd4..edfd18d91 100644 --- a/docs/PORTING.md +++ b/docs/PORTING.md @@ -4,6 +4,12 @@ This document is for engineers and coding agents porting solstone behavior from Python into the Rust workspace under `core/`. It records the wave-0 rules before any behavior moves. +**The map is separate from the doctrine.** This document is *how* to port. The +boundaries themselves — every plate, every strand, which end owns each contract, +and what must be carried forward out of the Python — are defined in +[`conversion/`](conversion/README.md). Read that for what and where; read this +for how. + ## Workspace Scope The Rust workspace lives at `core/`. It contains a thin `solstone-core` bin, diff --git a/docs/conversion/README.md b/docs/conversion/README.md new file mode 100644 index 000000000..4e816d1b7 --- /dev/null +++ b/docs/conversion/README.md @@ -0,0 +1,95 @@ +# The conversion dictionary — plates, strands, cables + +**The canonical definitions of the journal's conversion boundaries.** If two pieces of work need to agree on what a boundary is called, what it holds, or which side owns its contract, it is defined here. + +This directory is **definitional and present-tense**. It says what the boundaries *are*. It deliberately carries no schedule, no status, no ownership of work, and no record of who decided what — those live outside this repo, and anything of that kind appearing here is a defect. + +**Companion:** [`../PORTING.md`](../PORTING.md) is the porting *doctrine* — how to port, layering, error mapping, data boundaries, JSON and hashing, version lockstep. This directory is the *map*. Read `PORTING.md` for how, read here for what and where. + +## The files + +| File | Holds | +|---|---| +| [`plates.md`](plates.md) | every plate, what it holds, its contract tier | +| [`strands.md`](strands.md) | every strand, its two plates, which end owns the contract, and what must be carried forward | +| [`cables.md`](cables.md) | whole use cases through several strands | + +## Vocabulary + +| Term | Means | +|---|---| +| **plate** | A boundary of the journal where strands connect from other plates. ⛔ A plate has no "sides" — strands can be bi-directional, and in/out gets confusing | +| **strand** | The minimum viable path connecting two plates, in Rust, robust. May be bi-directional. **The contract lives at one *end* of the strand, never both** | +| **cable** | A whole use case, from one major system to another, through several strands with plates between them | + +⛔ **"Surface" is not one of these.** It is reserved for **owner-visible** things. ⛔ Not "thread" either, so it never collides with the Rust threading model. + +## Naming + +| | | +|---|---| +| Plate | `P-name` — ⛔ never `P1`/`P2`; a number reads as a priority | +| Sub-plate | dashes add granularity within a plate: `P-journal-segment` | +| Strand | `S:plate:plate` — 🔴 **the second plate always owns the contract** | +| Cable | `C:plate:plate:plate…` — every plate in the use case, in order | + +Worked: `S:device-ingest:segment-media` — `segment-media` owns the contract. `C:device-ingest:segment-media:journal-segment`. + +## One contract per strand, at one end + +**The owner is the one-to-many end.** It must serve all comers and cannot negotiate per-peer. **Tiebreak when the strand is one-to-one: the durable end owns it.** By convention the owner is written second in the strand name. + +📌 *Why this is a rule and not a preference:* nearly every contract drift in this tree is two places owning one thing with nothing binding them — a relay string matched verbatim on both sides, a nonce alphabet hand-copied between languages, a prompt preamble held as a constant in one place and as a sha256 of itself in another, a frontend switching on phases its backend enum does not have. One owner per contract makes that class **unrepresentable** rather than merely detectable. + +## Contract tiers — by what kind of boundary it is + +| Tier | Is | Example | +|---|---|---| +| **types** | types in a binary | — | +| **schema** | an API or interface format | the OpenAPI contract, the callosum envelope | +| **fixture** | a durable storage format | the `x-journal-contract` files, the path-resolution vectors, the identity vectors | + +⛔ Not graded by enforcement strength — graded by the **kind** of boundary. A vector is a type of fixture. + +## Writing and reading + +🔴 **Write new, read old.** A new implementation does not have to write what Python wrote, as long as there is a contract for what it *does* write. When it **reads**, it stays compatible with the older format. Bit-identical writes are **not** required. ⛔ The one thing that is never acceptable is leaving older journal data unseen. + +🔴 **Identity is written beside the data, never re-derived from a name or a position.** Where a derived value doubles as a *lookup key*, writing it differently forks the namespace instead of just changing a format. **The answer is not to teach every reader two spellings — it is to stop deriving:** persist the identity next to the data, and let the derived string be a **label** that nothing resolves by. + +`device.json` in a segment directory is the worked example: the stream name became human-friendly precisely *because* it stopped being load-bearing. + +⚠ **Read-compat still applies to old data** — a reader handles records written before the identity was persisted. But **a writer never re-derives.** + +Known instances and what is left of each: + +| Instance | State | Residue | +|---|---|---| +| `entity_slug()` | `id` is written into `entity.json` on create (`think/entities/journal.py:59`, `:201`); `loading.py:86` reads `data.get("id") **or** entity_slug(name)` — the derivation is a **fallback** | the fallback branch, and the `entities/{entity_id}/` directory name | +| `ambiguity_id` | required on the row and hard-rejected when absent (`ambiguities.py:157-159` raises `missing ambiguity_id`) | essentially none | +| `sentence_id` | 🔴 **the real instance** — a 1-based ordinal recomputed at read time and stored nowhere | all of it | + +**The work: make the written identity mandatory, delete the derivation fallback, persist `sentence_id`.** + +## 🔴 Reserved words — two live collisions + +Two words mean an internal thing and an owner-facing thing at once, and **both collisions are already in shipped code.** + +| Word | ✅ Means ONLY | ⛔ Never means | Use instead | +|---|---|---|---| +| **health** | the **journal system's** health — is it running, is everything processed | the owner's physical health | **body** | +| **activities** | the **internal** facet activity model (`facets/{facet}/activities/{day}.jsonl`) | the owner's physical movement | **body motion · fitness · kinetics** | + +**All owner physiological data is `body`** — *body data*, *body records*, *body sources*. + +⚠ `apps/body/routes.py:1985` defines `_activity_analysis` over *wearable* activity while `think/activities.py` owns the internal model — one word, two meanings, live today. + +⛔ **Consequence for plate names:** `P-journal-health` is **system** health written durably. Owner body records are **`P-journal-body`**, a different plate. Never fold them. + +## Changing these files + +**The plate set and the strand definitions are fixed.** Within a boundary you own, correct a fact, refine a contract note, or record what must be carried forward — that is expected and welcome. + +⛔ **Do not, on your own initiative:** add, split, rename or re-scope a **plate**; change a **strand's** definition, tier, or which end owns its contract; define or re-scope a **cable**. Those are decided outside this repo. If the code contradicts a definition here, that is worth raising — say so plainly rather than quietly widening the definition to fit. + +⚠ **Every count and inventory here is a floor, not a census.** Two string-keyed dispatch registries resolve modules by name at call time with **zero static import edges** — `FORMATTERS` (`think/formatters.py:139-265`, reaches 12 modules) and `EDGE_SOURCES` (`think/edge_sources.py:39-73`, reaches 6), both via `import_module` + `getattr`. No import graph over this tree is complete. **A boundary is not clean until you count its callers.** diff --git a/docs/conversion/cables.md b/docs/conversion/cables.md new file mode 100644 index 000000000..ac0d156ff --- /dev/null +++ b/docs/conversion/cables.md @@ -0,0 +1,33 @@ +# cables — whole use cases, end to end + +**A cable connects a whole use case from one major system to another**, through several strands with plates between them. Named `C:plate:plate:plate…` — every plate in the path, in order. + +Definitions and vocabulary: [`README.md`](README.md). Siblings: [`plates.md`](plates.md) · [`strands.md`](strands.md). + +⚠ **A cable is a destination, not a work unit.** Its strands are. This file exists so the destinations stay visible while strands are built. + +--- + +## `C:device-ingest:segment-media:segment-sense:thinking:journal-segment` + +**A device's capture becoming a fully processed, thought-out result in the journal** — from device ingest, through sense media processing, through thinking, down to journal storage. The spine of the product. + +Strands: `S:device-ingest:segment-media` → `S:segment-media:journal-segment` → `S:segment-sense:journal-segment` → `S:segment-sense:thinking` → `S:thinking:journal-thinking` + +⚠ Every strand in this cable is Tier 1, so it completes when the core does. + +## `C:journal:index:web:cli-sol` + +**Journal content becoming answerable to an owner or a talent** — from journal storage, through the formatting layer, to the indexer, to the convey API, to the `sol` CLI. + +Strands: `S:index:format` → `S:journal:index` → `S:web:journal` → `S:cli-sol:web` + +--- + +## Candidate cables + +Recorded so they are not silently dropped. ⛔ None is authorized as a destination yet. + +- **link-to-trust** — `C:device-link:journal:web` — a device pairing and becoming visible to the owner as a trusted device. +- **body ingest** — `C:body-source:journal-body:web` — owner body data arriving and becoming readable. +- **honest state** — `C:system:system-health:journal-health:web` — the product reporting truthfully on itself, end to end. diff --git a/docs/conversion/plates.md b/docs/conversion/plates.md new file mode 100644 index 000000000..edf554aeb --- /dev/null +++ b/docs/conversion/plates.md @@ -0,0 +1,168 @@ +# plates — the boundaries strands connect to + +**A plate is a boundary of the journal where strands connect from other plates.** ⛔ It has no "sides" — the contract lives at one *end of a strand*, and by convention that is the second plate named in `S:a:b`. + +Definitions and vocabulary: [`README.md`](README.md). Siblings: [`strands.md`](strands.md) · [`cables.md`](cables.md). + +⛔ **Reserved words:** `health` = journal system health **only**; owner physiological data is **`body`**. `activities` = the internal facet model **only**; owner movement is **body motion / fitness / kinetics**. + +--- + +## `P-journal` + +The on-disk journal. 🔴 **Deliberately coarse.** Talk about `P-journal` at a high level; when doing work, talk about the sub-plate. + +| Sub-plate | Holds | +|---|---| +| `P-journal-segment` | the segment directory — chronicle day/stream/segment | +| `P-journal-stream` | stream declarations and state | +| `P-journal-thinking` | talent outputs and run logs | +| `P-journal-health` | durable **system**-health records and logs | +| `P-journal-body` | owner **body** records — never "health" | + +⚠ **6 formats carry a real `x-journal-contract`; roughly 20 more on-disk shapes are defined only by a Python writer**, 8 of them undocumented anywhere. The path spine itself — including the default-stream transposition that `layout.json` never states — is Python-only. + +## `P-device-link` + +Link identity, security, mTLS. + +⚠ **The machine-checked part of this boundary is the QR bootstrap only.** The `spl` conformance corpus covers pair-link forms `0x04`/`0x05`/`0x06`, address admission and ca_fp tags. The pairing ceremony, session lifecycle, framing and token JWTs are prose. + +🔴 **`convey/secure_listener/` is this plate's authorization boundary** — TLS termination, the in-handshake fingerprint check, admission, framing, mux, and inline WSGI dispatch. It is Python. + +**Identity:** the journal's own identity is a `jid` derived from its CA's P-256 SPKI. A **device's** identity is a **`did`** — the device certificate fingerprint, `sha256:` over **certificate DER** (`cert_fingerprint()`, `think/link/ca.py:269`), which is the value already stored per client. + +⛔ **Two kinds of fingerprint exist here on purpose and must never be crossed.** `cert_fingerprint()` hashes the **certificate DER** — that is the `did`. `spki_fingerprint_sha256()` (`ca.py:66`) hashes the **DER SPKI** and is the `0x06` pair-link CA pin; its own docstring says *"not cert DER."* Reversing them silently breaks off-LAN pairing. + +✅ The `did` is stable for the device's life — client certificates are 10-year and there is no renewal path in `think/link/`; revocation is via `authorized_clients.json`. ⚠ If a renewal path is ever added, a device's `did` changes and prior segments carry the old one. + +**Carry forward, non-negotiable:** the certificate fingerprint is the identity and the label is only a label · revocation is the ledger, not the certificate · **an unreadable `authorized_clients.json` authorizes nobody** · loading never rewrites the ledger. + +🔴 **The mark derivation is the most dangerous thing in this plate to reimplement.** `think/link/mark.py` calls itself the frozen journal identity contract. Argon2id v0x13, `t=3`, `m=65536`, `p=1`, `hash_len=32`, salt `solstone-journal-mark-v1`, secret = the jid's 16 raw bytes; seven big-endian u32 words drive `pick`/`pick_distinct` over 60 icons, 16 colours, 7,776 words, plus two rotation bits rendered `45 if bit else 0`. ⛔ **`_ICON_NAMES = tuple(_GLYPHS)` makes the icon index the JSON insertion order of `mark_assets/glyphs.json`.** That order is *currently identical to sorted order*, so an implementation that sorts passes every derivation vector and diverges the first time a glyph is appended out of alphabetical order. ⚠ The colour list is spectrum-ordered — that is the tell that file order is the rule. **The mark is what an owner looks at to confirm they are pairing with their own journal; a subtly-wrong reimplementation presents as evidence of compromise.** + +## `P-device-ingest` + +The ingest **API**. Deliberately separate from `P-segment-media`. ⚠ 9 published operations, 7 named client consumers. + +## `P-segment-media` + +An ingested, not-yet-processed segment. + +## `P-segment-sense` + +Media processing. ⚠ Emits **two** strands with different contracts — processed data back to storage, and callosum events out. + +## `P-index` + +The SQLite index. **Ephemeral by design and always rebuildable — that property is required, not incidental.** ⚠ The index schema needs architecture work. + +🔴 **Not already Rust.** `think/indexer/native.py:6-11` says only CLI command writes went native, that *"query and interactive search stay in Python,"* and names eight remaining in-process Python writers. `indexer/journal.py` is 1,693 live lines; `edges.py` is 1,263. + +## `P-format` + +Consistent formatting of **structured journal data** for its consumers — the indexer and the convey apps. + +🔴 **No import graph shows this plate's fan-out.** `FORMATTERS` (`think/formatters.py:139-265`) reaches 12 modules by **string key** via `import_module` + `getattr` (`:283-286`), with zero static import edges. It is the de facto read-side inventory of every on-disk shape, and it lives only in Python. + +## `P-thinking` + +🔴 **A grouping plate.** Holds **two contracts: `generate` and `cogitate`**. Everything connects to it. `P-local`, `P-BYO` and `P-SPP` sit behind it. + +⚠ The runtime preamble every talent is written against exists as a **sha256 only** in the cross-language fixture — drift is detectable, the text is not reproducible. + +## `P-local` + +Local model runtime, inside the security boundary. ~9k lines of live install/runtime machinery with zero dead modules. + +⚠ The boundary is loopback HTTP **plus a durable record** — `health/brain.json`, written and validated by `providers/brain_state.py` (2,692 lines, HMAC-fingerprinted, refresh lease, three lanes). Calling it a types boundary drops the durable half. + +⚠ **Do not carry** the in-loop USD ceiling — it fabricates a cloud price for local runs and force-stops them. Keep the **context**-fraction half of the same function. + +⚠ **Do not carry the loopback "guard" as written.** The `cmd` list is built with `"--host", "127.0.0.1"` hardcoded and the next statement is `if "0.0.0.0" in cmd: raise` — a membership test against a literal set three lines above. It cannot fire at runtime and would miss `--host=0.0.0.0`, `::`, or a hostname. ✅ **The honest statement of the same invariant is `services/spp_transport.py:190-197`** — it binds `("127.0.0.1", 0)` and writes down why same-UID reachability is acceptable and which constraints are actually load-bearing. Carry that one. + +## `P-BYO` + +An owner's own token to a known provider, or their own OpenAI-compatible URL and token. ⛔ **Egress.** Will split per provider. + +## `P-SPP` + +Confidential hosted processing. ⛔ **Egress.** Attested, non-retained. ⚠ **Fails closed by the absence of a fallback branch** — a refactor that tidies it can open a downgrade path with no test going red. Make "no downgrade path" an explicitly tested invariant. + +## `P-speaker-id` + +Per-statement embeddings → speaker fingerprints. ✅ A native kernel already sits behind an argv+stdio wire with an algorithm-identity handshake. + +⚠ `.npz` sidecar and `speaker_labels.json` are Python-only. ⚠ The `sentence_id` join key is recomputed at read time and stored nowhere. ⚠ The voiceprint corpus is real-person biometric data. 🔴 **Years of voiceprints must survive with no re-teach** — that is a shipped promise. + +## `P-entity` + +The entity store. ⚠ **47 distinct production modules** import `think.entities`, fanning in from at least four plates. + +🔴 **This store fails by BRICKING, not degrading** — the history tree and `ambiguities.jsonl` are self-validating, so a subtly-wrong implementation destroys the store rather than producing worse results. All three algorithm-identity hazards — `entity_slug()`, `casefold`, `rapidfuzz` at threshold 90 — live behind this boundary. + +## `P-facet` + +Facets and their per-facet contents, including facet-scoped entity and speaker material. ⛔ Distinct from `P-entity`: the entity store is the identity-bearing thing, the facet is the organizing structure over it. + +## `P-journal-config` + +`journal/config/journal.json`, read by **30 production modules**. Durable, `0o600`, mutated under `hold_lock` + `atomic_replace` with an explicit transaction type. + +✅ **Carry forward — this is the house style, not a local quirk.** `CorruptConfigError` (`think/utils.py:53-68`): a **missing** config returns deep-copied defaults; a config that **exists and will not parse raises**, in owner voice — *"I couldn't read your settings file… Your settings were NOT changed."* Two deliberately different postures on two failure modes, never silently substituting on the dangerous one. + +⚠ **The config file being the source of truth is an external commitment made in writing.** A contract-breaking pass here can violate it by accident. + +## `P-journal-retention` + +The logic that decides what raw media is retained, and what logs are retained for how long. `think/retention.py` (708 lines) **irreversibly deletes owner raw media**; `log_retention.py` (1,006) prunes logs. + +🔴 **Carry forward:** *read one extraction file strictly enough for irreversible deletion* (`retention.py:110`) — reads at most two lines and treats any `OSError` / `JSONDecodeError` / non-dict as **`"malformed"`, never as "empty, safe to purge"**, with an explicit guard at `:136-139` against a stray marker key making a header-only file look chunk-bearing. Plus `resolve_segment_gate`: `.npz` without `talents/speaker_labels.json` ⇒ incomplete. + +⚠ Retention imports `apps/backup/copy` and `think/offload.py` imports back out of retention, so deletion is coupled to backup. + +## `P-web` + +The journal web service — human interface, web apps, **and the API**. + +⛔ **Access model: 100% of `P-web` is either `localhost:5015` (human web, or a same-device CLI) or an authorized linked device. There is no third way in — the boundary *is* the authorization.** ⛔ A third access path is not a decision to make inside this repo. + +⚠ 77 of 135 routes uncontracted including 20 state-changing; `/api/shell` — the most load-bearing endpoint in the owner UI — is not in the contract. ⛔ Zero of 177 contracted operations declare authentication, so a port generated from the contract inherits an unauthenticated surface. + +## `P-CLI` + +Splits almost immediately. + +| | Reach | May do | +|---|---|---| +| `P-CLI-sol` | any device | **API only**, over a link | +| `P-CLI-journal` | the same journal device only | **modify the journal directly**, or the API over localhost | + +## `P-system` + +Operations for managing asynchronous activity — starting things, running things. + +**Carry forward:** the task-request refusal **classifies rather than guesses** — it distinguishes `"wedged"` (runtime > 2× cap) from `"still_running"` and emits a skip event with both refs and the reason. A refusal that says *which* refusal it is. + +## `P-system-health` + +Health of running things — current status, in-memory. ⛔ **System** health, never owner body data. + +🔴 The per-day health JSONL grammar is **entirely Python string literals**. The callosum envelope likewise — two constants and a docstring carrying the whole control plane. + +## `P-body-source` + +Owner **body** data arriving from outside — Oura, Apple Health. ⚠ **Ingress, not egress** — nothing of the owner's journal leaves. + +⚠ The normalized shard format is defined only by its **reader** (`apps/body/routes.py:311-334`), and `imports/health-dedupe.sqlite` uses raw `sqlite3` entirely outside `journal_io` discipline. 🔴 **Excluded from every backup with no rebuild path** — a restore silently empties the owner's body history. + +--- + +## ⛔ Egress — where the covenant applies + +Journal and devices are **one secure environment**. No per-plate privacy tracking inside it; transport security is already covered. + +**Actual egress — three:** `P-BYO` (the owner's own key, owner-directed) · `P-SPP` (attested, non-retained) · **support requests** — ⚠ the `_SECRET_*` redaction in `apps/support/diagnostics.py:29-50` is the **last thing** between a journal config and an external service. + +**Blind by construction, therefore not egress:** relay transit · push notifications · encrypted backups. + +🔴 **Push is only reimplemented in an end-to-end encrypted form** — the journal encrypting and the receiving device decrypting with the link cryptographic identities. ⛔ The current plaintext path, which carries journal-derived chat content to a push service and which unpairing does not revoke, does **not** come across. diff --git a/docs/conversion/strands.md b/docs/conversion/strands.md new file mode 100644 index 000000000..5742ef17c --- /dev/null +++ b/docs/conversion/strands.md @@ -0,0 +1,185 @@ +# strands — the work units + +**A strand is the minimum viable path connecting two plates, in Rust, robust.** It may be bi-directional; its contract lives at exactly **one end**, and by convention that end is written second in the name. + +Definitions and vocabulary: [`README.md`](README.md). Siblings: [`plates.md`](plates.md) · [`cables.md`](cables.md). + +**Read your strand's section, not the file.** Organized for surgical section edits. + +⚠ **Every count here is a floor, not a census** — see `README.md` on the two string-keyed dispatch registries that defeat counting. + +⛔ **Reserved words:** `health` = journal system health only; owner physiological data is `body`. `activities` = the internal facet model only. + +--- + +## Tier 1 — the core + +### `S:device-link:journal` +**Connects** `P-device-link` → `P-journal` · **Owner** `P-journal` · **Tier** fixture + schema + +Link identity. A device proves who it is; the journal decides whether it is authorized. + +⚠ **24 production (non-test) modules import `solstone.think.link`**, which makes it the least isolable thing in the tree — an argument for converting it early rather than deferring it. + +The identity derivation, the two fingerprint kinds, the `did`, and the mark's exact parameters are in [`plates.md`](plates.md) § `P-device-link`. ⛔ Do not restate them here. + +**Carry forward:** the certificate fingerprint is the identity, the label is only a label · revocation is the ledger, not the certificate · **an unreadable `authorized_clients.json` authorizes nobody** · loading never rewrites the ledger. + +### `S:device-ingest:segment-media` +**Connects** `P-device-ingest` → `P-segment-media` · **Owner** `P-device-ingest` · **Tier** schema + +Segments arriving from a device. This is the **ingest envelope** — a wire shape, not bytes on disk: `observe/protocol.schema.json` carries `file_kind: "ingest_envelope"` and `producer_write_paths` of the ingest endpoint. + +⛔ **The client no longer asserts its own identity.** The journal authenticates the certificate at the transport, derives the `did`, and records it. Nothing routes, authenticates, or names a stream from a client-supplied string. + +**Carry forward:** create-exclusive byte writes that never overwrite — on collision compare SHA-256 and record the already-held case · content identity **refuses rather than guesses** · the segment-key timezone semantics — `HHMMSS` is **device-local wall clock, not UTC**. + +⚠ The audited client bundle and the plate's declared operations are **different sets**. The bundle omits the owner's location-data delete route — the covenant-critical one — and adds three operations belonging to other plates, so the capture clients contractually depend on chat and callosum. + +### `S:segment-media:journal-segment` +**Connects** `P-segment-media` → `P-journal` · **Owner** `P-journal` · **Tier** fixture + +Raw media landing durably, and the **segment sidecars**. + +**`device.json`** — optional; when present it carries at minimum the **`did`**, plus optional ephemeral device metadata the device supplies (battery level, screen lock, posture) as **free-form fields opaque to the journal and validated only for the required `did`**. + +⛔ **`device.json` is journal-authored, never client-uploaded.** Reserved sidecar names are never written from client bytes and never appear in segment listings as held. The device supplies its fields as data over the API; the journal writes the file. + +🔒 **Nothing is ever looked up by stream name. Attribution always reads the segment.** The stream name is a human-friendly, unique, cross-platform-filesystem-safe label — a deterministic projection of a display name like `iPhone (2)` to `iPhone_2`. One device may own several streams (a watch app via a phone is a different stream). + +⚠ **The projection's uniqueness must be case-insensitive.** APFS and NTFS fold case by default; ext4 does not, so two names differing only in case collide on two of three platforms and not on the one most likely to be developing them. Also exclude `<>:"/\|?*`, trailing dots and spaces, and the Windows reserved device names. + +**Carry forward:** **unresolved error → hold raw**, via `exit 69` — the handler writes nothing, leaves the input, and the scanner records *neither* success nor failure. The deferral emitter deliberately swallows its own bus failure so a down bus cannot turn a deferral into data loss. + +### `S:segment-sense:journal-segment` +**Connects** `P-segment-sense` → `P-journal` · **Owner** `P-journal` · **Tier** fixture + +Processed sense output written back. + +🔴 `_solstone_processing` has a version string and **no schema**, and is absent from both sibling schemas that enumerate every *other* header key. ⚠ **Read by five planes** — the fifth is `think/retention.py:133`, the highest-stakes one, because it decides irreversible deletion. + +**Carry forward:** **terminal-empty written before the raw is unlinked** — die between them and the file survives with a terminal marker, never the reverse · atomic promote through a same-dir temp, header **last** · detections stored raw, filtered at read time. + +### `S:segment-sense:system` +**Connects** `P-segment-sense` → `P-system` · **Owner** `P-system` · **Tier** schema + +Callosum events emitted as processing happens. + +There **is** a published machine-readable registry — `CALLOSUM_REGISTRY` (`convey/contract/assemble.py:43-99`), emitted as `x-callosum-registry`, 11 tracts and ~60 events, plus two published SSE operations. ⚠ **It is already drifted three ways:** events produced but undeclared · events emitted and prose-documented but absent from the registry · events declared with no literal producer. ⚠ The drift count is a floor — tracts passed through variables are invisible to a literal grep. + +### `S:segment-sense:journal-segment-events` +**Connects** `P-segment-sense` → `P-journal` · **Owner** `P-journal` · **Tier** fixture + +The **durable** half of the callosum contract, split from the wire half above. The whole bus envelope is appended verbatim into `{day}/[{stream}/]{segment}/events.jsonl`, with readers in `think/segment.py` and `apps/observer/utils.py`. + +⚠ **That writer bypasses `journal_io`** — bare `open(…, "a")` — and swallows failures at debug level. ⛔ The segment-sidecar family wants one write discipline; do not let a new sidecar inherit this one. + +### `S:segment-sense:thinking` +**Connects** `P-segment-sense` → `P-thinking` · **Owner** `P-thinking` · **Tier** schema + +⚠ The talent event vocabulary is **NDJSON on stdout** — a real inter-process wire and the run-log format that gets persisted — defined as `total=False` TypedDicts with no schema, fixture, or validator. **The shape most likely to be lost silently.** + +### `S:thinking:journal-thinking` +**Connects** `P-thinking` → `P-journal` · **Owner** `P-journal` · **Tier** fixture + +Talent output landing durably. Carries the failure semantics: retries, back-offs, days being complete, segments being complete. + +**Carry forward:** the talent-use lifecycle where **the filename is the lock and the state** — exclusive `open(…, "x")` on `{use_id}_active.jsonl` is the claim, rename on completion, and on restart every leftover `_active` is terminalized with an error so an interrupted talent is never indeterminate. + +🔴 **`_active.jsonl` is not only a talent convention — it is a deletion gate in two other subsystems.** Seven production readers, including `think/log_retention.py:368` (skips pruning them) and **`think/retention.py:199` (treats presence as "segment incomplete, do not purge raw media")**. A cleaner claim filename is a format change on the writer side and a **silent capability loss on the deletion side.** + +### `S:thinking:local` +**Connects** `P-thinking` → `P-local` · **Owner** `P-thinking` · **Tier** schema + fixture + +The local model lane. ⚠ Not a types boundary — it is loopback HTTP **plus a durable record**. See [`plates.md`](plates.md) § `P-local` for the two things not to carry. + +### `S:journal:index` +**Connects** `P-journal` → `P-index` · **Owner** `P-index` · **Tier** fixture + +🔴 **Not already Rust** — see [`plates.md`](plates.md) § `P-index`. + +⚠ **Fan-in is nine writers across six plates**, not one: `think/backup/restore.py:31`, `convey/chat_stream.py:182`, `think/importers/cli.py:27`, `think/day_accumulator.py:12`, `apps/observer/prune.py:27`, `apps/observer/share_delete.py:19`, `think/entities/merge.py:2043`, `think/segment.py:19`. + +⚠ Plus an **invisible runtime dependency** on `apps/speakers/edges.py` and the entity store via the `EDGE_SOURCES` registry, so the index build runs `find_matching_entity` — the `rapidfuzz`-at-threshold-90 path. + +**Carry forward:** the index is **always rebuildable** — ephemeral by design, and an interrupted update never leaves a partial result. ⚠ That property is required, not incidental. + +### `S:index:format` · `S:web:format` +**Owner** `P-format` in both · **Tier** schema + +The indexer's and the convey apps' consumption of consistently formatted structured journal data. ⚠ `P-format` owns both because it is the one-to-many end — it serves all consumers and cannot negotiate per-consumer. ⛔ The name encodes contract ownership, not data flow. + +### `S:index:*` — the read / query path +**Owner** ⚠ unassigned · **Tier** schema + +Nine production readers across search, tools, voice, talents and connections. The `search_journal` / `search_counts` / `known_agents` interface — which `P-thinking` calls **directly, in-process, on every talent that searches** — belongs to no strand yet. + +### `S:*:P-entity` · `S:*:P-facet` +**Owner** `P-entity` · `P-facet` · **Tier** fixture + +See [`plates.md`](plates.md) — this store fails by **bricking**, not degrading. + +### `S:*:journal-config` +**Owner** `P-journal-config` · **Tier** fixture + +See [`plates.md`](plates.md) § `P-journal-config` for the fail-closed posture that is the house style. + +### `S:web:thinking` — chat +**Owner** `P-thinking` · **Tier** schema + +The primary owner-facing use of the model. `convey/chat.py` (2,532 lines) + `chat_stream.py` (512) + `convey/sol_initiated/` (1,034). It **spawns talents**, so it is a producer into `P-thinking`, and it is in the audited native-client bundle — the capture clients depend on it. + +⚠ **Push cannot be scoped until this exists** — push's trigger is the **chat tract** (`push/triggers.py:63-64`), so push is a callosum consumer downstream of the chat orchestrator, not a standalone journal→device path. + +### `S:*:system` — the command channel +**Owner** `P-system` · **Tier** schema + +`_handle_task_request` takes `message["cmd"]` off the unix socket and hands the argv to the task queue. **This is how the scheduler, importers, backup, and the sense/think pipeline all cause work to run** — six production producers. ⚠ Distinct from liveness/status. + +### `S:journal:establish` +**Owner** ⚠ needs assigning · **Tier** fixture + +First-run journal establishment. **Creates the identity root** that `S:device-link:journal` depends on: the mark-lock route promotes the staged CA and persists the instance identity. + +⚠ Eight `/init/*` routes are session-gate-exempt and admitted **before** the journal-is-active check. That is the same `localhost:5015` human-entry basis as everything else on `P-web`, before a session exists to gate — **not** a third access path. + +--- + +## Tier 1 — retention + +`P-journal-retention` connects through three strands, each a different contract: + +| Strand | For | Owner | Tier | +|---|---|---|---| +| `S:journal-retention:journal-config` | the **posture / settings** it reads | `P-journal-config` | fixture | +| `S:journal-retention:system` | **when it runs** | `P-system` | schema | +| `S:journal-retention:journal` | **tending the files** — changes, and recording status | `P-journal` | fixture | + +Where retention is the provider it owns the contract — it is the one-to-many end with ten production call sites. See [`plates.md`](plates.md) for the irreversible-deletion carry-forward. + +--- + +## Tier 2 + +| Strand | Owner | Note | +|---|---|---| +| `S:system:system-health` | `P-system-health` | Liveness and current status of running things | +| `S:system-health:journal-health` | `P-journal` | 🔴 Per-day health JSONL grammar is entirely Python string literals | +| `S:web:journal` | `P-journal` | ⛔ 100% of `P-web` is `localhost:5015` or an authorized linked device | +| `S:cli-journal:journal` | `P-journal` | Same-device only; may modify the journal directly | +| `S:cli-sol:web` | `P-web` | Any device, API only, over a link. ⚠ May share `S:web:journal`'s contract | +| `S:segment-sense:speaker-id` | `P-speaker-id` | A refinement of sense | +| `S:speaker-id:journal-facet` | `P-journal` | ⚠ **Years of voiceprints must survive with no re-teach** — a shipped promise. **Carry forward:** the label-source resolver returns *ambiguous* and warns the owner rather than picking a source | +| `S:body-source:journal-body` | `P-journal` | Owner **body** data — ingress, not egress. ⚠ The shard format is defined only by its reader; 🔴 excluded from every backup with no rebuild path | +| `S:file-import:journal-segment` | `P-journal` | Generic import. ⚠ **A second write door into `chronicle/`** that consults no contract and writes with bare `write_bytes` — the create-exclusive guarantee is a property of **one** of the two doors | + +## Tier 3 — additive, absent until approved in + +`S:thinking:byo` — ⛔ egress, the owner's own key · `S:thinking:spp` — ⛔ egress, attested. ⚠ Fails closed by the *absence* of a fallback branch; make "no downgrade path" an explicitly tested invariant. + +## Not yet placed + +- **push** — ⚠ cannot be scoped until `S:web:thinking` exists; its trigger is the chat tract. Only reimplemented end-to-end encrypted. +- **encrypted backup** — blind by construction, but ⚠ **retention imports it**, so a core deletion path depends on it. 🔴 The 64-char recovery key **is** the repository password. **Carry forward:** the exclusion-list comment records a paid-for lesson about basename-at-any-depth matching that a rebuild would otherwise re-learn; and `brain.json` / `scheduler.json` / the supervisor-ready marker are excluded deliberately, so post-restore provider and scheduler state is an intentional blank. +- **support request** — ⛔ real egress; the `_SECRET_*` redaction is the last thing before an external service. +- **merge / transfer** — deferred. ⚠ `think/merge.py:30-33` imports three **private** functions out of `think/entities/merge.py`, so the frozen thing depends on the unstranded store.