# Human review and training-data custody

## Boundary

Review is a private, append-only human judgment surface over immutable source prompts and exact completed candidate runs. It is the evidence and dataset-custody layer around training. It does not train models, mutate candidate output, infer missing prompt context, or grant a checkpoint deployment authority.

Tinker or another explicitly configured trainer owns optimization. The immutable adapter catalog owns released artifacts and coordinated activation. Review owns:

- complete prompt/evidence custody;
- blinded presentation of exact candidate runs;
- judgeability, pairwise preference, tie, correction, and skip decisions;
- supersession without destructive edits;
- privacy- and provenance-gated training export;
- coverage and exclusion evidence for a review campaign.

## Review prompt

`stream.thought.source.review.prompt@1` is a source event. Its payload contains:

- a campaign id, version, and human label;
- the complete prompt shown to candidate models;
- optional bounded evidence required to judge the response;
- one criterion id/version, label, instructions, allowed reason codes, and allowed response tags;
- the candidate agent ids expected to run over the event;
- whether this already-reviewed synthetic/public prompt is eligible for external training export.

The event privacy is authoritative. A prompt may set `externalExportEligible: true` only when its event is `public-source`. Private and sensitive prompts can be reviewed but are not declassified by the browser.

The prompt event is the shared trigger for every candidate run. Pairwise candidates generated from different trigger events are not comparable training examples even when their visible text happens to match.

## Candidate output

Review candidates use canonical output contract `stream.thought.output.review-response@1`, containing one bounded `response` string. Candidate runs must be completed, must name the same review-prompt trigger, must come from distinct candidate agents declared by that prompt, and must each have exactly one contract-valid output event. The run and context receipts must prove the prompt was the only input, was included whole, was not projected through `payloadFields`, gained no tool evidence, and had no external action authority.

The model, provider, execution adapter, learned adapter, catalog generation, prompt hash, and trace remain ordinary run provenance. Candidate output is never copied into the review-item event.

## Review item

`stream.thought.review.item.created@1` freezes one exact pair:

- review prompt event id;
- the two candidate run ids, output event ids, and digests of every training-relevant run/context/provenance field;
- a deterministic blinded display order;
- the criterion identity inherited from the prompt.

Creation validates all references and joins prompt, run, and output privacy. The item id is deterministic over the prompt, criterion, and unordered pair, so repeated materialization addresses the same event. A later retry or replacement run requires a new review item.

Every later queue, decision, and export path resolves the frozen output event ids and recomputes the candidate receipt digests. A changed completed-run row, output pointer, context manifest, model, prompt hash, adapter identity, catalog identity, or privacy fails closed instead of silently rebinding a human label. Before the first decision, the Review projection withholds run ids and provenance. This is presentational blinding, not a claim that an operator with direct Jazz access cannot inspect underlying rows.

## Review decision

`stream.thought.review.decision@1` records exactly one disposition:

- `prefer`: one candidate wins, with `slight` or `strong` strength;
- `tie`: both are adequately equivalent for the criterion;
- `correct`: neither is acceptable and a contract-valid replacement response is supplied;
- `underdetermined`: the evidence cannot identify the preferred response;
- `malformed`: the review item or rubric is defective;
- `skip`: no judgment is supplied.

Every decision may carry bounded confidence, at most two allowlisted reason codes, at most two allowlisted response tags, and an optional bounded note. `prefer` must name one candidate run. `correct` must supply a replacement response. Other dispositions must not smuggle a preference or replacement.

Decisions append. A later decision may explicitly name the prior active decision it supersedes. The active projection chooses the latest valid decision for an item and treats older concurrent leaves as inactive, so a process race cannot create multiple exported labels. Browser retries carry a random submission id and are idempotent.

Judgeability is part of the data, not failed labor. `tie`, `underdetermined`, `malformed`, and `skip` never become DPO examples. They remain campaign-quality evidence and can prevent an invalid benchmark from quietly acquiring forced labels.

## Training eligibility

Review separates three facts:

1. a human decision exists;
2. the decision is suitable quality evidence;
3. its full prompt/output/provenance chain is authorized for external training export.

The browser's **use for training** control can make a `prefer` or `correct` decision quality-eligible. It can make the decision externally eligible only when the prompt was preauthorized for external export and the complete prompt, item, candidate-run, output, and decision privacy join is `public-source`. The browser has no private declassification control.

The review exporter emits `thoughtstream.training-example.v4` only for active externally eligible `prefer` and `correct` decisions:

- exact public review prompt and optional evidence;
- criterion identity and bounded human judgment metadata;
- contract-valid chosen response;
- one or two exact rejected candidate responses;
- empty trajectories: Review labels bind frozen prompt/response pairs, not trace projections that can gain later operational chunks;
- complete model, execution-adapter, learned-adapter, and catalog provenance for each candidate;
- campaign id/version.

It omits source/run/event ids, actor/route/correlation/idempotency ids, notes, browser submission ids, cookies, CSRF tokens, capability signatures, private checkpoint references, and arbitrary source payload fields. Legacy v3 examples remain valid and can coexist in a v4 dataset manifest.

## Browser authority

The inspector remains read-only by default. Review writes are a separately configured capability:

- only the allowlisted OAuth browser session can obtain write state and submit a review decision;
- Basic fallback remains read-only and never receives a CSRF token;
- the public proxy accepts POST only on the exact review-decision route, validates a bounded JSON body and the session CSRF token, then signs the exact method/path/body with a separately injected review capability;
- the loopback inspector verifies the signed request, freshness, and one-time nonce before reading the body or appending an event;
- no cookie, Authorization header, OAuth token, DID, CSRF token, or generic Jazz mutation reaches the inspector;
- absent or mismatched capability configuration leaves every data route GET/HEAD-only.

The write endpoint accepts only the fixed decision schema. It cannot create prompts, create review items, run models, export datasets, activate adapters, publish, send, or mutate arbitrary events.

## Interface

The private **Review** tab shows unresolved items first. Each card presents complete prompt/evidence and two blinded candidates. The first controls are judgeability dispositions. When judgeable, the reviewer can record a directional strength, tie, or correction, then optional bounded tags and confidence. Model and adapter provenance is revealed only after submission or from the reviewed-history view.

Changing a decision creates a superseding event. The UI shows current decision, previous-decision count, campaign coverage, and whether the active decision is evaluation-only or export-eligible. It does not show aggregate preferences before submission.

## Initial training workflow

1. Import fully specified public-safe review prompts split by scenario family.
2. Generate multiple real candidate runs from the frozen parent and candidate configurations.
3. Materialize exact blinded pairs after runs settle.
4. Run a small UI canary with training disabled.
5. Review judgeability and pairwise decisions.
6. Export only active, preauthorized public-source examples.
7. Train outside thought stream.
8. Return candidate checkpoint and held-out evaluation receipts.
9. Promote only through the immutable adapter-release and coordinated-activation contract.

Hard authorization, routing, privacy, and formatting contracts remain deterministic gates outside the preference objective.