From 3ce0f0778272e53f2f5df39e4afd861dbbf85c08 Mon Sep 17 00:00:00 2001 From: Refinement Systems Date: Sun, 13 Sep 2026 18:20:16 +0200 Subject: [PATCH] notes update --- NOTES.md | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/NOTES.md b/NOTES.md index dcc3c02..81a8204 100644 --- a/NOTES.md +++ b/NOTES.md @@ -376,6 +376,82 @@ REF_ORDER-aware role-naming prompts are all in `scripts/smoke.sh` / `scripts/sweep-klein.sh`. What is still pending is the GPU validation itself (prompt ladder + order A/B under dual-ref, then the reproject A/B). +**Update, later the same day: the validation ran — see the next section. +norepro freezes motion; the order/prompt axes are dead; reproject is +load-bearing.** + +## Klein dual-ref GPU validation: norepro freezes motion; reference gain measured (2026-09-13) + +**Result: dual-ref WITHOUT reprojection cannot carry motion — a regime +mismatch, not a bug. Reprojection is load-bearing.** Found during the first +`sweep-klein.sh` dual-ref run (stopped after the `enhance-slight` leg, so +partial legs could be analyzed), then pinned down with 4b debug probes and a +tail-based reference-gain readout. + +**Freeze ledger** (all `--mode stateful`; motion stops within a few frames +of the start — composition locks while texture keeps chattering): + +- 9b `prompt-neutral` `…_dualref-norepro` — freezes. +- 9b `prompt-enhance-slight` `…_dualref-norepro` (role-naming prompt) — freezes. +- 4b frame-first, empty prompt, norepro, 30 frames — freezes. +- 4b frame-first, role-naming prompt, norepro, 30 frames — freezes. +- 9b `…_dualref` (reproject ON) — motion continues. This also explains why + early dual-ref results "looked like" blend mode: both carried motion via + the warp. + +Neither reference order nor a role-naming prompt rescues motion, so +`REF_ORDER` is deprioritized as an axis (kept in the CLI for completeness). +analyze_drift on the 4b frame-first pair: no fixed point — Δprev ≈ 9.6 +(neutral) / 15.3 (prompt) MAD at save-every 5; Δoriginal 23.0 vs 35.7 — +frozen composition + perpetual texture churn; the prompt escalates texture +only (its mp4 is ~3× the neutral one's). + +**Not a pipeline-list bug.** In diffusers 0.40.0 `Flux2KleinPipeline` each +reference is separately preprocessed, VAE-encoded and packed, then +concatenated on the sequence axis (nothing dropped; `mu` shifts with total +reference token count via `compute_empirical_mu`). Key detail: +`_prepare_image_ids` assigns reference *i* the time coordinate T = 10 + 10*i +— the reference LIST is encoded as a temporally-indexed sequence of +observations of ONE scene (restoration-style multi-frame conditioning), not +"named subjects" an editor arbitrates between. The model card documents no +multi-reference prompt convention, so role-naming phrasing is a guess — and +it only modulates texture anyway. + +**Reference-gain readout** (4b, 5 source frames then 10-frame tails +freeze/free/black branching from ONE shared end state, empty prompt, fixed +seed; pod `output/debug/refweight`). MAD between tail videos, t=1 → t=10: + + frz-free 4.21 → 33.71 ([P, last_source] vs [P] alone) + frz-black 1.65 → 13.01 ([P, last_source] vs [P, black]) + free-black 4.92 → 31.82 + + per-pass chatter (Δprev): freeze ≈ 2.5, black ≈ 3.0, free ≈ 4.0 + mean luma t1→t10: freeze 114→106, black 113→108, free 118→139 + +Reading: the second reference has real but LOW per-pass gain (~1.6 MAD vs a +maximally different ref2 after one pass) that compounds over passes; ANY +second reference — even black, whose content is not reproduced (no +darkening: black-run luma tracks freeze-run) — stabilizes the consensus +against buffer-echo drift ([P] alone drifts +21 luma in 10 passes and +churns most). Motion death in the main loop follows: at 59.94 fps the +inter-frame displacement is far below the frame reference's per-pass pull, +so position is carried ONLY by the self-reinforcing state — and only the +flow warp moves the state. Bonus finding: under dual-ref the black tail ≈ +freeze tail (no decay driver) — the black scenario barely differs from +freeze for klein. + +**Implications / follow-ups:** + +- dual-ref + reproject is the viable klein video regime; the artifact + ceiling is warp quality (grayscale Farneback flow, BORDER_REPLICATE smear, + no disocclusion rejection — see `imaging.make_reprojector`). Next + experiment: mask disocclusions via forward-backward flow consistency and + patch them from the fresh frame before pairing. +- The `neutral` sweep leg under dual-ref is effectively "freeze from frame + ~5" — not a useful preservation baseline as-is. +- The anchored control (`--mode anchored`) was not needed for this verdict; + optional. + ## GPU selection: re-check the whole Runpod catalog (action item, opened 2026-09-13) **Context:** the 2026-09-13 klein-validation run is on an **L40S** (48 GB, -- 2.51.2