FLUX.2 klein in dltb — the video pipeline under test #

Reference documentation for the klein video loop: how the current pipeline (dltb-klein --conditioning dual-ref --reproject [--reproject-mask], driven by scripts/sweep-klein.sh / scripts/sweep-klein-mask.sh) works end to end, and how it got here. Investigation details live in NOTES.md (sections linked below); this doc is the consolidated, current-state view.

Status at time of writing (2026-09-13, concluded): the reproject-mask A/B was analyzed and positive on both klein models, but the neutral-prompt control showed the per-pass compounding deformation is intrinsic to the closed klein self-reference loop (it persists, weaker, with an empty prompt). The experiment line is concluded and back on the drawing board; --reproject-mask remains opt-in (the planned follow-through — mask as dual-ref default, sweep axis — is shelved with the line).

1. The conceptual frame #

The experiment simulates a DLSS-5-style inference loop:

P_n = f(state(P_{n-1}), N_n, motion_vectors_n, artistic_direction)

where P is the model's own previous output ("carried temporal state") and N the freshly rendered frame. FLUX.2 klein (4B / 9B) plays the role of f. It is a reference-image editor, unlike the img2img models in this repo:

• no partial noising, hence no --strength — it regenerates from pure noise attending to reference tokens;
• --guidance-scale > 1 is inert (CFG hard-disabled in step-wise distilled checkpoints, no guidance embedding; NOTES.md, "FLUX.2 klein: --guidance-scale is inert"). --negative-prompt equally so;
• the prompt is the de-facto per-pass edit-strength knob (empty/neutral preserves; an edit instruction compounds every pass), and --num-inference-steps (default 4, the card default) is the other direct intensity knob.

2. The loop, per frame (current pipeline) #

Tools: dltb-klein (arg surface) → continuous.run() (loop) with the make_conditioning hook → klein._dual_ref_conditioning. Geometry 768×768 (klein ModelSpec defaults, pass_size=True), bf16, fixed seed 1234 (--fixed-seed default: same noise every pass — DLSS-5-like determinism that holds only while conditioning is bit-identical, see §5).

1. Ingest: each source frame → prepare_frame: RGB, center-crop to the target aspect, LANCZOS resize to 768².
2. Frame 1: no state yet; single clean reference N_1 → P_1.
3. Frames n ≥ 2 (combine(carried=P_{n-1}, new_frame=N_n, prev_source=N_{n-1})):
   1. Flow: estimate_flow(N_{n-1}, N_n) — two Farneback passes on grayscale (forward A→B, backward B→A when masking is on). Flows are estimated between source frames only; the model output never participates.
   2. Warp (--reproject, default on): the carried state is reprojected along the forward flow — target pixel p fetches state at p − fwd(p) (bilinear, BORDER_REPLICATE). With --reproject-mask, disoccluded pixels are patched from N_n first (§4). This warp emulates engine motion vectors and is load-bearing (§5).
   3. Pairing: ordered(state', N_n) → [state', N_n] with --ref-order state-first (default) or [N_n, state'] with frame-first. --anchor-blend is ignored in this mode — so --mode anchored is the only single-reference control under dual-ref (under blend conditioning, a = 1.0 would reproduce it exactly).
4. Prompt: role-naming, indices derived from ref order — "image {F} is the current frame; keep the appearance of image {S}, <edit instruction>". Note: composition comes from the references; the prompt modulates texture intensity only (§5).
5. Pass: run_pass → Flux2KleinPipeline(prompt, image=[state', N_n], num_inference_steps=4, guidance_scale=1.0 (spec default; inert anyway), width=height=768, generator) — the list flows through untouched (§3).
6. Emit: result becomes P_n (the new carried state); appended to processed_stateful.mp4; every 10th frame saved as PNG.
7. Tails (after the last source frame, branching from one shared end_state.png): freeze = [P, last_source] (static-menu case), free = [P] alone (buffer echo), black = [P, black] (renderer crash). No flow/warp in tails — static input means zero motion vectors.

Run-directory tag: <stem>_dualref[-norepro|-fbmask]_tails<modes><N> (-norepro = warp off, -fbmask = warp + mask on).

3. Inside Flux2KleinPipeline (diffusers 0.40.0) #

What actually happens to the reference list inside one pass:

1. Prompt → Qwen3 text encoder → text tokens.

2. Reference preprocessing — each list element independently: area check (downscale to ≤ 1 MP if needed; 768² = 0.59 MP passes untouched), dimensions snapped to multiples of vae_scale_factor * 2, normalize, VAE-encode, _patchify_latents (2×2 patch packing), then batch-norm whitening with the VAE's running stats. Each 768² reference ≈ 2.3k tokens (dim 128).

3. Token coordinates — the decisive detail (_prepare_image_ids): every reference token gets a 4D coordinate (T, H, W, L); reference i sits at T = 10 + 10·i, the generation target's noise tokens at T = 0. One spatiotemporal sequence enters attention:

   [text] [target noise @ t=0] [state' tokens @ t=10] [N_n tokens @ t=20]
   

   The reference list is encoded as a temporally-indexed sequence of observations of one scene — restoration-style multi-frame conditioning, not "named subjects" an editor arbitrates between. Consequences in §5. Side effect: total reference token count (~4.6k for two refs) feeds compute_empirical_mu, which shifts the timestep schedule — dual-ref subtly changes the denoising schedule vs single-ref.

4. Denoising (4 steps): torch.cat([latents, image_latents], dim=1) — target and reference tokens concatenated on the sequence axis; the transformer self-attends across all of them each step. References are clean (never noised); the target starts as pure noise and is denoised toward a consensus that preserves reference appearance. CFG branch unreachable (is_distilled).

5. Decode: unpatchify → VAE decode → 768² PIL → P_n.

The mask (§4) operates entirely upstream of all of this: it edits the pixel content of reference 1 before torch ever sees it.

4. The disocclusion mask (--reproject-mask) #

imaging.make_reprojector(mask_disocclusions=True, fb_tau=1.5, dilate_px=5):

• Forward–backward circularity: if target pixel p truly corresponds to something in the previous frame, the two independent flow estimates must agree on one track: fwd(p) ≈ −bwd(p). Round-trip residual |fwd(p) + bwd(p)| > fb_tau (1.5 px) → no trustworthy history at p (newly revealed content, newly occluded content, or flow hallucination — all three mean: don't use the warp).
• Out-of-bounds sampling: p − fwd(p) outside the frame would have been BORDER_REPLICATE-smeared → masked unconditionally.
• Dilation (5 px ellipse) grows the mask over the smear's penumbra (bilinear rim mixing, threshold misses).
• Composite: masked pixels take the fresh frame's pixel — a disocclusion is by definition content just revealed in N_n; the fresh frame is its only witness. Hard binary swap (feathered compositing is parked, NOTES.md).
• In blend conditioning the same warp runs pre-blend; at masked pixels the alpha blend then yields exactly N_n. At a = 1.0 the whole warp — mask included — is skipped, since the blend discards it (stateful a=1 equals --mode anchored; the skip lives in continuous._blend_conditioning, not in the dual-ref hook, which is unaffected).

Cost: one extra CPU Farneback pass per frame. Measured on the tracked clip (consecutive pairs, run-identical constants): fires on 1.9–7.3% of pixels per pass (≈4% mean), OOB 0.2–0.6%; fires concentrate at genuine motion (a flow-magnitude gate changes nothing); false positives are benign (fallback is same-scene content). Because the patch becomes part of P_n, the next pass carries it forward — TAA-style history rebuild, but performed implicitly by the generative loop instead of an explicit confidence buffer.

Known limits: the forward flow is used as a per-target-pixel backward map (small-motion approximation, fine at this clip's 1–5 px inter-frame displacement); no temporal confidence weighting; fb_tau/dilate_px are clip-tuned constants.

5. Established empirical properties #

The loop's "physics", each verified on-GPU (NOTES.md, "Klein dual-ref GPU validation" and "reproject-mask A/B"):

1. Reprojection is load-bearing. Without the warp, dual-ref freezes into a static consensus within a few frames — regardless of reference order, prompt, or model (4b and 9b; five-run freeze ledger). At 59.94 fps the fresh frame's per-pass pull (~1.6 MAD toward maximally different content) is far below inter-frame displacement, so only the warped state carries position. The warp also puts [state', N_n] back inside the training distribution (two aligned observations of one scene).

2. Restoration-consensus regime. The T-coordinate encoding (§3) makes the reference bundle "one scene observed repeatedly"; the output is a stable consensus of it. Corollaries: any second reference — even black, whose content is not reproduced — stabilizes against buffer-echo drift ([P] alone drifts hard); reference order and role-naming prompts do not rescue motion (REF_ORDER deprioritized); composition comes from the references, the prompt modulates texture only.

3. Same-seed ≠ same trajectory under perturbed conditioning. Klein redraws texture from noise every pass, so a ~4%-of-pixels conditioning perturbation (the mask) decorrelates the render within ~10 passes (frame-identical at pass 1, MAD 34 by pass 10, plateau ~90). A/Bs of this loop must be statistical, not frame-identity.

4. The mask works end-to-end (4b A/B, 300 frames + freeze tail, identical seed/prompt/steps). Output-vs-source inside vs outside disocclusion zones, mean over 30 saved frames:

   leg	MAD | disocc	MAD | rest	HF energy | disocc	saturation | disocc
   plain (warp only)	92.5	89.0	26.8	146
   fbmask (warp+mask)	31.0	78.2	10.8	105
   source (ground truth)	—	—	5.5	98

   Patched fresh-frame content demonstrably propagates into the output; the unmasked leg's streaks are high-frequency, oversaturated replicate smears (they read as "blurry" at video speed but are spectrally sharp). Residual ~2× HF in fbmask zones — split finding from the neutral-prompt control: the spectral/color vividness was prompt-driven (zone HF/saturation drop to near-source with PROMPT="" on both legs), while the global compounding drift is prompt-independent (Δoriginal neutral 88.5/89.0 vs enhanced 78.4/92.0 — end frames similar either way). Under a neutral prompt the mask wins on every zone axis (MAD 20.9 vs 48.9, churn 22.6 vs 52.5). Global "heavy deformation" in both legs is this by-design compounding drift of a 300-pass run, orthogonal to the mask.

   9b confirmation (same script, both legs): verdict replicates — zones MAD 58.5→33.1, HF 21.8→11.9, saturation 215→123 (source 5.5/98). The twist: 9b's unmasked streaks are not raw smears but plausible pseudo-3D structure with extreme color (saturation 2.2× source) — a stronger editor resolves invalid history into coherent-looking artifacts. fbmask zones are nearly model-independent (correct fresh content is preserved either way). Runtime measured: ~6.2 s/pass (9b) / ~3.2 s/pass (4b) on RTX 6000 Ada, 768², dual-ref, 4 steps.

5. Blend conditioning (historical): klein under pixel-blend anchoring was too weak at --anchor-blend 0.6–0.8 and "melted" at 0.1 — the anomaly that started the dual-ref redesign.

6. Historical development #

• 2026-09-12 — calibration anomaly. Cross-model stateful sweep (blend conditioning): sd/sdxl-turbo and flux-schnell showed the expected cartoonish degradation; flux2-klein-9b was too weak at 0.6–0.8 and melted at 0.1. Experiments paused pending a topology rethink.
• 2026-09-12 — dltb-klein redesign. New tool (dltb-klein) + scripts/sweep-klein.sh: prompt-as-strength ladder, steps probes; klein framed as a reference editor (prompt = edit knob).
• 2026-09-12 — guidance proved inert. Mid-sweep log flood; three-point source proof in diffusers 0.40.0; guidance probes removed, replaced by steps probes; one-time warning added to the tool.
• 2026-09-13 — dual-reference conditioning implemented. continuous.run() generalized with the make_conditioning factory hook; klein._dual_ref_conditioning passes [R(P_{n-1}), N_n] as two clean references (verified: the pipeline natively accepts a list, concatenates per-reference latents on the sequence axis). --ref-order, role-naming prompts, REPROJECT=ab A/B wired into the sweep.
• 2026-09-13 — first dual-ref sweep, interrupted. Early legs looked "similar to stateful blend"; suspected motion artifacts; sweep stopped after enhance-slight (partial legs preserved). Debug session: empty- --input hardening; norepro freeze observed; frame-first and role-naming probes (both freeze); analyze_drift shows locked composition + perpetual texture churn.
• 2026-09-13 — mechanism found. Diffusers source read: T-coordinate reference encoding → restoration-consensus regime. Tail-based reference-gain readout (freeze/free/black from one shared state): second reference has real but low per-pass gain that compounds; any second reference stabilizes; black content is not reproduced. Verdict: reproject load-bearing, dual-ref-norepro structurally dead for motion.
• 2026-09-13 — reproject-mask A/B, then conclusion. FB-consistency disocclusion mask implemented in imaging.make_reprojector; --reproject-mask flag in dltb-continuous/dltb-klein; scripts/sweep-klein-mask.sh runner. Results in §5.4 — confirmed on 4b and 9b. The neutral-prompt control (same day) showed the compounding deformation is intrinsic to the closed loop, not prompt-driven; the experiment line was concluded and shelved (mask stays opt-in), pending a new approach to the self-reference drift problem.

7. File map #

8. Open questions (after the 2026-09-13 conclusion) #

The mask, reproject, and dual-ref characterization stand (§5); the open problem is the intrinsic per-pass compounding of a self-referencing regeneration loop — even an empty prompt deforms the output over ~100s of passes. Drawing-board candidates:

• Explicit history/confidence buffer OUTSIDE the model (true TAA-style accumulation, with klein only as the per-frame renderer).
• Periodic hard re-anchoring (state reset every k frames) to bound drift.
• A different conditioning topology entirely (klein's T-coordinate multi-reference layout invites feeding genuinely time-spaced frames — motion extrapolation rather than restoration).
• Or: accept the drift as the object of study (it is the drift experiment's subject) and scope klein out of the "healthy pipeline" simulation.
• Untouched: whether klein's multi-reference T-coordinates could be exploited directly (e.g., feeding genuinely time-spaced references for motion extrapolation rather than restoration).