diff --git a/AGENTS.md b/AGENTS.md index eeff245..e37b7ac 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -148,6 +148,11 @@ Key invariants: - `flux2-klein-9b` is gated: needs `HF_TOKEN` (checked at startup). - Klein is a reference-image editor: no `--strength`; the prompt is the per-pass edit-strength knob; blend active range is far below img2img models. +- `--mode anchored` ≡ `--mode stateful --anchor-blend 1.0` under pixel-blend + conditioning: `blend(P, N, 1) = N` exactly and the flow/warp is skipped at + a=1.0 (its result would be blended away), so frames and cost are identical. + NOT equivalent under klein `dual-ref`, where the blend is ignored and + anchored is the only single-reference loop. ## Local (macOS) gotchas diff --git a/NOTES.md b/NOTES.md index ff3cc00..6836872 100644 --- a/NOTES.md +++ b/NOTES.md @@ -761,3 +761,37 @@ frames, every 5th, ensemble): `analyze_drift` reported CONVERGED (delta_prev - Not encoded in the CSV/JSON names: `--every`, `--dreamsim-type` and `--patch`. Pass `--out`/`--json` when comparing variants, same rule as the run tags (AGENTS.md). + +## `--mode anchored` is the `--anchor-blend 1.0` endpoint (2026-09-15) + +Analysis, then a small change. For the default pixel-blend conditioning, + + blend((1-a)*R(P_{n-1}) + a*N_n) --a=1.0--> f(N_n) + +because `Image.blend(c, n, 1.0) = n` exactly (uint8 -> float -> *1.0 -> round +is the identity), so the carried state — and any optical-flow warp of it — +contributes exactly zero. `--mode stateful --anchor-blend 1.0` and +`--mode anchored` produce pixel-identical frames, same seeds, same tails; +`dltb-distance-feedback`'s docstring already leaned on this endpoint as a +correctness check (a=1.0 there = repeated independent passes = boil test). + +The only cost difference was reprojection: stateful defaults to +`--reproject` ON, and at a=1.0 the per-frame Farneback flow + warp was +computed and then blended away — pure waste. `continuous._blend_conditioning` +now skips flow/warp when `anchor_blend >= 1.0`, and `run()` prints a NOTE +when that skip is active (the header still says `reproject=on` because the +setting, not the work, drives the run tag). So stateful a=1.0 is now +pixel- AND cost-identical to anchored. + +Why keep the flag at all: + +- **klein dual-ref.** `--conditioning dual-ref` ignores `--anchor-blend` + entirely (weighting is the model's job via attention over two clean + references), so the blend endpoint does not exist there; `--mode anchored` + is the only single-reference (boil-test) topology for klein. +- Self-documenting CLI, distinct run tag (`_anchored`, + `processed_anchored.mp4` — smoke.sh asserts the latter), and it costs one + `or` clause in the main loop. + +Docs updated to say the flag is redundant except under klein dual-ref +(continuous.py + klein.py docstrings/help, README, AGENTS.md gotchas). diff --git a/README.md b/README.md index 02cbdbb..2edff35 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,9 @@ The console scripts share the library code in `src/dltb/` (`models`, `imaging`, model's previous output (`P_n = f(P_{n-1})`); saves every `--save-every`th frame and assembles an mp4 timelapse. - `dltb-continuous` — video pipeline simulation: each source frame processed - once, either independently (`--mode anchored`, the boil test) or with carried + once, either independently (`--mode anchored`, the boil test — equivalent to + `--mode stateful --anchor-blend 1.0`: same frames, same cost, so the flag + only matters for klein dual-ref) or with carried state blended into each new frame (`--mode stateful`, optical-flow reprojection on by default), plus failure tails that branch from the shared end-of-video state (`--tail-modes freeze,free,black`). @@ -29,7 +31,9 @@ The console scripts share the library code in `src/dltb/` (`models`, `imaging`, is the de-facto per-pass edit-strength knob (`--num-inference-steps` is the other). `--conditioning dual-ref` passes the carried state and fresh frame as two separate clean reference images (`[P, N]`, or `[N, P]` with - `--ref-order frame-first`) instead of one pixel blend; under dual-ref the + `--ref-order frame-first`) instead of one pixel blend (`--anchor-blend` is + ignored there, so `--mode anchored` is the only single-reference control); + under dual-ref the flow warp is load-bearing (without it the loop freezes into a static consensus — NOTES.md, 2026-09-13) and `--reproject-mask` caps its artifact floor by patching disoccluded pixels from the fresh frame. diff --git a/doc/algorithm/continuous.md b/doc/algorithm/continuous.md index 061acf7..71532f3 100644 --- a/doc/algorithm/continuous.md +++ b/doc/algorithm/continuous.md @@ -43,13 +43,21 @@ Stateful details: - **Blend** (`--anchor-blend alpha`, default 0.3): `blend = (1−a)·carried + a·new_frame` (linear `Image.blend` in pixel space). `a = 0` is free-running (pure self-iteration), `a = 1` re-anchors - completely every frame. The blend is the anchor strength of the loop; + completely every frame — which is `--mode anchored` **exactly**: + `blend(P, N, 1) = N`, the carried state contributes zero, and the + flow/warp is skipped at `a = 1` too (since 2026-09-15; its result would + be blended away), so the two modes are pixel- and per-frame-cost + identical. `anchored` remains a separate mode only for klein dual-ref + (§6), where alpha is ignored and it is the only single-reference + (boil-test) topology. The blend is the anchor strength of the loop; klein's active range sits far below the img2img models'. - **First frame**: `current` starts `None`, so frame 1 is always processed as a fresh source regardless of mode — the loop needs a seed state. - **Reprojection** `R` is stateful-only and defaults ON (§4); `--mode anchored` ignores it (and `--anchor-blend`), which the run tag reflects - (§7). + (§7). In stateful, `a = 1` skips the per-frame flow/warp as well (a NOTE + is printed when that skip is active, since the header reports the + `--reproject` *setting*, not the work). ## 3. The main loop, step by step @@ -78,7 +86,9 @@ Per source frame `n = 1, 2, …` (stopping after `--max-frames` if given): - `anchored` or `current is None` → `source = new_frame`; - `stateful` → `source = combine(current, new_frame, prev_source)` (default: warp the carried state along the flow `prev_source → - new_frame` when reprojection is on, then blend; §4–§6). + new_frame` when reprojection is on, then blend; §4–§6). At + `a = 1` the warp is skipped — its result would be blended away — and + `source = N_n` exactly, i.e. the anchored case. 3. `prev_source = new_frame` (flow is always estimated between consecutive *source* frames, never between processed ones). 4. `one_pass(source, n)`: @@ -107,6 +117,15 @@ effective main-loop recursion source_n = (1−a) · warp(P_{n−1}, flow N_{n−1} → N_n) + a · N_n +At `a = 1` the warp term has weight zero, so the flow/warp step is skipped +entirely (`blend(P, N, 1) = N` exactly in pixel space) — `stateful a=1` +makes `--mode anchored` redundant at the same per-frame cost. The +reprojector itself is still constructed (one lazy cv2 import — setup noise, +not per-frame work), and a NOTE is printed because the run header reports +the `--reproject` setting, not the work actually done. (At `a = 0` the +opposite holds: the fresh frame drops out and the warp is the *entire* +input, so it stays load-bearing.) + Mechanics: - `estimate_flow(prev, next)` → forward flow (backward flow too, only when @@ -239,7 +258,9 @@ Findings established on real runs, not guessed: output). - Mode `anchored` accepts (and ignores) `--anchor-blend`/`--reproject` after validation; the run tag honestly reflects only what ran, but the CLI does - not warn. + not warn for that direction. The converse — `stateful --anchor-blend 1.0` + with reprojection on — prints a NOTE (§4), because the header's + `reproject=on` reflects the setting while the flow/warp is skipped. - Everything else about a pass (guidance being inert for klein, size rules, offloading for the big models) is inherited from `models.py` / `imaging.run_pass` and documented in README.md. diff --git a/doc/algorithm/flux2klein.md b/doc/algorithm/flux2klein.md index 53b1ece..259c78f 100644 --- a/doc/algorithm/flux2klein.md +++ b/doc/algorithm/flux2klein.md @@ -62,7 +62,10 @@ that holds only while conditioning is bit-identical, see §5). **load-bearing** (§5). 3. **Pairing**: `ordered(state', N_n)` → `[state', N_n]` with `--ref-order state-first` (default) or `[N_n, state']` with - `frame-first`. `--anchor-blend` is ignored in this mode. + `frame-first`. `--anchor-blend` is ignored in this mode — so + `--mode anchored` is the only single-reference control under + dual-ref (under blend conditioning, `a = 1.0` would reproduce it + exactly). 4. **Prompt**: role-naming, indices derived from ref order — `"image {F} is the current frame; keep the appearance of image {S}, "`. Note: composition comes from the references; the @@ -135,7 +138,11 @@ dilate_px=5)`: frame is its only witness. Hard binary swap (feathered compositing is parked, NOTES.md). - In `blend` conditioning the same warp runs pre-blend; at masked pixels - the alpha blend then yields exactly `N_n`. + the alpha blend then yields exactly `N_n`. At `a = 1.0` the whole warp — + mask included — is skipped, since the blend discards it (`stateful a=1` + equals `--mode anchored`; the skip lives in + `continuous._blend_conditioning`, not in the dual-ref hook, which is + unaffected). Cost: one extra CPU Farneback pass per frame. Measured on the tracked clip (consecutive pairs, run-identical constants): fires on **1.9–7.3%** of diff --git a/src/dltb/continuous.py b/src/dltb/continuous.py index 69d144c..7d8dea8 100644 --- a/src/dltb/continuous.py +++ b/src/dltb/continuous.py @@ -25,6 +25,13 @@ the freshly rendered frame from the engine. (--anchor-blend alpha = weight of the fresh frame; 0.0 = free-running, 1.0 = fully re-anchored per frame) + anchored is redundant with stateful alpha = 1.0 for the pixel blend: the + frames are identical (blend(P, N, 1) = N exactly) and so is the cost (the + flow/warp is skipped there, since its result would be blended away). The + mode remains distinct for dltb-klein's dual-ref conditioning, where alpha + is IGNORED and --mode anchored is the only way to run the single-reference + boil test. + REPROJECTION (--reproject, default ON for stateful): Real temporal pipelines (TAA, DLSS 2-5) never blend raw history: motion vectors warp the carried state before blending (here estimated with @@ -106,12 +113,18 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace: p.add_argument("--mode", choices=["anchored", "stateful"], default="stateful", help="Loop topology (see module docstring). anchored = " "independent per-frame boil test; stateful = carried " - "state blended with each new frame. (The anchor-loss " - "scenario is covered by --tail-modes free.)") + "state blended with each new frame. anchored is " + "redundant with stateful --anchor-blend 1.0 (same " + "frames, same cost) for every model except klein " + "dual-ref conditioning, where the blend is ignored. " + "(The anchor-loss scenario is covered by " + "--tail-modes free.)") p.add_argument("--anchor-blend", type=float, default=0.3, help="stateful mode: weight alpha of the fresh frame in the blend " "(1-a)*previous_output + a*new_frame. 0.0 = free-running, " - "1.0 = fully re-anchored every frame") + "1.0 = fully re-anchored every frame (== --mode anchored; " + "the flow/warp is skipped there because blend(P, N, 1) = N " + "exactly)") p.add_argument("--reproject", action=argparse.BooleanOptionalAction, default=True, help="stateful mode only: warp the carried state by optical flow " "estimated between consecutive source frames before blending " @@ -147,7 +160,11 @@ def _blend_conditioning(args, estimate_flow, warp): def combine(carried, new_frame, prev_source): c = carried - if estimate_flow is not None and prev_source is not None: + # alpha = 1.0 blends the carried state away exactly + # (blend(c, n, 1) = n), so flow + warp would be discarded; skipping + # them makes stateful a=1.0 cost-identical to --mode anchored. + if (estimate_flow is not None and prev_source is not None + and args.anchor_blend < 1.0): c = warp(c, estimate_flow(prev_source, new_frame), fallback_pil=new_frame) return Image.blend(c, new_frame, args.anchor_blend) @@ -183,6 +200,11 @@ def run(args: argparse.Namespace, make_conditioning=None) -> None: print(f"dltb-{Path(__file__).stem}: NOTE: --reproject-mask has no effect " "without --reproject in stateful mode; ignoring it.", flush=True) fb_mask = False + if use_reproject and not dual_ref and args.anchor_blend == 1.0: + print(f"dltb-{Path(__file__).stem}: NOTE: --anchor-blend 1.0 blends the " + "carried state away exactly (blend(P, N, 1) = N); the reprojection " + "flow/warp is skipped, so this run equals --mode anchored.", + flush=True) mode_tag = args.mode if args.mode == "stateful": diff --git a/src/dltb/klein.py b/src/dltb/klein.py index c3cb756..d3bc0de 100644 --- a/src/dltb/klein.py +++ b/src/dltb/klein.py @@ -36,6 +36,9 @@ CONDITIONING (--conditioning, stateful mode only): blend (default) pixel-blend the (reprojected) carried state and the fresh frame into ONE reference image: image = (1-a)*R(P_{n-1}) + a*N_n + (a = 1.0 reproduces --mode anchored exactly, as in dltb-continuous; + the flow/warp is skipped there because its result would be + blended away) dual-ref pass the carried state and the fresh frame as TWO SEPARATE clean reference images: image = [R(P_{n-1}), N_n]. Flux2KleinPipeline @@ -44,7 +47,9 @@ CONDITIONING (--conditioning, stateful mode only): no ghosted blend mush for the editor to parse. The model itself decides how to weight state vs. anchor through attention, which is categorically closer to DLSS 5's multi-input conditioning than the - pixel blend. --anchor-blend is ignored in this mode. + pixel blend. --anchor-blend is ignored in this mode, so --mode + anchored is NOT reachable via the blend here -- it is the only + single-reference (boil-test) control under dual-ref. --ref-order {state-first,frame-first} (default state-first) selects [P, N] vs [N, P]; whether klein treats reference order as @@ -124,7 +129,10 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace: p.add_argument("--mode", choices=["anchored", "stateful"], default="stateful", help="Loop topology (see module docstring). anchored = " "independent per-frame boil test; stateful = carried " - "state conditioned with each new frame.") + "state conditioned with each new frame. Under blend " + "conditioning, stateful --anchor-blend 1.0 is the same " + "run; under dual-ref the blend is ignored and anchored " + "is the only single-reference loop.") p.add_argument("--conditioning", choices=["blend", "dual-ref"], default="blend", help="stateful mode only: how carried state + fresh frame become " "model input. blend = one pixel-blended reference " @@ -140,7 +148,8 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace: "the blend (1-a)*previous_output + a*new_frame. Klein's " "active range is far below the img2img models (try " "0.03/0.05/0.2); 0.0 = free-running, 1.0 = fully " - "re-anchored every frame. IGNORED under dual-ref.") + "re-anchored every frame (= --mode anchored). IGNORED " + "under dual-ref.") p.add_argument("--reproject", action=argparse.BooleanOptionalAction, default=True, help="stateful mode only: warp the carried state by optical flow " "estimated between consecutive source frames before "