# DLTB - Deep Learning Tripping Balls

This is an attempt to have DLSS5 at home, but weirder.

Trying to test what happens when a generative loop is fed its own output,
and what a healthy pipeline (fresh frame + carried state + motion vectors)
does instead.

## Tools

The console scripts share the library code in `src/dltb/` (`models`, `imaging`,
`output`, `args`):

- `dltb-oneshot` — single image, single model pass: the anchored regime
  reduced to its fixed point. (Anchored *iteration* with `--fixed-seed` just
  repeats the same image every pass, so one pass already says everything.)
- `dltb-iterate` — free-running image self-iteration: each pass consumes the
  model's previous output (`P_n = f(P_{n-1})`); saves every `--save-every`th
  frame and assembles an mp4 timelapse.
- `dltb-continuous` — video pipeline simulation: each source frame processed
  once, either independently (`--mode anchored`, the boil test — equivalent to
  `--mode stateful --anchor-blend 1.0`: same frames, same cost, so the flag
  only matters for klein dual-ref) or with carried
  state blended into each new frame (`--mode stateful`, optical-flow
  reprojection on by default), plus failure tails that branch from the shared
  end-of-video state (`--tail-modes freeze,free,black`).
- `dltb-klein` — the same video loop, restricted to the FLUX.2 klein editors
  (`flux2-klein-4b`, ungated, default / `flux2-klein-9b`, gated). Klein is a
  reference-image editor: no `--strength`, per-pass change scales ~linearly
  with the blend (default `0.1`, far below the img2img models), and the prompt
  is the de-facto per-pass edit-strength knob (`--num-inference-steps` is the
  other). `--conditioning dual-ref` passes the carried state and fresh frame
  as two separate clean reference images (`[P, N]`, or `[N, P]` with
  `--ref-order frame-first`) instead of one pixel blend (`--anchor-blend` is
  ignored there, so `--mode anchored` is the only single-reference control);
  under dual-ref the
  flow warp is load-bearing (without it the loop freezes into a static
  consensus — NOTES.md, 2026-09-13) and `--reproject-mask` caps its artifact
  floor by patching disoccluded pixels from the fresh frame.
  `scripts/sweep-klein.sh` walks its prompt ladder and steps probes under
  either conditioning; `scripts/sweep-klein-mask.sh` A/Bs the disocclusion
  mask on top of dual-ref reprojection
  (guidance is inert for klein — CFG is disabled and there is no guidance
  embedding in the distilled checkpoints).
- `dltb-assemble` — CPU-only local post-processing: encode mp4s from a run's
  saved PNG frames (main loop + per-tail-mode videos). Complements the split
  workflow: frames are computed on a rented GPU pod, the run directory is
  copied home, and videos are (re-)assembled locally — no torch, no
  accelerator (`uv run python src/dltb/assemble.py` also works; needs only
  numpy/PIL/imageio).
- `dltb-distance` — perceptual counterpart to `analyze_drift.py`: scores a
  run's frames with DreamSim (learned DINOv2/CLIP ensemble, not pixels) and
  reports the distance to the run's source image (`dreamsim_to_ref`, drift) and
  to the previous frame (`dreamsim_to_prev`, perceptual fixed-point detection —
  ~0 while the loop still chatters means the chatter is invisible). Weights are
  downloaded on first use (see Requirements); runs on CUDA, Apple MPS, or CPU.
- `dltb-distance-loop` — the free-running loop *and* the perceptual metrics
  in one run, restricted to the strength-capable img2img models (`sd-turbo`,
  `sdxl-turbo`, `flux-schnell`): every pass is saved, then DreamSim scores each
  frame against the prepared original and its predecessor into
  `distance_metrics.csv`, and both series are charted to `distance_plot.png`
  (frame number on X, distance on Y). The diffusion pipeline is unloaded before
  DreamSim loads, so the two models never share accelerator memory.
- `dltb-distance-feedback` — the same measurement around `dltb-continuous`'s
  stateful loop with a *static* source: the "video" is one image repeated for
  `--frames` frames, so `source_n = (1-a)*P_{n-1} + a*N` unrolls with no motion
  (no reprojection, no tails — flow between identical frames is zero). The
  endpoints are free correctness checks: `--anchor-blend 0` is exactly
  `dltb-distance-loop` (free-running), `--anchor-blend 1` is repeated
  independent passes (boil test); in between, the loop shows how far the model's
  carried state drags a run away from a fresh render of its own input. Every
  pass's actual model input (the blend) is saved under `inputs/`, and DreamSim
  scores three series into `distance_metrics.csv` / `distance_plot.png`: frame
  vs. original, frame vs. previous frame, and model input vs. original.
- `dltb-stabilization` — CPU-only verdict on a metrics CSV: did each series
  stabilize, at which frame, and at what value — or why it needs attention
  (still rising at the end, wandering, oscillating, drifting). Assumes the
  run's tail is stationary, bands it robustly (median ± k·MAD), scans
  backward for the last out-of-band smoothed sample, then diagnoses the
  plateau; reads the `distance_metrics.csv` the distance tools write (or
  `drift_metrics.csv` from `analyze_drift.py`) and writes
  `stabilization_report.csv` next to the input. No torch, no accelerator
  (`uv run python src/dltb/detect_stabilization.py` also works; needs only
  numpy — matplotlib only for `--plot`).
- `dltb-stability-sweep` — sweeps `dltb-distance-feedback`'s `--anchor-blend`
  over `--min-blend/--max-blend/--step` with everything else fixed: ONE
  pipeline load for the whole range, ONE DreamSim load for all verdicts, and
  `detect_stabilization` judging the frame-vs-previous and frame-vs-reference
  series per blend (the input-vs-ref series is ignored). Blends where **both**
  series stabilize are reduced to their two plateau-opening frames plus
  `run.json` under `stable/<tag>/`; every other run moves wholesale to
  `unstable/<tag>/` for inspection; the stable ones are summarized in
  `stability_sweep.csv` (`anchor-blend, v_prev, i_prev, v_ref, i_ref`) and
  charted to `stability_plot.png` (stable points only, no placeholders).

## Documentation

Lives in `doc`:
- `doc/temp/`: temporary documents, will be deleted after no longer useful. Not to be referenced in non-temporary documents.
- `doc/algorithm/` algorithm description
- `doc/algorithm/flux2klein.md`: FLUX.2-klein pipeline
- `doc/algorithm/stabilization.md`: the stabilization detector behind
  `dltb-stabilization` (algorithm, verified robustness, limitations)

## Models

| `--model` | Backbone | Default resolution | Notes |
| --- | --- | --- | --- |
| `sd-turbo` | U-Net / CNN | 512x512 | fp16, low VRAM |
| `sdxl-turbo` | U-Net / CNN | 768x768 | fp16 (1024-native) |
| `flux-schnell` | transformer (MM-DiT) | 768x768 | bf16 needs ~34 GB VRAM; fits 24 GB with `--offload` |
| `flux2-klein-4b` | transformer | 768x768 | Apache-2.0, ~9 GB VRAM, no `--strength` |
| `flux2-klein-9b` | transformer | 768x768 | gated, `HF_TOKEN` required, ~20-29 GB VRAM |

### Local Apple Silicon

`sd-turbo` is the practical local target; `sdxl-turbo` is the stretch goal.
`flux2-klein-4b` can produce single frames on a 16 GB machine (fine for
one-shot generation, far too slow for video). `flux-schnell` (~34 GB bf16)
and `flux2-klein-9b` (gated, ~20-29 GB) do not fit a consumer Mac; nothing
blocks trying them anyway (`MODELS="..." scripts/smoke-local.sh`).

## Setup

The lockfile resolves platform-correct PyTorch wheels (`uv sync`): CUDA on
Linux, MPS on macOS arm64. No separate `requirements.txt` is needed.

```bash
uv sync
```

For running on Runpod (GPU sizing, storage layout, cache management), see
[README_RUNPOD.md](README_RUNPOD.md).

### Apple Silicon (local, MPS)

The tools run natively on the Apple GPU via PyTorch MPS — no CUDA, no pod:

```bash
uv run dltb-oneshot --model sd-turbo --input input_example/test_512.png \
    --num-inference-steps 1 --strength 1.0
```

- The compute backend is auto-detected (CUDA → MPS); `--device {cuda,mps,cpu}`
  overrides it. CPU is never selected implicitly (a silent CPU run would be
  ~100x slower); `--device cpu` forces it.
- Attention slicing is enabled automatically on MPS (16 GB-class unified
  memory).
- `scripts/smoke.sh` is the end-to-end local check (sd-turbo, ~2.5 GB model
  download); `scripts/smoke-local.sh` runs a single-frame pass for every model
  that fits (sd-turbo, sdxl-turbo, flux2-klein-4b) and reports wall clock.
- Keep local and pod runs in separate `--output-dir` subtrees (run tags do
  not encode the device; `untracked/output_<model>` is the local default).
- If an op is unimplemented on MPS, `PYTORCH_ENABLE_MPS_FALLBACK=1` runs it
  on CPU (silently slow; use only if an op errors).
- MPS is not bit-identical to CUDA: compare frames only within one device.

## Requirements

`flux2-klein-9b` requires you to log in with your HuggingFace account and
approve its license at
https://huggingface.co/black-forest-labs/FLUX.2-klein-9B, then set an
`HF_TOKEN` environment variable with an access token from
https://huggingface.co/settings/tokens (read-only is enough).

`dltb-distance`, `dltb-distance-loop` and `dltb-distance-feedback` download
DreamSim weights (~1.2 GB
checkpoint zip plus backbone checkpoints, ~2.7 GB for the default ensemble)
from GitHub releases into `untracked/models` (gitignored) on first use; `--cache-dir` moves it and
`--dreamsim-type` picks a cheaper single-backbone variant. `untracked/` is not
part of pod bundles, so each fresh pod downloads them again.

## Usage

```bash
# Single-pass baseline
uv run dltb-oneshot --model sd-turbo --input menu.png

# Free-running drift experiment (frames + timelapse.mp4)
uv run dltb-iterate --model sd-turbo --input menu.png --iterations 200

# Video pipeline simulation: stateful blend + all three failure tails
uv run dltb-continuous --model flux-schnell --input clip.mp4 --mode stateful \
    --anchor-blend 0.3 --max-frames 300 --tail-frames 60 --tail-modes freeze,free,black

# Klein editors: prompt is the per-pass edit-strength knob
uv run dltb-klein --input clip.mp4 --prompt "slightly enhance the fine details" \
    --tail-frames 60 --tail-modes freeze

# Klein dual-reference conditioning: [state, fresh frame] as two clean refs
uv run dltb-klein --input clip.mp4 --conditioning dual-ref \
    --prompt "image 2 is the current frame; keep the appearance of image 1"

# FLUX.1-schnell, CPU-offloaded to fit a 24 GB card
uv run dltb-iterate --model flux-schnell --input menu.png --offload

# Result analysis: pixel drift (CPU-only) and perceptual drift (DreamSim)
uv run python src/dltb/analyze_drift.py untracked/output_sd_turbo/menu_free-running/frames
uv run dltb-distance untracked/output_sd_turbo/menu_free-running/frames --every 5 --json sim.json

# Self-iteration with the DreamSim drift chart in one run
uv run dltb-distance-loop --model sd-turbo --input menu.png --strength 0.4 \
    --prompt "a bronze lion sculpture" --iterations 60

# Static-source feedback (one image as a 200-frame video, carried state
# blended in): 0.0 = dltb-distance-loop, 1.0 = repeated fresh passes
uv run dltb-distance-feedback --model sd-turbo --input menu.png --strength 0.4 \
    --anchor-blend 0.3 --frames 200 --prompt "a bronze lion sculpture"

# Where does that loop stabilize? Sweep anchor-blend: stable blends keep their
# two plateau frames + a summary table, unstable ones move to unstable/
uv run dltb-stability-sweep --model sd-turbo --input menu.png --strength 0.4 \
    --min-blend 0.05 --max-blend 0.6 --step 0.05 --frames 200 \
    --prompt "a bronze lion sculpture"
```

Runs land in `untracked/output_<model>/<input stem>_<tag>/` (e.g. `menu_free-running/`,
`clip_stateful-a0.3_tailsfreeze-free-black60/`): the untouched input frame,
the saved frames, and the output video(s). `dltb-distance-loop` and
`dltb-distance-feedback` use fixed tags (no input stem) under the same model
tree — `distance_s<strength>/`, `feedback_s<strength>_a<blend>/`, and the
sweep root `stability_s<strength>/` (with `stable/`, `unstable/`,
`stability_sweep.csv` and `stability_plot.png` inside); since prompt/steps/seed are not part of those
tags, pass `--output-dir` subtrees when varying them — their `run.json`
records the settings of each run.

Run `uv run <tool> --help` for all options (strength, steps, seed handling,
resolution, prompt, save frequency, video FPS, ...).

## Inputs

Sample inputs (a test image and a short clip, provenance in `SOURCES.txt`)
live in `input_example/`. The tools take any `--input` path, but the sweep
and smoke scripts read theirs from a config: copy
`input_example/inputs.env` to `untracked/input/inputs.env` (gitignored), point
`IMG=` and `CLIP=` at your files under `untracked/input/`, and every script
picks them up.
An environment variable (`CLIP=... scripts/sweep.sh`) still wins over the
file; see `scripts/inputs.sh`.