DLTB - Deep Learning Tripping Balls #

This is an attempt to have DLSS5 at home, but weirder.

Trying to test what happens when a generative loop is fed its own output, and what a healthy pipeline (fresh frame + carried state + motion vectors) does instead.

Tools #

The console scripts share the library code in src/dltb/ (models, imaging, output, args):

• dltb-oneshot — single image, single model pass: the anchored regime reduced to its fixed point. (Anchored iteration with --fixed-seed just repeats the same image every pass, so one pass already says everything.)
• dltb-iterate — free-running image self-iteration: each pass consumes the model's previous output (P_n = f(P_{n-1})); saves every --save-everyth frame and assembles an mp4 timelapse.
• dltb-continuous — video pipeline simulation: each source frame processed once, either independently (--mode anchored, the boil test — equivalent to --mode stateful --anchor-blend 1.0: same frames, same cost, so the flag only matters for klein dual-ref) or with carried state blended into each new frame (--mode stateful, optical-flow reprojection on by default), plus failure tails that branch from the shared end-of-video state (--tail-modes freeze,free,black).
• dltb-klein — the same video loop, restricted to the FLUX.2 klein editors (flux2-klein-4b, ungated, default / flux2-klein-9b, gated). Klein is a reference-image editor: no --strength, per-pass change scales ~linearly with the blend (default 0.1, far below the img2img models), and the prompt is the de-facto per-pass edit-strength knob (--num-inference-steps is the other). --conditioning dual-ref passes the carried state and fresh frame as two separate clean reference images ([P, N], or [N, P] with --ref-order frame-first) instead of one pixel blend (--anchor-blend is ignored there, so --mode anchored is the only single-reference control); under dual-ref the flow warp is load-bearing (without it the loop freezes into a static consensus — NOTES.md, 2026-09-13) and --reproject-mask caps its artifact floor by patching disoccluded pixels from the fresh frame. scripts/sweep-klein.sh walks its prompt ladder and steps probes under either conditioning; scripts/sweep-klein-mask.sh A/Bs the disocclusion mask on top of dual-ref reprojection (guidance is inert for klein — CFG is disabled and there is no guidance embedding in the distilled checkpoints).
• dltb-assemble — CPU-only local post-processing: encode mp4s from a run's saved PNG frames (main loop + per-tail-mode videos). Complements the split workflow: frames are computed on a rented GPU pod, the run directory is copied home, and videos are (re-)assembled locally — no torch, no accelerator (uv run python src/dltb/assemble.py also works; needs only numpy/PIL/imageio).
• dltb-distance — perceptual counterpart to analyze_drift.py: scores a run's frames with DreamSim (learned DINOv2/CLIP ensemble, not pixels) and reports the distance to the run's source image (dreamsim_to_ref, drift) and to the previous frame (dreamsim_to_prev, perceptual fixed-point detection — ~0 while the loop still chatters means the chatter is invisible). Weights are downloaded on first use (see Requirements); runs on CUDA, Apple MPS, or CPU.
• dltb-distance-loop — the free-running loop and the perceptual metrics in one run, restricted to the strength-capable img2img models (sd-turbo, sdxl-turbo, flux-schnell): every pass is saved, then DreamSim scores each frame against the prepared original and its predecessor into distance_metrics.csv, and both series are charted to distance_plot.png (frame number on X, distance on Y). The diffusion pipeline is unloaded before DreamSim loads, so the two models never share accelerator memory.
• dltb-distance-feedback — the same measurement around dltb-continuous's stateful loop with a static source: the "video" is one image repeated for --frames frames, so source_n = (1-a)*P_{n-1} + a*N unrolls with no motion (no reprojection, no tails — flow between identical frames is zero). The endpoints are free correctness checks: --anchor-blend 0 is exactly dltb-distance-loop (free-running), --anchor-blend 1 is repeated independent passes (boil test); in between, the loop shows how far the model's carried state drags a run away from a fresh render of its own input. Every pass's actual model input (the blend) is saved under inputs/, and DreamSim scores three series into distance_metrics.csv / distance_plot.png: frame vs. original, frame vs. previous frame, and model input vs. original.
• dltb-stabilization — CPU-only verdict on a metrics CSV: did each series stabilize, at which frame, and at what value — or why it needs attention (still rising at the end, wandering, oscillating, drifting). Assumes the run's tail is stationary, bands it robustly (median ± k·MAD), scans backward for the last out-of-band smoothed sample, then diagnoses the plateau; reads the distance_metrics.csv the distance tools write (or drift_metrics.csv from analyze_drift.py) and writes stabilization_report.csv next to the input. No torch, no accelerator (uv run python src/dltb/detect_stabilization.py also works; needs only numpy — matplotlib only for --plot).
• dltb-stability-sweep — sweeps dltb-distance-feedback's --anchor-blend over --min-blend/--max-blend/--step with everything else fixed: ONE pipeline load for the whole range, ONE DreamSim load for all verdicts, and detect_stabilization judging the frame-vs-previous and frame-vs-reference series per blend (the input-vs-ref series is ignored). Blends where both series stabilize are reduced to their two plateau-opening frames plus run.json under stable/<tag>/; every other run moves wholesale to unstable/<tag>/ for inspection; the stable ones are summarized in stability_sweep.csv (anchor-blend, v_prev, i_prev, v_ref, i_ref) and charted to stability_plot.png (stable points only, no placeholders).

Documentation #

Lives in doc:

• doc/temp/: temporary documents, will be deleted after no longer useful. Not to be referenced in non-temporary documents.
• doc/algorithm/ algorithm description
• doc/algorithm/flux2klein.md: FLUX.2-klein pipeline
• doc/algorithm/stabilization.md: the stabilization detector behind dltb-stabilization (algorithm, verified robustness, limitations)

Models #

Local Apple Silicon #

sd-turbo is the practical local target; sdxl-turbo is the stretch goal. flux2-klein-4b can produce single frames on a 16 GB machine (fine for one-shot generation, far too slow for video). flux-schnell (~34 GB bf16) and flux2-klein-9b (gated, ~20-29 GB) do not fit a consumer Mac; nothing blocks trying them anyway (MODELS="..." scripts/smoke-local.sh).

Setup #

The lockfile resolves platform-correct PyTorch wheels (uv sync): CUDA on Linux, MPS on macOS arm64. No separate requirements.txt is needed.

uv sync

For running on Runpod (GPU sizing, storage layout, cache management), see README_RUNPOD.md.

Apple Silicon (local, MPS) #

The tools run natively on the Apple GPU via PyTorch MPS — no CUDA, no pod:

uv run dltb-oneshot --model sd-turbo --input input_example/test_512.png \
    --num-inference-steps 1 --strength 1.0

• The compute backend is auto-detected (CUDA → MPS); --device {cuda,mps,cpu} overrides it. CPU is never selected implicitly (a silent CPU run would be ~100x slower); --device cpu forces it.
• Attention slicing is enabled automatically on MPS (16 GB-class unified memory).
• scripts/smoke.sh is the end-to-end local check (sd-turbo, ~2.5 GB model download); scripts/smoke-local.sh runs a single-frame pass for every model that fits (sd-turbo, sdxl-turbo, flux2-klein-4b) and reports wall clock.
• Keep local and pod runs in separate --output-dir subtrees (run tags do not encode the device; untracked/output_<model> is the local default).
• If an op is unimplemented on MPS, PYTORCH_ENABLE_MPS_FALLBACK=1 runs it on CPU (silently slow; use only if an op errors).
• MPS is not bit-identical to CUDA: compare frames only within one device.

Requirements #

flux2-klein-9b requires you to log in with your HuggingFace account and approve its license at https://huggingface.co/black-forest-labs/FLUX.2-klein-9B, then set an HF_TOKEN environment variable with an access token from https://huggingface.co/settings/tokens (read-only is enough).

dltb-distance, dltb-distance-loop and dltb-distance-feedback download DreamSim weights (~1.2 GB checkpoint zip plus backbone checkpoints, ~2.7 GB for the default ensemble) from GitHub releases into untracked/models (gitignored) on first use; --cache-dir moves it and --dreamsim-type picks a cheaper single-backbone variant. untracked/ is not part of pod bundles, so each fresh pod downloads them again.

Usage #

# Single-pass baseline
uv run dltb-oneshot --model sd-turbo --input menu.png

# Free-running drift experiment (frames + timelapse.mp4)
uv run dltb-iterate --model sd-turbo --input menu.png --iterations 200

# Video pipeline simulation: stateful blend + all three failure tails
uv run dltb-continuous --model flux-schnell --input clip.mp4 --mode stateful \
    --anchor-blend 0.3 --max-frames 300 --tail-frames 60 --tail-modes freeze,free,black

# Klein editors: prompt is the per-pass edit-strength knob
uv run dltb-klein --input clip.mp4 --prompt "slightly enhance the fine details" \
    --tail-frames 60 --tail-modes freeze

# Klein dual-reference conditioning: [state, fresh frame] as two clean refs
uv run dltb-klein --input clip.mp4 --conditioning dual-ref \
    --prompt "image 2 is the current frame; keep the appearance of image 1"

# FLUX.1-schnell, CPU-offloaded to fit a 24 GB card
uv run dltb-iterate --model flux-schnell --input menu.png --offload

# Result analysis: pixel drift (CPU-only) and perceptual drift (DreamSim)
uv run python src/dltb/analyze_drift.py untracked/output_sd_turbo/menu_free-running/frames
uv run dltb-distance untracked/output_sd_turbo/menu_free-running/frames --every 5 --json sim.json

# Self-iteration with the DreamSim drift chart in one run
uv run dltb-distance-loop --model sd-turbo --input menu.png --strength 0.4 \
    --prompt "a bronze lion sculpture" --iterations 60

# Static-source feedback (one image as a 200-frame video, carried state
# blended in): 0.0 = dltb-distance-loop, 1.0 = repeated fresh passes
uv run dltb-distance-feedback --model sd-turbo --input menu.png --strength 0.4 \
    --anchor-blend 0.3 --frames 200 --prompt "a bronze lion sculpture"

# Where does that loop stabilize? Sweep anchor-blend: stable blends keep their
# two plateau frames + a summary table, unstable ones move to unstable/
uv run dltb-stability-sweep --model sd-turbo --input menu.png --strength 0.4 \
    --min-blend 0.05 --max-blend 0.6 --step 0.05 --frames 200 \
    --prompt "a bronze lion sculpture"

Runs land in untracked/output_<model>/<input stem>_<tag>/ (e.g. menu_free-running/, clip_stateful-a0.3_tailsfreeze-free-black60/): the untouched input frame, the saved frames, and the output video(s). dltb-distance-loop and dltb-distance-feedback use fixed tags (no input stem) under the same model tree — distance_s<strength>/, feedback_s<strength>_a<blend>/, and the sweep root stability_s<strength>/ (with stable/, unstable/, stability_sweep.csv and stability_plot.png inside); since prompt/steps/seed are not part of those tags, pass --output-dir subtrees when varying them — their run.json records the settings of each run.

Run uv run <tool> --help for all options (strength, steps, seed handling, resolution, prompt, save frequency, video FPS, ...).

Inputs #

Sample inputs (a test image and a short clip, provenance in SOURCES.txt) live in input_example/. The tools take any --input path, but the sweep and smoke scripts read theirs from a config: copy input_example/inputs.env to untracked/input/inputs.env (gitignored), point IMG= and CLIP= at your files under untracked/input/, and every script picks them up. An environment variable (CLIP=... scripts/sweep.sh) still wins over the file; see scripts/inputs.sh.