From 46006c02406475138484af89d4ab7824e65136f3 Mon Sep 17 00:00:00 2001 From: Refinement Systems Date: Sun, 13 Sep 2026 10:27:27 +0200 Subject: [PATCH] ready for a flux2klein run --- AGENTS.md | 12 ++- NOTES.md | 20 ++--- README_RUNPOD.md | 37 +++++--- input_example/inputs.env | 6 ++ scripts/smoke.sh | 52 +++++++++-- scripts/sweep-klein.sh | 106 +++++++++++++++++------ scripts/sweep-prompt.sh | 183 +++++++++++++++++++++++++++++++++++++++ scripts/sweep.sh | 40 ++++++--- 8 files changed, 387 insertions(+), 69 deletions(-) create mode 100755 scripts/sweep-prompt.sh diff --git a/AGENTS.md b/AGENTS.md index 6cf29ee..ce438f6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,8 +36,11 @@ Experiment drivers (bash, env-var configurable, all support `DRY_RUN=1`): ```bash DRY_RUN=1 scripts/sweep.sh # video sweep: boil test, blends, tails -DRY_RUN=1 scripts/sweep-klein.sh # klein prompt ladder + steps probes +DRY_RUN=1 scripts/sweep-klein.sh # klein prompt ladder + steps probes + + # reproject A/B (default under dual-ref) +DRY_RUN=1 scripts/sweep-prompt.sh # free-running prompt x strength sweep on one image scripts/smoke.sh # tiny run of every tool; needs GPU + # (SMOKE_KLEIN=1 adds dltb-klein, both conditionings) python3 src/dltb/analyze_drift.py # CPU-only drift metrics ``` @@ -132,6 +135,7 @@ scripts/smoke.sh && scripts/sweep.sh - 48 GB VRAM recommended; `--offload` for the two big models on smaller cards. - Container disk is **ephemeral on stop AND restart** — copy `output/` out before stopping. Pod sshd isn't started by default (NOTES.md has the fix). -- HF cache is keep-one per model (sweep.sh evicts at model boundaries only); - `hf-cache.sh keep/clean` refuse to run without `HF_HOME` set — keep that - safety guard. +- HF cache: swept models are kept by default (all five ≈ 87.5 GB fit the 150 GB + pod disk); `EVICT_CACHE=1` (env or `inputs.env`, like `IMG`/`CLIP`) restores + keep-one eviction at sweep model boundaries. `hf-cache.sh keep/clean` refuse + to run without `HF_HOME` set — keep that safety guard. diff --git a/NOTES.md b/NOTES.md index 2355c45..b41b414 100644 --- a/NOTES.md +++ b/NOTES.md @@ -170,7 +170,7 @@ dropped on the floor: see [FLUX.2 klein: --guidance-scale is inert](#flux2-klein---guidance-scale-is-inert-step-wise-distilled) below. -## flux2-klein-9b anchor-blend calibration (paused — needs a redesign) +## flux2-klein-9b anchor-blend calibration (superseded by the dltb-klein redesign) Context: for `sd-turbo` / `sdxl-turbo` / `flux-schnell`, the stateful sweep at `--anchor-blend 0.1/0.3/0.5` produced an effect judged too strong (a @@ -191,8 +191,7 @@ all, or whether a different conditioning/topology is needed (`flux2-klein-9b` is a reference-image editor: no `--strength`, full 4-step regeneration). 2026-09-21: that topology redesign is now in the tree as `dltb-klein` + -`scripts/sweep-klein.sh` (prompt-as-strength ladder, guidance probes). The -pre-restructure scripts they were derived from live under `reference/`. +`scripts/sweep-klein.sh` (prompt-as-strength ladder, guidance probes). Later the same day the guidance-probe leg turned out to be inert for klein — see the next section. @@ -225,9 +224,9 @@ never reaches the model, via three independent points: - `--negative-prompt` is equally inert (negative embeddings are only computed under CFG). - The sweep was killed mid-probe; the meaningful legs (prompt ladder, - weathering attractor) were already on disk. `output/flux2-klein-9b/guidance2.0/` - is a partial duplicate of `prompt-enhance-slight/` — delete it (and - `guidance4.0/` if it ever started). + weathering attractor) were already on disk. (A partial `guidance2.0/` + duplicate of `prompt-enhance-slight/` sat in the pod's output tree — moot + since the pod's container disk is ephemeral.) **Follow-ups applied 2026-09-13:** the guidance leg of `sweep-klein.sh` is replaced by a `--num-inference-steps` probe (`STEPS="2 8"`, bracketing the @@ -304,8 +303,9 @@ state and the fresh frame can both be conditioning inputs: uv run dltb-klein --model flux2-klein-4b --input input/video_cropped.mp4 \ --conditioning dual-ref --ref-order state-first --max-frames 2 -Remaining follow-ups: add a `SMOKE_KLEIN=1` leg to `scripts/smoke.sh` (2 frames -+ 1 tail frame, flux2-klein-4b; off by default because of the 15 GB download); -the `CONDITIONING=dual-ref REF_ORDER={state-first,frame-first}` toggle in -`scripts/sweep-klein.sh` is done (it also swaps in role-naming prompts). +Remaining follow-ups: none in-tree — the `SMOKE_KLEIN=1` smoke leg, the +reproject A/B (`REPROJECT=1|0|ab`, default `ab` under dual-ref), and +REF_ORDER-aware role-naming prompts are all in `scripts/smoke.sh` / +`scripts/sweep-klein.sh`. What is still pending is the GPU validation itself +(prompt ladder + order A/B under dual-ref, then the reproject A/B). diff --git a/README_RUNPOD.md b/README_RUNPOD.md index 08d0d35..b01b979 100644 --- a/README_RUNPOD.md +++ b/README_RUNPOD.md @@ -12,7 +12,7 @@ real pod; prices and availability drift, so re-check the console. | Container disk | **150 GB**, ephemeral — holds the repo, the HF cache and the outputs | | Volume disk | **0** (none) | | Network volume | **none** — it would pin the pod to one datacenter and kill consumer-GPU availability | -| Cache policy | **keep-one model**, enforced automatically by `scripts/sweep.sh` | +| Cache policy | **keep all swept models** (~87.5 GB fits); `EVICT_CACHE=1` restores keep-one for small disks | | Data copy-out | before **stop / restart / terminate** — the container disk is wiped on all three | ```bash @@ -105,9 +105,12 @@ showed ~85 MB used with the 12 GB image present. | `flux2-klein-9b` | 32.3 GB | | **all five** | **~87.5 GB** | -With the keep-one policy the steady-state requirement is one model (~33 GB max) -+ xet (≤10 GB) + outputs (a full video sweep is a few GB) ≈ **45 GB**. 150 GB -gives ~3× headroom; it fits all five at once too, but there is no reason to. +The default cache policy keeps every swept model (all five ≈ 87.5 GB) + xet +(≤10 GB) + outputs (a full video sweep is a few GB) — comfortable on the +150 GB disk, and revisiting an earlier model costs no re-download. With +`EVICT_CACHE=1` (keep-one, `input/inputs.env`-configurable) the steady-state +requirement drops to one model (~33 GB max) + xet + outputs ≈ **45 GB**, for +smaller container disks. ### Why no volume disk / network volume @@ -200,7 +203,8 @@ cd imgiter- uv sync --frozen # re-points the editable install from /opt/imgiter to this tree scripts/sweep.sh # full sweep; add OFFLOAD=1 on <48 GB GPUs for the big models -scripts/sweep-klein.sh # klein prompt ladder + guidance probes (single model) +scripts/sweep-klein.sh # klein prompt ladder + steps probes + reproject A/B (single model) +scripts/sweep-prompt.sh # free-running prompt (x strength) sweep on one image ``` - Dependencies are baked into the image at `/opt/imgiter/.venv` @@ -223,7 +227,9 @@ bundle`, send. A one-off run with a different clip does not need the file: Smoke test after deploying a new bundle — one tiny run of every tool (`dltb-oneshot`, `dltb-iterate`, `dltb-continuous` both modes + tails, and the -hf-cache helper), with artifact checks: +hf-cache helper), with artifact checks. `SMOKE_KLEIN=1` adds the `dltb-klein` +leg (both blend and dual-ref conditioning, `flux2-klein-4b` by default — a +~15 GB download, which is why it is off by default): ```bash scripts/smoke.sh @@ -238,12 +244,21 @@ Useful `sweep.sh` env overrides: `MODELS`, `BLENDS`, `BASELINE`, `MAX_FRAMES`, `REPROJECT`, `CLIP`, `EXTRA_ARGS`, `DRY_RUN`, `SKIP_GPU_CHECK`. Klein regime (`scripts/sweep-klein.sh`, drives `dltb-klein`): `MODEL` -(default `flux2-klein-9b`; `4b` is ungated), `BLEND` (default `0.1`), -`STEPS` (default "2 8", bracketing the default 4), plus the shared +(default `flux2-klein-9b`; `4b` is ungated), `CONDITIONING` (`blend`, the +default, or `dual-ref`), `REF_ORDER` (`state-first`, default; dual-ref only), +`REPROJECT` (`1`/`0`/`ab`; defaults to `ab` under dual-ref — every leg runs +with and without reprojection, the A/B), `BLEND` (default `0.1`), `STEPS` +(default "2 8", bracketing the default 4), plus the shared `CLIP`/`MAX_FRAMES`/`TAIL_FRAMES`/`TAIL_MODES`/`SAVE_EVERY`/`EXTRA_ARGS`/ `DRY_RUN`/`SKIP_GPU_CHECK`. Single-model, so no hf-cache eviction between runs. +Prompt sweep on one image (`scripts/sweep-prompt.sh`, drives `dltb-iterate`): +`MODEL` (default `sd-turbo`), `IMG`, `ITERATIONS` (default 20), `SAVE_EVERY` +(default 1 — every frame), `VIDEO_FPS`, `STRENGTHS` (default none; full +prompts × strengths cross product, classic img2img models only), +`STRENGTH_MODELS`, plus `EXTRA_ARGS`/`DRY_RUN`/`SKIP_GPU_CHECK`. + ## 5. Model cache management `scripts/hf-cache.sh`: @@ -255,9 +270,11 @@ scripts/hf-cache.sh clean # drop all caches DRY_RUN=1 DEBUG=1 scripts/hf-cache.sh keep flux-schnell # preview ``` -- `sweep.sh` calls `keep ` at the top of every model iteration, so the +- With `EVICT_CACHE=1` (off by default — see `input_example/inputs.env`), + `sweep.sh` calls `keep ` at the top of every model iteration, so the disk only ever holds the model being swept. All lines are prefixed `hf-cache:` - and go into `output/sweep_*.log`. + and go into `output/sweep_*.log`. With the default `EVICT_CACHE=0` the cache + keeps every swept model (~87.5 GB for all five, fits the 150 GB disk). - Eviction happens **only at model boundaries**, never between the separate runs of one model — otherwise the same multi-GB weights would be re-downloaded once per run. diff --git a/input_example/inputs.env b/input_example/inputs.env index 7690f11..4c6f9a9 100644 --- a/input_example/inputs.env +++ b/input_example/inputs.env @@ -12,3 +12,9 @@ IMG="${IMG:-input_example/test_512.png}" CLIP="${CLIP:-input_example/video_cropped.mp4}" + +# Keep-one model-cache eviction at sweep model boundaries (scripts/sweep.sh). +# 0 = keep every swept model cached (default; all five ≈ 87.5 GB fit the +# 150 GB pod disk, so revisiting a model costs no re-download), +# 1 = evict every other cached model when a new model starts (small disks). +EVICT_CACHE="${EVICT_CACHE:-0}" diff --git a/scripts/smoke.sh b/scripts/smoke.sh index 0036fdb..a050551 100755 --- a/scripts/smoke.sh +++ b/scripts/smoke.sh @@ -14,8 +14,8 @@ # # smoke.sh -- post-deploy smoke test: one tiny run of every tool. # -# Verifies the three console scripts and the cache helper against a real GPU -# using the cheapest model, inputs from input/inputs.env (falling back to the +# Verifies the console scripts and the cache helper against a real GPU using +# the cheapest model, inputs from input/inputs.env (falling back to the # tracked input_example/ files), and minimal budgets: # # 0. scripts/hf-cache.sh status (dltb.models import + cache probe) @@ -24,6 +24,12 @@ # 3. dltb-continuous stateful 3 reprojected source frames + 2 frames # per tail (freeze, free, black) # 4. dltb-continuous anchored 2 frames boil test +# 5. dltb-klein (optional) SMOKE_KLEIN=1: both conditionings +# (blend + dual-ref), 2 source frames + +# 1 freeze-tail frame -- the cheapest +# end-to-end check of the klein loop, +# including the dual-ref reference-list +# code path # # Every step writes under output/smoke/ (wiped at start); the expected # artifacts are checked for existence and non-emptiness afterwards. Any @@ -34,9 +40,10 @@ # MODEL=sdxl-turbo scripts/smoke.sh # smoke another model # SKIP_GPU_CHECK=1 scripts/smoke.sh # bypass the CUDA preflight # -# MODEL defaults to sd-turbo (2.4 GB). hf-cache keep-one means the disk -# holds whichever model the last sweep ended on -- check step 0's output -# and set MODEL= to skip the re-download. +# MODEL defaults to sd-turbo (2.4 GB). With the default EVICT_CACHE=0 every +# swept model stays cached, so check step 0's output and set MODEL= to skip any re-download (under EVICT_CACHE=1 the disk holds only +# whichever model the last sweep ended on). # # Environment: # MODEL model key to smoke (default sd-turbo) @@ -44,6 +51,9 @@ # present, else the tracked # input_example/ files; see # scripts/inputs.sh) +# SMOKE_KLEIN=1 also smoke dltb-klein, both conditionings (off by default: +# flux2-klein-4b is a ~15 GB download) +# KLEIN_MODEL klein model for that leg (default flux2-klein-4b) # SKIP_GPU_CHECK=1 bypass the CUDA preflight # # Log: output/smoke_.log @@ -155,5 +165,35 @@ run uv run dltb-continuous --model "$MODEL" --input "$CLIP" \ --output-dir "$OUT" check "$OUT/${clip_stem}_anchored/processed_anchored.mp4" +# 5. dltb-klein, optional (SMOKE_KLEIN=1): both conditionings, klein-default +# settings (no strength; 4 steps), 2 source frames + 1 freeze-tail frame. +# klein run tags: stateful-a vs dualref, + _tailsfreeze1. +if [[ "${SMOKE_KLEIN:-0}" == "1" ]]; then + KLEIN_MODEL="${KLEIN_MODEL:-flux2-klein-4b}" + log "" + log "smoke: klein leg (model=$KLEIN_MODEL, both conditionings)" + if [[ "$KLEIN_MODEL" == "flux2-klein-9b" && -z "${HF_TOKEN:-}" ]]; then + log "SMOKE FAIL: HF_TOKEN is not set (required for flux2-klein-9b)" + exit 1 + fi + + run uv run dltb-klein --model "$KLEIN_MODEL" --input "$CLIP" \ + --mode stateful --conditioning blend --anchor-blend 0.1 --reproject \ + --max-frames 2 --tail-frames 1 --tail-modes freeze \ + --save-every 1 --output-dir "$OUT/klein-blend" + d="$OUT/klein-blend/${clip_stem}_stateful-a0.1_tailsfreeze1" + check "$d/processed_stateful.mp4" + check "$d/tail_freeze.mp4" + + run uv run dltb-klein --model "$KLEIN_MODEL" --input "$CLIP" \ + --mode stateful --conditioning dual-ref --ref-order state-first \ + --reproject \ + --max-frames 2 --tail-frames 1 --tail-modes freeze \ + --save-every 1 --output-dir "$OUT/klein-dualref" + d="$OUT/klein-dualref/${clip_stem}_dualref_tailsfreeze1" + check "$d/processed_stateful.mp4" + check "$d/tail_freeze.mp4" +fi + log "" -log "smoke: PASS -- all three tools produced their artifacts under $OUT" +log "smoke: PASS -- all requested tools produced their artifacts under $OUT" diff --git a/scripts/sweep-klein.sh b/scripts/sweep-klein.sh index 45b5652..8c88d08 100755 --- a/scripts/sweep-klein.sh +++ b/scripts/sweep-klein.sh @@ -16,8 +16,20 @@ # no ghosted blend for the editor to parse; state/anchor weighting # is done by the model. BLEND is ignored; REF_ORDER picks [P,N] # (state-first, default) or [N,P]. The prompt table switches to -# role-naming prompts ("image 2 is the current frame; ...", which -# assumes state-first order -- swap the roles if REF_ORDER=frame-first). +# role-naming prompts whose image indices are DERIVED from +# REF_ORDER ("image N is the current frame; keep the appearance of +# image M, ..."), so the instruction keeps its semantic roles under +# either order -- no manual role swapping. +# +# REPROJECT (stateful mode): 1 = warp the carried reference by source-frame +# optical flow before pairing (emulates engine motion vectors); 0 = pass +# both references clean. Under dual-ref the default is AB: every leg runs +# TWICE, once per setting -- the sweep itself is the reproject A/B (does +# warp alignment help, or do warp artifacts get amplified by an editor +# trained on clean references?). The run tags distinguish the variants +# (..._dualref vs ..._dualref-norepro), so nothing collides; both variants +# of a leg complete before the next leg starts, so partial results can be +# copied off the pod mid-sweep. Pin one variant with REPROJECT=1 or 0. # # What this sweep runs (all --mode stateful, one source pass per prompt): # @@ -25,10 +37,12 @@ # i.e. rising per-pass edit intensity. The freeze tail # shows whether the edit keeps compounding on static # input (the generative-ratchet / static-menu case). -# 2. Semantic attractor : a THEMATIC instruction (weathering). If klein works -# as trained, the loop converges toward "maximally -# weathered" instead of melting - directed attractor -# vs. undirected collapse. +# 2. Semantic attractor : a THEMATIC instruction (coral overgrowth -- the +# tracked example clip is an octopus in a coral +# habitat). If klein works as trained, the loop +# converges toward "maximally overgrown" instead +# of melting - directed attractor vs. undirected +# collapse. # 3. Steps probes : the mild prompt at --num-inference-steps 2 / 8 # (bracketing the klein card default of 4, which the # ladder legs already run). Steps are the one direct @@ -49,6 +63,7 @@ # MODEL=flux2-klein-4b BLEND=0.2 scripts/sweep-klein.sh # CONDITIONING=dual-ref scripts/sweep-klein.sh # CONDITIONING=dual-ref REF_ORDER=frame-first scripts/sweep-klein.sh +# CONDITIONING=dual-ref REPROJECT=1 scripts/sweep-klein.sh # pin one A/B variant # # Environment overrides: # MODEL klein model key (default flux2-klein-9b; 4b is ungated) @@ -66,6 +81,9 @@ # TAIL_MODES tail scenario list (default freeze; "freeze,free" etc. # NOTE: under dual-ref, free = single-reference regeneration # from state alone) +# REPROJECT 1 | 0 | ab (default: ab under dual-ref = both +# variants per leg, the A/B; 1 under blend. See the REPROJECT +# paragraph above) # SAVE_EVERY save every Nth frame (default 10) # STEPS steps-probe values (default "2 8", bracketing the default 4; # empty = skip the probe) @@ -109,19 +127,42 @@ case "$REF_ORDER" in *) echo "sweep-klein: REF_ORDER must be state-first or frame-first (got '$REF_ORDER')" >&2; exit 1 ;; esac +# Reprojection: default AB under dual-ref (the sweep doubles as the A/B), +# pinned on under blend (parity with scripts/sweep.sh). +REPROJECT="${REPROJECT:-}" +if [[ -z "$REPROJECT" ]]; then + if [[ "$CONDITIONING" == "dual-ref" ]]; then REPROJECT=ab; else REPROJECT=1; fi +fi +case "$REPROJECT" in + 1|0|ab) ;; + *) echo "sweep-klein: REPROJECT must be 1, 0, or ab (got '$REPROJECT')" >&2; exit 1 ;; +esac +if [[ "$REPROJECT" == "ab" ]]; then REPROJECT_LIST="1 0"; else REPROJECT_LIST="$REPROJECT"; fi + # ---------------------------------------------------------------- prompts ---- # slug|prompt pairs. The slug becomes the output subdirectory; keep slugs short, # lowercase, hyphenated. Empty prompt = preservation baseline (no flag passed). +# Prompts match the tracked example clip (octopus in a coral habitat; see +# input_example/SOURCES.txt) -- point CLIP at your own and adjust OVERGROWTH. +# +# Role-naming for dual-ref: image indices are resolved from REF_ORDER, so +# "image $FRAME_IMG is the current frame; keep the appearance of image +# $STATE_IMG" keeps its semantic roles whether the list is [P, N] or [N, P]. +if [[ "$REF_ORDER" == "state-first" ]]; then + STATE_IMG=1 + FRAME_IMG=2 +else + STATE_IMG=2 + FRAME_IMG=1 +fi + if [[ "$CONDITIONING" == "dual-ref" ]]; then - # Role-naming prompts. Phrasing assumes state-first order (image 1 = carried - # state/appearance reference, image 2 = current frame/content target); - # swap the roles if REF_ORDER=frame-first. PROMPT_TABLE=( "neutral|" - "enhance-slight|image 2 is the current frame; keep the appearance of image 1, slightly enhancing fine details" - "enhance-photo|image 2 is the current frame; keep the appearance of image 1, enhancing details and lighting to look photorealistic" - "enhance-dramatic|image 2 is the current frame; keep the appearance of image 1, dramatically enhancing every texture and surface detail" - "weathering|image 2 is the current frame; keep the appearance of image 1, adding more weathering, moss and water stains to the stone" + "enhance-slight|image ${FRAME_IMG} is the current frame; keep the appearance of image ${STATE_IMG}, slightly enhancing fine details" + "enhance-photo|image ${FRAME_IMG} is the current frame; keep the appearance of image ${STATE_IMG}, enhancing details and lighting to look photorealistic" + "enhance-dramatic|image ${FRAME_IMG} is the current frame; keep the appearance of image ${STATE_IMG}, dramatically enhancing every texture and surface detail" + "overgrowth|image ${FRAME_IMG} is the current frame; keep the appearance of image ${STATE_IMG}, adding more coral and marine growth over every surface" ) else PROMPT_TABLE=( @@ -129,14 +170,14 @@ else "enhance-slight|slightly enhance the fine details" "enhance-photo|enhance details and lighting, make it photorealistic" "enhance-dramatic|dramatically enhance every texture and surface detail" - "weathering|add more weathering, moss and water stains to the stone" + "overgrowth|add more coral and marine growth over every surface" ) fi # Steps probes reuse the mild-enhancement prompt. PROBE_SLUG="enhance-slight" if [[ "$CONDITIONING" == "dual-ref" ]]; then - PROBE_PROMPT="image 2 is the current frame; keep the appearance of image 1, slightly enhancing fine details" + PROBE_PROMPT="image ${FRAME_IMG} is the current frame; keep the appearance of image ${STATE_IMG}, slightly enhancing fine details" else PROBE_PROMPT="slightly enhance the fine details" fi @@ -190,31 +231,44 @@ run() { uv run dltb-klein "$@" 2>&1 | tee -a "$LOG" } -log "sweep-klein: model=$MODEL clip=$CLIP conditioning=$CONDITIONING ref_order=$REF_ORDER blend=$BLEND" +log "sweep-klein: model=$MODEL clip=$CLIP conditioning=$CONDITIONING ref_order=$REF_ORDER blend=$BLEND reproject=$REPROJECT" +if [[ "$CONDITIONING" == "dual-ref" ]]; then + log "sweep-klein: dual-ref roles: image $STATE_IMG = carried state, image $FRAME_IMG = current frame" +fi log "sweep-klein: max_frames=${MAX_FRAMES:-} tail=${TAIL_FRAMES}x${TAIL_MODES} steps='${STEPS:-}'" log "sweep-klein: log=$LOG" # ------------------------------------------------------- 1+2. prompt ladder ---- +# Inner loop over the reproject settings: both A/B variants of a leg finish +# before the next leg starts (comparable pairs land on disk early). for entry in "${PROMPT_TABLE[@]}"; do slug="${entry%%|*}" prompt="${entry#*|}" - args=(${common[@]+"${common[@]}"} --output-dir "output/$MODEL/prompt-$slug") - if [[ -n "$prompt" ]]; then args+=(--prompt "$prompt"); fi + for R in $REPROJECT_LIST; do + args=(${common[@]+"${common[@]}"} $([[ "$R" == "1" ]] && echo --reproject || echo --no-reproject) + --output-dir "output/$MODEL/prompt-$slug") + if [[ -n "$prompt" ]]; then args+=(--prompt "$prompt"); fi - log "" - log "########## prompt '$slug': ${prompt:-} ##########" - run "${args[@]}" ${EXTRA_ARGS:+$EXTRA_ARGS} + log "" + log "########## prompt '$slug' [reproject=$R]: ${prompt:-} ##########" + run "${args[@]}" ${EXTRA_ARGS:+$EXTRA_ARGS} + log "sweep-klein: leg done: output/$MODEL/prompt-$slug (safe to copy off mid-sweep)" + done done # --------------------------------------------------------- 3. steps probe ---- if [[ -n "${STEPS:-}" ]]; then for N in $STEPS; do - log "" - log "########## steps probe: '$PROBE_SLUG' @ num-inference-steps=$N ##########" - run ${common[@]+"${common[@]}"} \ - --prompt "$PROBE_PROMPT" --num-inference-steps "$N" \ - --output-dir "output/$MODEL/steps$N" ${EXTRA_ARGS:+$EXTRA_ARGS} + for R in $REPROJECT_LIST; do + log "" + log "########## steps probe: '$PROBE_SLUG' @ num-inference-steps=$N [reproject=$R] ##########" + run ${common[@]+"${common[@]}"} \ + $([[ "$R" == "1" ]] && echo --reproject || echo --no-reproject) \ + --prompt "$PROBE_PROMPT" --num-inference-steps "$N" \ + --output-dir "output/$MODEL/steps$N" ${EXTRA_ARGS:+$EXTRA_ARGS} + log "sweep-klein: leg done: output/$MODEL/steps$N (safe to copy off mid-sweep)" + done done fi diff --git a/scripts/sweep-prompt.sh b/scripts/sweep-prompt.sh new file mode 100755 index 0000000..2b3a93e --- /dev/null +++ b/scripts/sweep-prompt.sh @@ -0,0 +1,183 @@ +#!/usr/bin/env bash + +# Permission to use, copy, modify, and/or distribute this software for +# any purpose with or without fee is hereby granted. +# +# THE SOFTWARE IS PROVIDED “AS IS” AND THE AUTHOR DISCLAIMS ALL +# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES +# OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE +# FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY +# DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER +# IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT +# OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + +# +# sweep-prompt.sh -- free-running prompt sweep on ONE image (dltb-iterate). +# +# For each prompt (optionally crossed with strength values) run a SHORT +# free-running self-iteration of one model on a single input image: +# +# P_n = f(P_{n-1}, prompt [, strength]) +# +# Every pass is saved (frames/frame_NNNN.png, --save-every 1) and assembled +# into a per-run timelapse.mp4, so a whole leg is a few dozen images -- cheap +# enough to walk a ladder of prompts in one sitting. The prompt table defaults +# match the tracked example input (a bronze lion sculpture; see +# input_example/SOURCES.txt): a neutral preservation baseline, a rising +# enhancement ladder, and one thematic compounding instruction (patina). +# +# STRENGTH axis: full prompts x strengths cross product, but only for the +# classic img2img models (STRENGTH_MODELS) -- klein is a reference editor +# with no --strength, and there the prompt is already the per-pass knob +# (see scripts/sweep-klein.sh for the klein regime). Setting STRENGTHS with +# a klein model logs a note and runs the prompt ladder alone. +# +# Each run writes its own --output-dir subtree (dltb-iterate's directory tag +# is a constant _free-running, so without subtrees the legs would +# silently overwrite each other): +# +# output//prompt-/ no strength axis +# output//prompt-/strength/... with the axis +# +# Every subtree is complete the moment its run finishes, so partial results +# can be copied off the pod while the sweep keeps going. +# +# Usage (from any directory inside the repo): +# scripts/sweep-prompt.sh +# DRY_RUN=1 scripts/sweep-prompt.sh +# MODEL=sdxl-turbo ITERATIONS=40 scripts/sweep-prompt.sh +# MODEL=sd-turbo STRENGTHS="0.4 0.7" scripts/sweep-prompt.sh +# +# Environment overrides: +# MODEL model key (default sd-turbo, cheapest) +# IMG input image (default: input/inputs.env if +# present, else the tracked +# input_example/test_512.png; +# see scripts/inputs.sh) +# ITERATIONS passes per run (default 20) +# SAVE_EVERY save every Nth frame (default 1 -- every frame) +# VIDEO_FPS timelapse fps (default 12) +# STRENGTHS strength values, space-separated (default: none -> single +# run per prompt at the tool's default strength) +# STRENGTH_MODELS models the axis applies to +# (default sd-turbo sdxl-turbo flux-schnell) +# EXTRA_ARGS extra flags, word-split, appended to every run +# DRY_RUN=1 print commands without executing anything +# SKIP_GPU_CHECK=1 bypass the CUDA preflight +# +# Log: output/sweep_prompt_.log +# Stops at the first failing run (set -euo pipefail). + +set -euo pipefail + +cd "$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + +# Input files: env > input/inputs.env (user) > input_example/inputs.env. +source scripts/inputs.sh + +MODEL="${MODEL:-sd-turbo}" +IMG="${IMG:-input_example/test_512.png}" +ITERATIONS="${ITERATIONS:-20}" +SAVE_EVERY="${SAVE_EVERY:-1}" +VIDEO_FPS="${VIDEO_FPS:-12}" +STRENGTHS="${STRENGTHS:-}" +STRENGTH_MODELS="${STRENGTH_MODELS:-sd-turbo sdxl-turbo flux-schnell}" +EXTRA_ARGS="${EXTRA_ARGS:-}" +DRY_RUN="${DRY_RUN:-0}" + +# -------------------------------------------------------------- prompts ---- +# slug|prompt pairs; the slug becomes the output subdirectory. Empty prompt = +# preservation baseline (no --prompt flag passed). Defaults describe the +# tracked example image (bronze lion sculpture) -- point IMG at your own and +# adjust PATINA/PROBE lines accordingly. +PROMPT_TABLE=( + "neutral|" + "enhance-slight|slightly enhance the fine details" + "enhance-photo|enhance details and lighting, make it look like a professional photograph" + "enhance-dramatic|dramatically enhance every texture and surface detail" + "patina|add more blue-green patina and weathering to the bronze surface" +) + +# -------------------------------------------------------------- preflight ---- +if [[ "$DRY_RUN" != "1" ]]; then + [[ -f "$IMG" ]] || { echo "sweep-prompt: image not found: $IMG" >&2; exit 1; } + command -v uv >/dev/null || { echo "sweep-prompt: 'uv' not on PATH" >&2; exit 1; } + + if [[ "$MODEL" == "flux2-klein-9b" && -z "${HF_TOKEN:-}" ]]; then + echo "sweep-prompt: HF_TOKEN is not set (required for flux2-klein-9b)." >&2 + echo " Accept the license at https://huggingface.co/black-forest-labs/FLUX.2-klein-9B" >&2 + echo " then: export HF_TOKEN=hf_... and re-run." >&2 + exit 1 + fi + + if [[ "${SKIP_GPU_CHECK:-0}" != "1" ]]; then + if ! uv run python -c 'import sys, torch; sys.exit(0 if torch.cuda.is_available() else 1)'; then + echo "sweep-prompt: torch reports no CUDA device -- run this on a GPU pod" >&2 + echo " (SKIP_GPU_CHECK=1 to bypass, DRY_RUN=1 to preview)." >&2 + exit 1 + fi + fi +fi + +# Strength axis applies to classic img2img models only. +STRENGTH_AXIS=0 +case " $STRENGTH_MODELS " in + *" $MODEL "*) STRENGTH_AXIS=1 ;; +esac +if [[ "$STRENGTH_AXIS" != "1" ]]; then + STRENGTHS="" # klein et al.: no --strength semantics; prompts only +fi + +mkdir -p output +LOG="output/sweep_prompt_$(date -u +%Y%m%d-%H%M%S).log" + +log() { echo "$@" | tee -a "$LOG"; } + +run() { + log "" + log "== $(date -u +%Y-%m-%dT%H:%M:%SZ) dltb-iterate $*" + if [[ "$DRY_RUN" == "1" ]]; then + log "DRY RUN" + return 0 + fi + uv run dltb-iterate "$@" 2>&1 | tee -a "$LOG" +} + +log "sweep-prompt: model=$MODEL img=$IMG iterations=$ITERATIONS save_every=$SAVE_EVERY" +log "sweep-prompt: strength_axis=$STRENGTH_AXIS strengths='${STRENGTHS:-}'" +log "sweep-prompt: log=$LOG" +if [[ "$STRENGTH_AXIS" != "1" ]]; then + log "sweep-prompt: NOTE: $MODEL has no --strength semantics -- prompt ladder only" +fi + +for entry in "${PROMPT_TABLE[@]}"; do + slug="${entry%%|*}" + prompt="${entry#*|}" + + if [[ "$STRENGTH_AXIS" == "1" && -n "$STRENGTHS" ]]; then + strengths="$STRENGTHS" + else + strengths="" # single run at the tool's default strength + fi + + for S in ${strengths:-default}; do + [[ "$S" == "default" ]] && S="" + out="output/$MODEL/prompt-$slug" + args=(--model "$MODEL" --input "$IMG" --iterations "$ITERATIONS" + --save-every "$SAVE_EVERY" --video-fps "$VIDEO_FPS") + if [[ -n "$S" ]]; then + out="$out/strength$S" + args+=(--strength "$S") + fi + args+=(--output-dir "$out") + if [[ -n "$prompt" ]]; then args+=(--prompt "$prompt"); fi + + log "" + log "########## prompt '$slug'${S:+ @ strength $S}: ${prompt:-} ##########" + run "${args[@]}" ${EXTRA_ARGS:+$EXTRA_ARGS} + log "sweep-prompt: leg done: $out (safe to copy off mid-sweep)" + done +done + +log "" +log "sweep-prompt: done -- log: $LOG" diff --git a/scripts/sweep.sh b/scripts/sweep.sh index 7de9901..4893ac1 100755 --- a/scripts/sweep.sh +++ b/scripts/sweep.sh @@ -22,9 +22,13 @@ # 3. semantic anchor at the BASELINE blend (--prompt "$DESC", freeze tail) # 4. strength probe, classic img2img models (--strength "$STRENGTH", freeze tail) # -# Before step 1 of each model, scripts/hf-cache.sh evicts every other model's -# HuggingFace cache (keep-one policy: the disk only needs room for the model -# being swept). All runs of one model share the cache; the next model evicts it. +# Cache policy: by default swept models are KEPT in the HuggingFace cache (all +# five fit the 150 GB pod disk at once, ~87.5 GB, so re-runs on an earlier +# model cost nothing). EVICT_CACHE=1 restores the keep-one policy: before step +# 1 of each model, scripts/hf-cache.sh evicts every other cached model, so the +# disk only ever needs room for the model being swept (for small container +# disks). Eviction happens at model boundaries only -- never between the runs +# of one model. # # Stateful video runs reproject the carried state by optical flow before # blending (--reproject, on by default in dltb-continuous): @@ -64,6 +68,11 @@ # STRENGTH strength-probe value (default 0.55) # STRENGTH_MODELS models for the probe (default sd-turbo sdxl-turbo flux-schnell) # DESC semantic-anchor prompt +# (default: describes the tracked example clip -- an +# octopus in a coral habitat; see input_example/SOURCES.txt) +# EVICT_CACHE 1 = keep-one cache policy: evict every other cached model +# at each model boundary (default 0 = keep everything; all +# five models ≈ 87.5 GB fit the 150 GB pod disk) # OFFLOAD=1 add --offload to every run (small GPUs; flux-schnell on 24 GB) # REPROJECT 1 = optical-flow reprojection (default), 0 = naive blend # EXTRA_ARGS extra flags, word-split, appended to every run @@ -89,9 +98,10 @@ TAIL_FRAMES="${TAIL_FRAMES:-60}" SAVE_EVERY="${SAVE_EVERY:-10}" STRENGTH="${STRENGTH:-0.55}" STRENGTH_MODELS="${STRENGTH_MODELS:-sd-turbo sdxl-turbo flux-schnell}" -DESC="${DESC:-first-person gameplay footage in a sunlit stone courtyard}" +DESC="${DESC:-underwater footage of an octopus in a coral habitat}" OFFLOAD="${OFFLOAD:-0}" REPROJECT="${REPROJECT:-1}" +EVICT_CACHE="${EVICT_CACHE:-0}" DRY_RUN="${DRY_RUN:-0}" case "$REPROJECT" in @@ -161,21 +171,25 @@ run() { log "sweep: clip=$CLIP" log "sweep: models='$MODELS' blends='$BLENDS' baseline=$BASELINE" -log "sweep: max_frames=${MAX_FRAMES:-} tail_frames=$TAIL_FRAMES save_every=$SAVE_EVERY reproject=$REPROJECT" +log "sweep: max_frames=${MAX_FRAMES:-} tail_frames=$TAIL_FRAMES save_every=$SAVE_EVERY reproject=$REPROJECT evict_cache=$EVICT_CACHE" log "sweep: log=$LOG" for MODEL in $MODELS; do log "" log "################ $MODEL ################" - # Keep-one cache policy. Evict other models here, at the model boundary -- - # never between the runs of one model: sweep.sh starts a fresh - # `uv run dltb-continuous` per run, so evicting there would re-download - # the same multi-GB weights once per run. - if [[ "$DRY_RUN" == "1" ]]; then - DRY_RUN=1 scripts/hf-cache.sh keep "$MODEL" 2>&1 | tee -a "$LOG" - elif ! scripts/hf-cache.sh keep "$MODEL" 2>&1 | tee -a "$LOG"; then - log "sweep: WARNING -- cache eviction failed for $MODEL; continuing (disk may fill)" + # Optional keep-one cache eviction (EVICT_CACHE=1). Runs at the model + # boundary only -- never between the runs of one model: sweep.sh starts a + # fresh `uv run dltb-continuous` per run, so evicting there would + # re-download the same multi-GB weights once per run. + if [[ "$EVICT_CACHE" == "1" ]]; then + if [[ "$DRY_RUN" == "1" ]]; then + DRY_RUN=1 scripts/hf-cache.sh keep "$MODEL" 2>&1 | tee -a "$LOG" + elif ! scripts/hf-cache.sh keep "$MODEL" 2>&1 | tee -a "$LOG"; then + log "sweep: WARNING -- cache eviction failed for $MODEL; continuing (disk may fill)" + fi + else + log "sweep: cache eviction off (EVICT_CACHE=0) -- swept models stay cached" fi out_base="output/$MODEL" -- 2.51.2