PLAN-mSD512.md #

Migration: /Volumes/MicroSD-512 -> /Volumes/photos library iCloud Version: 2.2 (POST-INVENTORY, EXECUTABLE) | Host: macOS Mac Studio, zsh Operator: human approves | Agent: executes all phases and reports before each phase transition (revised f18, see below). Supersedes v2.0 and v1.1. Phase 0 is COMPLETE; its measured results are baked in.

0. Mission #

Copy all keep-worthy data from the slow SD card (SRC) to the external APFS volume (DST) with content-hash verification, no duplication of data already at DST, zero data loss, then repoint the two Jellyfin libraries. Optimise for the measured 8.65 MiB/s read ceiling: EVERY BYTE OF SRC IS READ AT MOST ONCE.

0.5 Run order (quickstart) #

Scripts live in /Users/patricksingletary/dev/_shared/scripts/. All artifacts land in $AUDIT = ~/msd512-audit. Run from any cwd.

THE AGENT executes every script (revised f18). The operator is informed before each phase starts and approves before any phase transition. Operator may stop, override, or re-run any step at any time. The <=40-line digest per phase is the AGENT's report to the operator, not a paste-back.

0. syntax gate - MANDATORY before every execution (rule S4) #

zsh -n phase1-mapping.sh && chmod +x phase1-mapping.sh

1. mapping - fast, no SRC file reads. Re-run after each edit until #

the UNMAPPED section is empty or every entry is a deliberate SKIP. #

./phase1-mapping.sh $EDITOR ~/msd512-audit/mapping.tsv ./phase1-mapping.sh # repeat until clean [GATE-1]

2. dedup oracle - DST only, parallel, minutes not hours #

zsh -n phase2-dstindex.sh && ./phase2-dstindex.sh

3. copy + verify - the ONLY slow phase, ~5.8 h at 8.65 MiB/s [GATE-2] #

tmux survives terminal/SSH loss; caffeinate blocks sleep and disk idle. #

zsh -n phase3-copy.sh tmux new -s msd512 caffeinate -dimsu ./phase3-copy.sh

detach: Ctrl-b then d reattach: tmux attach -t msd512 #

watch: tail -f ~/msd512-audit/phase3.progress #

interrupted? just re-run the same command. It resumes from $LEDGER #

with zero SRC re-reads (rule R6). #

4. reconciliation - BLOCKING. Phase 5 is forbidden unless this is clean. #

zsh -n phase4-reconcile.sh && ./phase4-reconcile.sh

5. jellyfin repoint - quit Jellyfin.app FIRST [GATE-4] #

zsh -n phase5-jellyfin.sh && ./phase5-jellyfin.sh

then the manual UI steps in Phase 5, TV library first. #

Per-phase the AGENT reports only the ===== ... ===== digest (<=40 lines, rule R5) and checks with the operator before the next phase. Phase 5's manual UI scan steps are still operator actions (see Phase 5).

Phase durations, measured or estimated:

Phase	Reads SRC	Expected wall time
1 mapping	no	seconds
2 dst index	no	< 15 min (stop and report if longer)
3 copy+verify	YES	~5.8 h (176 GiB @ 8.65 MiB/s)
4 reconcile	no	minutes (1% spot re-hash on DST)
5 jellyfin	no	minutes + scan time

1. Measured facts (Phase 0, do not re-derive) #

SRC   /Volumes/MicroSD-512            ExFAT, SD slot, 477 GiB, 185 GiB used
DST   /Volumes/photos library iCloud  APFS,  USB,     1.8 TiB, 1.2 TiB free
BUNDLE $DST/photos-restore.photoslibrary   108,274 files / 201.4 GiB (READ-ONLY)

SRC payload:      1,833 files / 175.8 GiB / mtime 2015-09 .. 2026-06
SRC composition:  455 .mp4 (150.9 GB) + 100 .mkv (37.9 GB) = ~99.6% of bytes
                  872 .torrent (13.6 MB), 268 .srt, 50 .jpg (1.9 MB)
                  901 AppleDouble ._* files, 6 .DS_Store  -> EXCLUDED
SRC top dirs:     archive/z-New folder 333f/87.0GB  (2024-04..2026-05)
                  Jellyfin/Movie      447f/66.5GB  (2015-09..2025-12)
                  Jellyfin/TV         118f/35.3GB  (2025-01..2025-09)
                  archive/Twitch        8f/30MB
                  QBT/0-torrents      872f/13.6MB
Bundles on SRC:   NONE (b6 validated - the only bundle is $BUNDLE on DST)
Overlap:          size-only 1,396 files but 0.0 GiB; name+size 472 files 0.0 GiB
                  => NONE of the ~176 GiB of video exists at DST yet
Read speed:       8.65 MiB/s sequential  (small-file bench invalid, n=3)
Projection:       ~5.8 h for one full read pass
DST split:        in-bundle 108,274f / 201.4 GiB; loose 3,926f / 462.8 GiB
Jellyfin:         10.11.11, stopped, datadir 329 MB
                  root/default/Movies/Movie.mblink -> /Volumes/MicroSD-512/Jellyfin/Movie
                  root/default/TV/TV1.mblink       -> /Volumes/MicroSD-512/Jellyfin/TV
                  (video.mblink and collections.mblink are unrelated, do not touch)
Scan errors:      none (find/stat error logs empty)

2. Resolved decisions (do not re-ask the operator) #

GATE-0 imaging          DECLINED. Only 37% of the card is occupied and the
                        payload is large sequential video; a full-device image
                        would read 477 GiB (~15.7 h) vs 176 GiB (~5.8 h) of
                        real data. Read SRC directly, concurrency 1.
Q-NEW bundle-only dupes SKIP + log to in-bundle-only.tsv. Moot in practice:
                        only 50 tiny JPEGs could possibly be affected.
a4  already-copied data AVOID recopy via content hash, but the measured overlap
                        is ~0 GiB, so expect ~176 GiB of genuine new data.
c8  move vs copy        COPY. Never delete from SRC. Emit delete-candidates only.
c9  layout              MIRROR. Reorg is deferred to PLAN-REORG.
c10 hashing             b3sum > xxh128sum > shasum -a 256. Pick once, record it.
c11 collisions          Quarantine to $DST/_CONFLICTS/<relpath>.
b7  sidecars            .srt/.nfo/.jpg follow their media, exempt from dedup.
e16 watch state         NOT required. No DB surgery under any circumstances.
f18 execution           AGENT EXECUTES the scripts itself and reports a <=40-line
                        digest per phase. Agent checks with the operator BEFORE
                        starting each phase and before moving to the next phase.
                        Operator may override any proposed action.

3. Non-negotiable rules #

R1  SRC is READ-ONLY for this entire plan. No write, move, rename, delete, chmod,
    chown, or touch on SRC. Deletion is a separate future task, operator-only.
R2  $BUNDLE is READ-ONLY and is used ONLY as a dedup oracle. Never write into it,
    never open it with Photos.app during migration, never make it a copy target,
    never point a Jellyfin library at it or at the $DST root.
R3  Each SRC byte is read once. Hash in-stream while copying. Forbidden:
    rsync --checksum, any "hash pass then copy pass", any second verification
    read of SRC. Verification re-reads happen on the DST side only.
R4  Concurrency against SRC is exactly 1. Parallelism against DST/internal is fine.
R5  Bulk output goes to $AUDIT files. Chat receives digests of <=40 lines only.
    Never paste manifests, file lists, or hash tables into the conversation.
R6  Every phase is reuse-first and resumable. If a ledger entry exists for a file,
    that file is NEVER re-read from SRC. Rebuilding requires an explicit --force.
R7  I/O errors are logged and retried once at end-of-phase, never fatal mid-run.
R8  Jellyfin: NEVER edit *.db, *.db-wal, *.db-shm, or *.db.bak* with a text editor
    or with sqlite. 10.11 replaced library.db with jellyfin.db and a new schema;
    all pre-10.11 path-rewrite recipes are invalid. Path changes go through
    .mblink files or the web UI only.

4. Script hygiene (binding on every generated script) #

S1  Generated scripts are PURE ASCII. No em/en dashes, ellipses, curly quotes,
    or non-breaking spaces, including in comments and print strings.
S2  Every awk/sed/perl program is a SINGLE LINE. If too long, write it to
    $AUDIT/prog.awk and call awk -f.
S3  No apostrophes inside single-quoted strings.
S4  Operator runs `zsh -n <script>` before executing. A script that fails
    `zsh -n` is an agent defect, not an operator problem.
S5  emulate -L zsh; export LC_ALL=C; typeset -F SECONDS for timing.
S6  `wc -l` on macOS is space-padded: pipe through `tr -d ' '` before arithmetic.
S7  Compound braces need a trailing semicolon: { cmd; exit 1; }
S8  macOS `cat` has no -A. macOS stat is BSD: stat -f, not stat -c.
S9  `print -n` is unreliable in zsh; use `printf '%s'` when a trailing newline
    must be suppressed. This matters critically for .mblink files.

5. Environment #

SRC="/Volumes/MicroSD-512"
DST="/Volumes/photos library iCloud"          # contains spaces: always quote
BUNDLE="$DST/photos-restore.photoslibrary"
AUDIT="$HOME/msd512-audit"                    # fast internal disk
LEDGER="$AUDIT/ledger.tsv"                    # append-only, resumability source
DSTIDX="$AUDIT/dst.hashes.tsv"                # dedup oracle: hash<TAB>inbundle<TAB>path

Long-running jobs: see Section 0.5 for the tmux + caffeinate invocation. ExFAT notes: no POSIX ownership, no xattrs, 2-second mtime granularity. Do NOT use ditto/rsync -X for SRC files; plain byte copy plus touch -r is sufficient and correct. Allow +/-2 s when comparing mtimes across the two filesystems.

6. Destination layout (mirror, decided) #

$DST/photos-restore.photoslibrary   UNTOUCHED
$DST/Media/Jellyfin/Movie/          <- SRC/Jellyfin/Movie      (Jellyfin repoint)
$DST/Media/Jellyfin/TV/             <- SRC/Jellyfin/TV         (Jellyfin repoint)
$DST/Archive/z-New folder/          <- SRC/archive/z-New folder
$DST/Archive/Twitch/                <- SRC/archive/Twitch
$DST/QBT/0-torrents/                <- SRC/QBT/0-torrents
$DST/_CONFLICTS/<relpath>           same relpath, different content
$DST/_QUARANTINE/<relpath>          failed verification after one retry

Rationale for Media/Jellyfin/...: gives Jellyfin a stable, human-readable root that survives the future reorg, and keeps media out of the $DST root so no library ever accidentally scans $BUNDLE.

7. Ledger format ($LEDGER, append-only, tab-separated) #

status <TAB> hash <TAB> size <TAB> src_path <TAB> dst_path <TAB> note
status in: MIGRATED | DEDUP | DEDUP_BUNDLE | SKIPPED | CONFLICT | ERROR

Resume rule: a src_path present in $LEDGER with status != ERROR is never re-read.

---

PHASE 1 - Mapping proposal + selection list (no SRC file reads) #

Script: phase1-mapping.sh

#!/usr/bin/env zsh
emulate -L zsh
export LC_ALL=C
SRC="/Volumes/MicroSD-512"
DST="/Volumes/photos library iCloud"
AUDIT="$HOME/msd512-audit"
SM="$AUDIT/src.manifest.tsv"
[[ -s "$SM" ]] || { print "FATAL: missing $SM"; exit 1; }

# 1. seed mapping.tsv with the decided proposals (operator may edit)
if [[ ! -s "$AUDIT/mapping.tsv" ]]; then
  {
    printf 'src_subpath\tdst_subpath\taction\tnote\n'
    printf 'Jellyfin/Movie\tMedia/Jellyfin/Movie\tMIGRATE\tjellyfin movies\n'
    printf 'Jellyfin/TV\tMedia/Jellyfin/TV\tMIGRATE\tjellyfin tv\n'
    printf 'archive/z-New folder\tArchive/z-New folder\tMIGRATE\t87GB largest bucket\n'
    printf 'archive/Twitch\tArchive/Twitch\tMIGRATE\tsmall\n'
    printf 'QBT/0-torrents\tQBT/0-torrents\tMIGRATE\ttorrent files\n'
  } > "$AUDIT/mapping.tsv"
  print "[*] seeded $AUDIT/mapping.tsv - EDIT IT NOW, then re-run this script"
fi

# 2. report any SRC top-2 dir not covered by mapping.tsv
awk -F'\t' -v R="$SRC" 'NR==FNR{if(FNR>1)m[$1]=1;next} {rel=substr($3,length(R)+2); n=split(rel,a,"/"); d=(n>=3)?a[1]"/"a[2]:((n==2)?a[1]:"(root-files)"); if(!(d in m)){c[d]++; b[d]+=$2}} END{for(k in c) printf "UNMAPPED\t%d\t%.2f GB\t%s\n", c[k], b[k]/1e9, k}' "$AUDIT/mapping.tsv" "$SM" | sort -k2 -nr > "$AUDIT/unmapped.tsv"

# 3. build the work list: exclude junk, apply mapping, emit src<TAB>dst<TAB>size
awk -F'\t' -v R="$SRC" -v D="$DST" 'NR==FNR{if(FNR>1){k[$1]=$2; act[$1]=$3} next} {p=$3; rel=substr(p,length(R)+2); n=split(rel,q,"/"); f=q[n]; if(substr(f,1,2)=="._"){print "SKIPPED\t\t"$2"\t"p"\t\tappledouble" > "/dev/stderr"; next} if(f==".DS_Store"||tolower(f)=="thumbs.db"){print "SKIPPED\t\t"$2"\t"p"\t\tjunk" > "/dev/stderr"; next} d2=(n>=3)?q[1]"/"q[2]:((n==2)?q[1]:""); d1=q[1]; key=(d2 in k)?d2:((d1 in k)?d1:""); if(key==""){print "SKIPPED\t\t"$2"\t"p"\t\tunmapped" > "/dev/stderr"; next} if(act[key]!="MIGRATE"){print "SKIPPED\t\t"$2"\t"p"\t\taction="act[key] > "/dev/stderr"; next} sub("^"key"/","",rel); printf "%s\t%s/%s/%s\t%d\n", p, D, k[key], rel, $2}' "$AUDIT/mapping.tsv" "$SM" > "$AUDIT/worklist.tsv" 2>> "$AUDIT/ledger.tsv"

# 4. size-ordered variant is FORBIDDEN; keep SRC directory order for locality.
awk -F'\t' '{n++; b+=$3} END{printf "worklist: %d files, %.1f GiB\n", n, b/1073741824}' "$AUDIT/worklist.tsv" > "$AUDIT/worklist.summary.txt"

print "===== MAPPING ====="; cat "$AUDIT/mapping.tsv"
print "===== UNMAPPED (needs a decision) ====="; cat "$AUDIT/unmapped.tsv"
print "===== WORKLIST ====="; cat "$AUDIT/worklist.summary.txt"
print "===== SKIPPED BREAKDOWN ====="; awk -F'\t' '$1=="SKIPPED"{c[$6]++} END{for(k in c) printf "%d\t%s\n", c[k], k}' "$AUDIT/ledger.tsv" | sort -nr
print "===== SAMPLE MAPPINGS ====="; head -3 "$AUDIT/worklist.tsv" | cut -f1,2
print "===== END DIGEST ====="

GATE-1: operator reviews unmapped.tsv (expect the odd 4 KB mx] / _subs / _blind date (1987) [1080p] artifacts and any root-level files) and edits mapping.tsv. Agent re-runs until UNMAPPED is empty or every entry is a deliberate SKIP. Validation the agent must perform before GATE-1 passes: no dst_subpath is empty, starts with /, contains .., resolves inside $BUNDLE, or duplicates another mapping.

---

PHASE 2 - Targeted dedup oracle (DST only, parallel, cheap) #

Do NOT hash all 664 GiB of DST. Hash only DST files whose size collides with a SRC file; everything else is provably not a duplicate.

Script: phase2-dstindex.sh

#!/usr/bin/env zsh
emulate -L zsh
export LC_ALL=C
typeset -F SECONDS
AUDIT="$HOME/msd512-audit"
DST="/Volumes/photos library iCloud"
BUNDLE="$DST/photos-restore.photoslibrary"
[[ -s "$AUDIT/worklist.tsv" ]] || { print "FATAL: run phase1 first"; exit 1; }

if whence -p b3sum > /dev/null; then HASH=(b3sum); HN=blake3
elif whence -p xxh128sum > /dev/null; then HASH=(xxh128sum); HN=xxh128
else HASH=(shasum -a 256); HN=sha256; fi
print "$HN" > "$AUDIT/hash.algo"

awk -F'\t' '{print $3}' "$AUDIT/worklist.tsv" | sort -u > "$AUDIT/src.sizes.txt"
awk -F'\t' 'NR==FNR{z[$1]=1; next} ($2 in z){print $3}' "$AUDIT/src.sizes.txt" "$AUDIT/dst.manifest.tsv" > "$AUDIT/dst.candidates.txt"

t0=$SECONDS
tr '\n' '\0' < "$AUDIT/dst.candidates.txt" | xargs -0 -P $(sysctl -n hw.ncpu) -n 32 $HASH 2>> "$AUDIT/phase2.err" > "$AUDIT/dst.hashes.raw"
el=$(( SECONDS - t0 + 0.001 ))

# normalise "<hash>  <path>" into hash<TAB>inbundle<TAB>path
awk -v B="$BUNDLE" '{h=$1; $1=""; sub(/^[ \t]+/,""); p=$0; printf "%s\t%d\t%s\n", h, (index(p,B)==1)?1:0, p}' "$AUDIT/dst.hashes.raw" > "$AUDIT/dst.hashes.tsv"

print "===== PHASE2 ====="
printf 'algo=%s candidates=%s hashed=%s elapsed=%.1fs\n' "$HN" "$(wc -l < "$AUDIT/dst.candidates.txt" | tr -d ' ')" "$(wc -l < "$AUDIT/dst.hashes.tsv" | tr -d ' ')" $el
awk -F'\t' '{if($2==1)ib++; else lo++} END{printf "in-bundle=%d loose=%d\n", ib+0, lo+0}' "$AUDIT/dst.hashes.tsv"
head -4 "$AUDIT/phase2.err" 2>/dev/null
print "===== END DIGEST ====="

Expected from Phase 0 numbers: a small candidate set, all tiny files. If the candidate count exceeds ~20,000 or the elapsed time exceeds ~15 min, stop and report before Phase 3.

---

PHASE 3 - Single-pass copy + verify [GATE-2] #

The only slow phase. ~176 GiB at 8.65 MiB/s = ~5.8 h of SRC reading, plus DST read-back for verification (fast). Run under tmux + caffeinate per Section 0.5. Fully resumable.

Script: phase3-copy.sh

#!/usr/bin/env zsh
emulate -L zsh
setopt PIPE_FAIL
export LC_ALL=C
typeset -F SECONDS
SRC="/Volumes/MicroSD-512"
DST="/Volumes/photos library iCloud"
AUDIT="$HOME/msd512-audit"
LEDGER="$AUDIT/ledger.tsv"
HN=$(cat "$AUDIT/hash.algo")
case "$HN" in
  blake3) HASH=(b3sum) ;;
  xxh128) HASH=(xxh128sum) ;;
  *)      HASH=(shasum -a 256) ;;
esac
hash_of() { "${HASH[@]}" < "$1" | awk '{print $1}'; }

[[ -d "$SRC" ]] || { print "FATAL: SRC not mounted"; exit 1; }
[[ -d "$DST" ]] || { print "FATAL: DST not mounted"; exit 1; }
[[ -s "$AUDIT/worklist.tsv" ]] || { print "FATAL: run phase1 first"; exit 1; }
mkdir -p "$AUDIT/tmp"
touch "$LEDGER"

# already-done set (resume) and dedup oracle in memory
typeset -A DONE HASHIDX INBUNDLE
while IFS=$'\t' read -r st h sz sp dp nt; do
  [[ "$st" == "ERROR" ]] && continue
  [[ -n "$sp" ]] && DONE[$sp]=1
  [[ -n "$h" && -n "$dp" ]] && HASHIDX[$h]="$dp"
done < "$LEDGER"
while IFS=$'\t' read -r h ib p; do
  [[ -n "$h" ]] && : ${HASHIDX[$h]::="$p"}
  [[ "$ib" == "1" ]] && INBUNDLE[$h]=1
done < "$AUDIT/dst.hashes.tsv"

t0=$SECONDS; nmig=0; ndup=0; nerr=0; ncon=0; bytes=0
total=$(wc -l < "$AUDIT/worklist.tsv" | tr -d ' ')
idx=0

while IFS=$'\t' read -r s t sz; do
  idx=$(( idx + 1 ))
  [[ -n "${DONE[$s]}" ]] && continue
  [[ -d "$SRC" ]] || { print "ABORT: SRC vanished at item $idx. Re-run to resume."; break; }
  [[ -r "$s" ]] || { printf 'ERROR\t\t%d\t%s\t\tunreadable\n' "$sz" "$s" >> "$LEDGER"; nerr=$(( nerr + 1 )); continue; }

  mkdir -p "${t:h}" 2>/dev/null
  part="$t.part"
  # SINGLE READ of SRC: tee writes DST, hash consumes the same stream
  h_src=$( cat -- "$s" | tee -- "$part" | "${HASH[@]}" | awk '{print $1}' )
  rc=$?
  if [[ $rc -ne 0 || -z "$h_src" ]]; then
    rm -f -- "$part"
    printf 'ERROR\t\t%d\t%s\t\tstream rc=%d\n' "$sz" "$s" "$rc" >> "$LEDGER"
    nerr=$(( nerr + 1 )); continue
  fi

  wsz=$(stat -f %z -- "$part" 2>/dev/null)
  h_dst=$(hash_of "$part")
  if [[ "$h_src" != "$h_dst" || "$wsz" != "$sz" ]]; then
    mkdir -p "$DST/_QUARANTINE/${${t#$DST/}:h}" 2>/dev/null
    mv -f -- "$part" "$DST/_QUARANTINE/${t#$DST/}" 2>/dev/null || rm -f -- "$part"
    printf 'ERROR\t%s\t%d\t%s\t\tverify-mismatch\n' "$h_src" "$sz" "$s" >> "$LEDGER"
    nerr=$(( nerr + 1 )); continue
  fi

  # zero-byte files never dedup by hash alone
  if [[ -n "${HASHIDX[$h_src]}" && "$sz" != "0" ]]; then
    ex="${HASHIDX[$h_src]}"
    rm -f -- "$part"
    if [[ -n "${INBUNDLE[$h_src]}" ]]; then
      printf 'DEDUP_BUNDLE\t%s\t%d\t%s\t%s\tbundle-only\n' "$h_src" "$sz" "$s" "$ex" >> "$LEDGER"
      printf '%s\t%s\n' "$s" "$ex" >> "$AUDIT/in-bundle-only.tsv"
    else
      printf 'DEDUP\t%s\t%d\t%s\t%s\talready-present\n' "$h_src" "$sz" "$s" "$ex" >> "$LEDGER"
    fi
    ndup=$(( ndup + 1 )); continue
  fi

  if [[ -e "$t" ]]; then
    if [[ "$(hash_of "$t")" == "$h_src" ]]; then
      rm -f -- "$part"
      printf 'DEDUP\t%s\t%d\t%s\t%s\tidentical-at-target\n' "$h_src" "$sz" "$s" "$t" >> "$LEDGER"
      ndup=$(( ndup + 1 )); continue
    fi
    c="$DST/_CONFLICTS/${t#$DST/}"
    mkdir -p "${c:h}" 2>/dev/null
    mv -f -- "$part" "$c"
    touch -r "$s" -- "$c" 2>/dev/null
    printf 'CONFLICT\t%s\t%d\t%s\t%s\tdifferent-content-same-path\n' "$h_src" "$sz" "$s" "$c" >> "$LEDGER"
    ncon=$(( ncon + 1 )); continue
  fi

  mv -f -- "$part" "$t"
  touch -r "$s" -- "$t" 2>/dev/null
  HASHIDX[$h_src]="$t"
  printf 'MIGRATED\t%s\t%d\t%s\t%s\t\n' "$h_src" "$sz" "$s" "$t" >> "$LEDGER"
  nmig=$(( nmig + 1 )); bytes=$(( bytes + sz ))

  if (( idx % 25 == 0 )); then
    el=$(( SECONDS - t0 + 0.001 ))
    printf '[%d/%d] mig=%d dup=%d err=%d %.1f GiB @ %.2f MiB/s\n' $idx $total $nmig $ndup $nerr $(( bytes/1073741824.0 )) $(( bytes/1048576.0/el )) >> "$AUDIT/phase3.progress"
  fi
done < "$AUDIT/worklist.tsv"

el=$(( SECONDS - t0 + 0.001 ))
print "===== PHASE3 ====="
printf 'processed=%d/%d migrated=%d dedup=%d conflict=%d error=%d\n' $idx $total $nmig $ndup $ncon $nerr
printf 'copied %.1f GiB in %.2f h => %.2f MiB/s\n' $(( bytes/1073741824.0 )) $(( el/3600 )) $(( bytes/1048576.0/el ))
awk -F'\t' '$1=="ERROR"{c[$6]++} END{for(k in c) printf "ERR %d\t%s\n", c[k], k}' "$LEDGER"
tail -3 "$AUDIT/phase3.progress" 2>/dev/null
print "===== END DIGEST ====="

GATE-2: operator approves before first run. GATE-3: agent stops and reports if error rate exceeds 1% of processed files, or if sustained throughput drops below ~4 MiB/s (half the Phase 0 baseline), which is the signature of a failing card. The ledger is append-only, so re-running after any interruption resumes exactly where it stopped, with zero SRC re-reads.

---

PHASE 4 - Reconciliation (must pass before Phase 5) #

Script: phase4-reconcile.sh

#!/usr/bin/env zsh
emulate -L zsh
export LC_ALL=C
AUDIT="$HOME/msd512-audit"
LEDGER="$AUDIT/ledger.tsv"
SM="$AUDIT/src.manifest.tsv"

src_total=$(wc -l < "$SM" | tr -d ' ')
awk -F'\t' '{c[$1]++} END{for(k in c) printf "%s\t%d\n", k, c[k]}' "$LEDGER" | sort > "$AUDIT/status.counts.tsv"
acct=$(awk -F'\t' '{print $4}' "$LEDGER" | sort -u | wc -l | tr -d ' ')

cut -f4 "$LEDGER" | sort -u > "$AUDIT/accounted.txt"
cut -f3 "$SM" | sort -u > "$AUDIT/all.src.txt"
comm -23 "$AUDIT/all.src.txt" "$AUDIT/accounted.txt" > "$AUDIT/unaccounted.tsv"

# integrity spot check: re-hash 1% of migrated files at DST
HN=$(cat "$AUDIT/hash.algo")
case "$HN" in blake3) HASH=(b3sum);; xxh128) HASH=(xxh128sum);; *) HASH=(shasum -a 256);; esac
awk -F'\t' '$1=="MIGRATED"{print $2"\t"$5}' "$LEDGER" | awk 'NR%100==1' > "$AUDIT/spot.list"
bad=0; n=0
while IFS=$'\t' read -r h p; do
  n=$(( n + 1 ))
  a=$("${HASH[@]}" < "$p" 2>/dev/null | awk '{print $1}')
  [[ "$a" == "$h" ]] || { bad=$(( bad + 1 )); print "SPOTFAIL\t$p" >> "$AUDIT/spot.fail.tsv"; }
done < "$AUDIT/spot.list"

# delete candidates: SRC files with a verified DST twin (REVIEW ONLY, never executed)
awk -F'\t' '$1=="MIGRATED"||$1=="DEDUP"{printf "%s\t%s\t%s\n", $4, $5, $2}' "$LEDGER" > "$AUDIT/delete-candidates.tsv"

print "===== PHASE4 ====="
printf 'src files=%d accounted=%d unaccounted=%d\n' $src_total $acct "$(wc -l < "$AUDIT/unaccounted.tsv" | tr -d ' ')"
cat "$AUDIT/status.counts.tsv"
printf 'spot-check: %d re-hashed, %d mismatches\n' $n $bad
awk -F'\t' '$1=="MIGRATED"{b+=$3} END{printf "migrated bytes: %.1f GiB\n", b/1073741824}' "$LEDGER"
printf 'delete-candidates: %s files (REVIEW ONLY)\n' "$(wc -l < "$AUDIT/delete-candidates.tsv" | tr -d ' ')"
head -5 "$AUDIT/unaccounted.tsv"
print "===== END DIGEST ====="

BLOCKING: Phase 5 does not start unless unaccounted == 0, spot mismatches == 0, and every ERROR row has been retried once and resolved or explicitly accepted. The agent NEVER deletes anything on SRC. delete-candidates.tsv is output only.

---

PHASE 5 - Jellyfin repair (10.11.11, macOS app) [GATE-4] #

Constraints specific to this install:

• Version 10.11.11. In 10.11 the library data moved out of library.db into a new jellyfin.db with a rewritten schema (a library.db.old backup is left behind by the migration). All pre-10.11 SQL path-rewrite recipes (UPDATE TypedBaseItems ...) are INVALID here. DO NOT run them.
• Never edit *.db, *.db-wal, *.db-shm, *.db.bak* with a text editor or with sqlite.
• Watch state is not required (decision e16), so a full rescan is fine.
• Datadir: $HOME/Library/Application Support/Jellyfin (the two DATADIR lines in the Phase 0 digest are one folder seen twice on a case-insensitive FS).
• Only two links change: root/default/Movies/Movie.mblink and root/default/TV/TV1.mblink. video.mblink and collections.mblink are unrelated - DO NOT TOUCH.
• A .mblink file must contain the path with NO trailing newline, or the library scan fails with a path-parsing error. Use printf '%s', never echo.

Script: phase5-jellyfin.sh

#!/usr/bin/env zsh
emulate -L zsh
export LC_ALL=C
AUDIT="$HOME/msd512-audit"
JF="$HOME/Library/Application Support/Jellyfin"
DST="/Volumes/photos library iCloud"

pgrep -f -i jellyfin > /dev/null && { print "FATAL: Jellyfin is running. Quit it first."; exit 1; }
[[ -d "$JF" ]] || { print "FATAL: datadir not found: $JF"; exit 1; }
[[ -d "$DST/Media/Jellyfin/Movie" && -d "$DST/Media/Jellyfin/TV" ]] || { print "FATAL: migrated media dirs missing. Phase 3/4 incomplete."; exit 1; }

BK="$AUDIT/jellyfin-backup-$(date +%Y%m%d-%H%M%S)"
ditto "$JF" "$BK" || { print "FATAL: backup failed"; exit 1; }
print "backup: $BK ($(du -sh "$BK" | cut -f1))"

M="$JF/root/default/Movies/Movie.mblink"
T="$JF/root/default/TV/TV1.mblink"
for f in "$M" "$T"; do [[ -f "$f" ]] || { print "FATAL: missing $f"; exit 1; }; done
print "before: $(cat "$M") | $(cat "$T")"

# CRITICAL: no trailing newline, or the library scan fails with a parse error
printf '%s' "$DST/Media/Jellyfin/Movie" > "$M"
printf '%s' "$DST/Media/Jellyfin/TV"    > "$T"

print "after:  $(cat "$M") | $(cat "$T")"
print "byte check (must be 0 newlines):"
for f in "$M" "$T"; do printf '%s -> trailing_newlines=%s\n' "${f:t}" "$(tail -c 1 "$f" | od -An -c | tr -d ' ' | grep -c '\\n')"; done

grep -rl -i 'MicroSD' "$JF" --include='*.xml' 2>/dev/null | head -20 > "$AUDIT/jf.remaining-refs.txt"
find "$JF/data/playlists" -type f 2>/dev/null | head -5 >> "$AUDIT/jf.remaining-refs.txt"
print "===== REMAINING MicroSD REFS (xml only, fix by hand) ====="; cat "$AUDIT/jf.remaining-refs.txt"
print "===== END DIGEST ====="

Then, manually:

1. Start Jellyfin. Dashboard -> Libraries: confirm Movies and TV now show the new paths. If a path still shows the old one, fix it in the UI instead (remove old folder, add new folder) - the UI is always safe, the .mblink edit is just faster.
2. Scan ONE library first (TV, 118 files) with "Replace all metadata" OFF. Verify the item count matches Phase 4 migrated counts for that tree.
3. Only after TV verifies, scan Movies.
4. NEVER point any library at $DST root or at photos-restore.photoslibrary. Scanning that bundle would enumerate 108k files and produce junk items.
5. If items are missing after the scan: Scheduled Tasks -> "Scan all libraries", then the clean-database / remove-missing-items task.
6. Rollback if anything breaks: quit Jellyfin, rm -rf "$JF", then ditto "$BK" "$JF".

---

PHASE 6 - Decommission [GATE-5, not now] #

Only after Phase 4 passes clean, Jellyfin verified working on both libraries, and the operator confirms an independent backup exists. The agent PRESENTS the verify-then-erase procedure for the SD card; it never executes it.

---

8. Gates #

Gate	Blocks	Requires
GATE-0	-	RESOLVED: imaging declined
GATE-1	Phase 3	unmapped.tsv empty or all deliberate; mapping.tsv validated
GATE-2	Phase 3 start	operator approves the ~5.8 h run
GATE-3	Phase 3 resume	approval after error spike or throughput collapse
GATE-4	Phase 5	Phase 4 clean + Jellyfin backup confirmed
GATE-5	Phase 6	explicit "erase approved"

9. Failure handling #

SRC I/O error        -> ERROR row in ledger, continue, retry once at phase end
SRC unmounts mid-run -> detected via [[ -d $SRC ]], clean stop, re-run to resume
KERNEL PANIC         -> OBSERVED 2026-08-16 12:41. See 9.1 below. Resume-safe.
DST full             -> stop before writing; report bytes still required
verify mismatch      -> quarantine, retry once, then ERROR
Phase 4 assertion    -> HARD STOP. No Phase 5, no Phase 6.
$BUNDLE integrity    -> if ever in doubt, stop and report. Never attempt repair.

9.1 Kernel panic on the internal SD reader (OBSERVED) #

On 2026-08-16 at 12:41:50 the host kernel-panicked mid-Phase-3 with:

panic(cpu 2): "apcie[1:pcie-sdreader]::handleCompletionTimeoutInterrupt:
completion timeout ... ltssm 0x11=L0" @ AppleT810xPCIePort.cpp:1723

Cause: a PCIe completion-timeout on the internal SD card reader controller (Mac Studio 2022 / M1 Max / T6000) under sustained sequential read. The card itself was still readable afterwards (spot-read 10 MB @ ~7 MB/s). Phase 3's single-threaded multi-hour read of SRC is the triggering load; this is a hardware-level controller failure the plan did not originally anticipate.

Impact: NONE on integrity. Zero files had been fully migrated (ledger showed only SKIPPED rows); the only artifact was a stale <dst>.part (first file, ~80% copied), which Phase 3 truncates and overwrites on resume. Append-only ledger + resume rule R6 made recovery lossless and free of SRC re-reads.

Resume action: just re-run the Phase 3 command from Section 0.5. It resumes exactly where it stopped. The machine was rebooted after the panic.

If the panic RECURS on the internal reader:

• RESOLVED 2026-08-16: operator moved the card to an external USB3.0 card reader. Mount path is UNCHANGED (/Volumes/MicroSD-512, now disk4s1 via USB/external instead of disk8s1 via internal SD). No script edits required; Phase 3 re-runs verbatim against the same SRC path. Spot-read ~8.9 MB/s.
• Strongest mitigation: use an external USB SD card reader instead of the internal slot. Repoint SRC to the new mount (likely still ExFAT; re-verify the mount path in the scripts / Section 5) and re-run Phase 3.
• Cheap alternative: keep re-running; the ledger makes each panic cost at most the partially-copied current file. Acceptable only if panics stay rare.
• Do NOT modify phase3-copy.sh to throttle or add a link health-check without operator approval; that is a plan change to be agreed first.

10. Token discipline #

• One shell command that emits a summary beats reading files into context.
• Aggregate with awk/sort/head/wc; never cat a manifest into chat.
• One digest per phase, <=40 lines. Never restate this plan; cite phase numbers.
• Agent executes scripts but reports a <=40-line digest and checks with the operator before each phase transition. At most 5 questions at a time, only when genuinely blocking.
• All analysis persists to $AUDIT/*.md, not into the conversation.

11. Deferred (requires a NEW plan, do not start) #

• PLAN-REORG: date-based reorganisation (YYYY/YYYY-MM) of the migrated tree, EXIF DateTimeOriginal with mtime fallback, running entirely on DST. Must run AFTER Jellyfin is verified, and must update the .mblink paths again if it touches Media/Jellyfin/**.
• SD erase / repurpose.
• Optional import of migrated photos into Photos.app.
• The 901 AppleDouble ._* files and the 4 KB odd-extension artifacts (mx], _subs, _blind date (1987) [1080p]) are currently SKIPPED; if any turn out to be real data, they are a separate one-off task.

12. Changelog #

v2.3  Revised decision f18: the AGENT now executes the scripts itself (was
      emit-only) and checks with the operator before each phase transition.
      Operator retains override/stop/re-run authority. Section 0.5, Section 10,
      and GATE descriptions updated. Phase 5 manual UI scan steps remain
      operator actions.
v2.2  Section 9 + new 9.1: documented the observed kernel panic on the internal
      SD reader (AppleT810xPCIePort completion timeout, 2026-08-16) that hit
      Phase 3 on file #1. Confirmed it caused zero data loss and that resume is
      lossless per R6. Added recurrence mitigation guidance (external USB reader).
v2.1  Added Section 0.5 run order (was chat-only prose, now in-file).
      Section 5 long-job note replaced by a cross-reference to 0.5.
      Phase 5 header states the .mblink no-trailing-newline requirement.
      Fixed phase1 worklist byte sum ($2 -> $3, the size column).
      Merged the two dst.hashes.tsv read loops in phase3 into one.
      Added guard clauses: phase2 and phase3 abort if worklist.tsv is missing.
      Added ===== END DIGEST ===== terminator to phase5.
      Restated DST in-bundle/loose split in Section 1.
v2.0  Post-inventory rewrite. Imaging declined, Phase 2 narrowed to
      size-colliding DST files, R8 forbids all Jellyfin DB surgery (10.11
      moved library data to jellyfin.db), layout fixed to Media/Jellyfin/**.
v1.1  Script hygiene rules S1-S9 added after a zsh parse failure.
v1.0  Initial plan, Phases 0-6.