diff --git a/.DS_Store b/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..5e0fe81b05ae27a61fbf257b2dc61d92846bb2c8 GIT binary patch literal 6148 zcmZQzU|@7AO)+F(5MW?n;9!8zEL;p&0Z1N%F(jFwBCH_uKxQPB7Z)VuvOFl>j) zjZ&i_Fd71bHv~X=mxUpPA)g_cp%^*;CKcpl7MBUW{(Zd$S)5r zNh~QXc1kRY2Ju4j^K+75?8Kz7%+&ID0TJi?ypqJsywoC)lHkmg)TG3snDETJl>Bn1 z{L;LXVz6GQ1ScmaXS{%Tb+x&HrH+D`VXclrwWW!fj)H}$VQnoZhp4i?bx?eEPHtX) zCnO{p86h+SFO-H+T?`BiaPOoPC+8&P=jVVV;et8EWzaH;X)nx=x1oOI;N;@w;pJ2C z@bvKV=Jkyi5Gu<|E%z@d$;{6y4ofX6&dkq?7vL|>$S?Oy&d&=dN(IS>C+Fvs=H?a0 z3kW7B<|LQqB$lK)=HvutR;3n4C02x_R+NC)AdSgI`8hcO`Nf$a6C)!^a7jf(73UX~ zID^fN7Z4~dP7O-UNi4}MOLa*sNiB}ZOwP{(nJEJD8WLvYyvZ*hC?qT*Dkd(WsHSaT zYH8;bk({5Ko0?Zr9Ga7ul$sM>2JuLAN@7W>b5UwyNoq<+ab{I&3`oE?KP59QGc_e7 zJ2NjOBrh>HH4Y@~lUQ8hUyz!YnsP-_N?Jx%j#nWxGcP5zBD6d+r6eOVu{b$3FC{ZC zJ-jTlI5R0HRe)WPQHPTQtXfH#S3!VPAO*@**WgtUU=#3$a&>eS6a<(AwsJre8yYF_ z3NQ;Ka&Ut6n&~R=3NS)BR@S_{0xSZ>oSau=q@*Qz<#-)AIRzNN3Y}dP6a+W~PH=E= zC_89D!bBNDF)%0}$T$FIgMGxQ>|g-qf$Bba1``H120w-*hFpevh7N{V3`-eyFdSsK z#BiD68N+*quMB?~IT`sFr5P0%RT)hfZ5f>zJsE=;qZwlv6BrX2lNhrZa~bm)3m6L- zs~Bq->lo`98yGtoyBT{K`xyHfXE4rUoWnSmaUSDx#+8h#7*{iHVcf>JgK;P0F2=)* zM;MPX9%sD3c!}`}<5k9MjMo_-Fur8`#Q2@@2gHR;u$1lx;loI%*=K14eMBSbscj!|MX1V%#u z5dzE*rU0n^cV%F})&Ga68YM?VU^E1VWe6~`xCFa6fhtWL-UHROp!zfcDi5mtLDexM vsGf%C11SN^GJ!f(3=9%r4nz*56;$nmt71k5NKHQ45P*fyC_Nei^bY|5%z+gH literal 0 HcmV?d00001 diff --git a/docs/.DS_Store b/docs/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..fb4a899fd608d834e21a124ed3c54aff81a81d65 GIT binary patch literal 8196 zcmZQzU|@7AO)+F(kYHe7;9!8z0^AH(0Z1N%F(jFwB5WY@7#IW?7)lu88FCo%8PXZj zp>m_tXb6mkz-S1JhQMeDjE2DA3IRrlb2xC+`w-btax?@+LtsRP0H}OWfVAxy9H4Xq zga%15FfuTJy8w&~3@oq!Vg&aC7(j9$tsokt6-0xyGBAQzU^Bp485p5j8NuBUkUjyh zQH%^=?F@`yn?d4W?F@`yn;94wA=()jp*Ax@dnk+$?F@_%?F@`y+rdVT5~Cq78Ukn` z0P4H*G9)sjGo&))q4eJ~8S)s?Q;L&wlJfI&K)H^Up@1QWA(0`Ep*X3$xF9JfKZ${X zVS7?RPG)h5fx$IKCT12^Hg9_ zj~5Ve&d)1J%*;zI0x1d3Oi4{jEQ$%w%uC5Hcgio#ODP8Hg-UR8a&pECh*wve8(8Wn zm>JgUC{$aTnCU23m>SmBa&m|&>strKXXoVR<#$5-2`erbcp-c^*~P%XfaD<@{>&*Z zbB6enISJ;^+faXUaB^|;@bbw-L`FnK$MD7r2$f}~mirf!Waj4;hou%3XXfX{3-A|b z<+t5S=j5-UPdD@s6YkjCVq z{G6PC{NhZIiIEW{xTGSYit~#~oWbTsWG3h5K@1QmElv$e%}FfDEK7AsEJ-aEfqNdp zVB}oKFQK5Ssb_3q>*(g?7j#8HP)Jxrl$SRuGbJ^zBr`2DwJ5YaGo>UWGO;*0H7_MI zFFm|0vp6#;Cp9E5F*j9^QHPWBilmgZfQ+n+JWL8*3x|L*M?`5-W?^Y&N#zw`K_LNN zB^eb?fvp@dDTyVizWFI2n^M3sGDmGg%qYev&M3jC$f(Jv&uGbL#c0E5%jm}F<cM#2Cz& zz?jLH#hA@l&e*`%$k@!-!q~|;iE%RHRK{tHvl*8#E@fQKxPoyV<95a!jJp_jGahC< z!+4hQ9OHGy2aFFHA2U8-{LJ`=@h{^)CQc?HCSfKKCN(B~CIcozCTk`arf{Z6rYNRp zrWmGFraY!{rW&SJh!>bZWeJEG2Or0$$3RaN?MXvj@OZsQ-A?%iL;A>f&eJ9Q8C#+V|dT- zmEkWVCnF!DG@}BeDx(RbEu%A|Cu1;UG-E7d0%Ia$5@R-FE@M7p0b?O!6=Mx!9b-LX z17jy+H)AhjA7ek`48~cEa~S6`&SPB8xRP-d<7&n&jN2G@Fz#gB#dw(U2;))4wpw4Jx=m-El@xzZ{|u xqy*Hp2Y1IAA(M(=)r<^~o;)K1q$khFzyQ)Za03B@r0qRg|Bu%H;0!WA1OPjIt(pJ; literal 0 HcmV?d00001 diff --git a/docs/BITWARDEN-KEYCHAIN-MACOS.md b/docs/BITWARDEN-KEYCHAIN-MACOS.md new file mode 100644 index 0000000..1901d5c --- /dev/null +++ b/docs/BITWARDEN-KEYCHAIN-MACOS.md @@ -0,0 +1,112 @@ +# Bitwarden CLI + macOS Keychain — Secret Infrastructure on macOS + +How machines in the fleet store and use Bitwarden credentials. Original setup: +ole Blu (MacBook Air, `ole-blu.local`). Replicated on Mac Studio 2022 (`mac.lan`). + +## Architecture + +| What | Where | Notes | +|------|-------|-------| +| Bitwarden API key (client id/secret) | `~/.config/bw/env`, mode 0600 | Two lines, both `export`-ed. Non-master secret; file-safe. | +| Bitwarden master password | macOS Keychain | Service `bw-master`, account ``. NEVER in a file. | +| Service credentials (API keys, app passwords) | Bitwarden vault items | Login type (type:1); fetched per-use at runtime. | + +Session keys (`BW_SESSION`) are transient — never written to disk. + +## Why this split + +- The API key only identifies/authenticates the CLI to Bitwarden's API; exposure + doesn't unlock the vault. It's low-sensitivity, so a 0600 file is acceptable and + enables non-interactive login. +- The master password unlocks everything. Keychain storage gives ACL-protected, + prompt-on-first-access, encrypted-at-rest handling with zero file copies. +- Scripts fetch vault secrets per-use and unset them — no `export SECRET=` in + shell profiles (readable via `ps eww`, lands in every child env). + +## Setup on a new macOS machine + +```zsh +# 1. Install CLI +brew install bitwarden-cli + +# 2. Copy env file from an existing machine (secret never enters chat/session logs) +mkdir -p ~/.config/bw +scp existing-machine:.config/bw/env ~/.config/bw/env +chmod 600 ~/.config/bw/env + +# 3. Log the CLI in (prints "already logged in" if done — that's fine) +source ~/.config/bw/env +bw login --apikey + +# 4. Store the master password in Keychain (interactive; user types it once) +security add-generic-password -s bw-master -a "$USER" -w +# ^ prompts: password, then confirm. Never echoed. + +# 5. Verify (do not print the password itself) +security find-generic-password -s bw-master -w | wc -c # non-zero +``` + +To get an API key if starting from scratch: https://vault.bitwarden.com → +Settings → Security → Keys → View API key. + +## Unlocking in scripts + +Paste this block at the top of any script that needs vault access: + +```zsh +[ -f "$HOME/.config/bw/env" ] && source "$HOME/.config/bw/env" +if [ -z "${BW_PASSWORD:-}" ]; then + BW_PASSWORD=$(security find-generic-password -s bw-master -w 2>/dev/null) || exit 1 +fi +export BW_PASSWORD +bw login --apikey 2>/dev/null || true +BW_SESSION=$(bw unlock --passwordenv BW_PASSWORD --raw 2>/dev/null) + +# ... do work, fetching secrets per-use (below) ... + +unset BW_SESSION BW_PASSWORD # clear when done +``` + +## Fetching secrets + +Fetch the item ONCE and extract both fields (separate `bw get` calls can return +mismatched values if the item changed between calls): + +```zsh +ITEM=$(bw get item "Porkbun API" --session "$BW_SESSION") +API_KEY=$(echo "$ITEM" | jq -r '.login.username') +API_SECRET=$(echo "$ITEM" | jq -r '.login.password') +# Validate — empty/null means wrong name or missing field +[ -z "$API_KEY" ] || [ "$API_KEY" = "null" ] && { echo "missing fields" >&2; exit 1; } +``` + +Password-only: `bw get password "Item Name" --session "$BW_SESSION"`. + +Item naming is a contract — scripts match exact names. Current conventions: + +| Bitwarden item | Username field | Password field | +|----------------|----------------|----------------| +| Porkbun API | pk1_... API key | sk1_... secret | +| Bluesky App Password | handle | app password | + +## Pitfalls + +- `source` sets shell variables, NOT environment variables. Always `export` BW_* + vars, or `bw unlock --passwordenv BW_PASSWORD` silently fails (empty `--raw`). +- `bw login --apikey --raw` returns nothing when already logged in. Use non-raw + `bw login --apikey` and gate on `bw unlock` instead. +- First Keychain read from a new process pops a macOS approval dialog ("security + wants to use your confidential information") — one-time per item per session. +- `bw get` matches the FIRST duplicate item name. After creating: check + `bw list items --search "Name" --session "$BW_SESSION" | jq 'length'` → 1. +- CLI caches vault data locally. After editing the vault elsewhere, `bw sync`. +- Never paste live secret values into AI agent sessions or shell history — they + persist. Reference the item name instead. Rotate + purge anything that leaked. +- First Keychain read from a new process may pop a macOS dialog — expected. + +## Fleet status + +| Machine | Status | +|---------|--------| +| ole Blu (MacBook Air, ole-blu.local) | Original — full setup | +| Mac Studio 2022 (mac.lan) | Complete 2026-09-11: env via scp, login verified, Keychain item added, unlock + sync verified (664 items) | diff --git a/docs/PLAN-mSD512.md b/docs/PLAN-mSD512.md new file mode 100644 index 0000000..81edbda --- /dev/null +++ b/docs/PLAN-mSD512.md @@ -0,0 +1,663 @@ +# PLAN-mSD512.md + +Migration: `/Volumes/MicroSD-512` -> `/Volumes/photos library iCloud` +Version: 2.2 (POST-INVENTORY, EXECUTABLE) | Host: macOS Mac Studio, zsh +Operator: human approves | Agent: executes all phases and reports before each +phase transition (revised f18, see below). +Supersedes v2.0 and v1.1. Phase 0 is COMPLETE; its measured results are baked in. + +## 0. Mission + +Copy all keep-worthy data from the slow SD card (SRC) to the external APFS +volume (DST) with content-hash verification, no duplication of data already at +DST, zero data loss, then repoint the two Jellyfin libraries. Optimise for the +measured 8.65 MiB/s read ceiling: EVERY BYTE OF SRC IS READ AT MOST ONCE. + +## 0.5 Run order (quickstart) + +Scripts live in /Users/patricksingletary/dev/_shared/scripts/. +All artifacts land in $AUDIT = ~/msd512-audit. Run from any cwd. + +THE AGENT executes every script (revised f18). The operator is informed before +each phase starts and approves before any phase transition. Operator may stop, +override, or re-run any step at any time. The <=40-line digest per phase is the +AGENT's report to the operator, not a paste-back. + +# 0. syntax gate - MANDATORY before every execution (rule S4) +zsh -n phase1-mapping.sh && chmod +x phase1-mapping.sh + +# 1. mapping - fast, no SRC file reads. Re-run after each edit until +# the UNMAPPED section is empty or every entry is a deliberate SKIP. +./phase1-mapping.sh +$EDITOR ~/msd512-audit/mapping.tsv +./phase1-mapping.sh # repeat until clean [GATE-1] + +# 2. dedup oracle - DST only, parallel, minutes not hours +zsh -n phase2-dstindex.sh && ./phase2-dstindex.sh + +# 3. copy + verify - the ONLY slow phase, ~5.8 h at 8.65 MiB/s [GATE-2] +# tmux survives terminal/SSH loss; caffeinate blocks sleep and disk idle. +zsh -n phase3-copy.sh +tmux new -s msd512 +caffeinate -dimsu ./phase3-copy.sh +# detach: Ctrl-b then d reattach: tmux attach -t msd512 +# watch: tail -f ~/msd512-audit/phase3.progress +# interrupted? just re-run the same command. It resumes from $LEDGER +# with zero SRC re-reads (rule R6). + +# 4. reconciliation - BLOCKING. Phase 5 is forbidden unless this is clean. +zsh -n phase4-reconcile.sh && ./phase4-reconcile.sh + +# 5. jellyfin repoint - quit Jellyfin.app FIRST [GATE-4] +zsh -n phase5-jellyfin.sh && ./phase5-jellyfin.sh +# then the manual UI steps in Phase 5, TV library first. + +Per-phase the AGENT reports only the `===== ... =====` digest +(<=40 lines, rule R5) and checks with the operator before the next phase. +Phase 5's manual UI scan steps are still operator actions (see Phase 5). + +Phase durations, measured or estimated: + +| Phase | Reads SRC | Expected wall time | +|----------------|-----------|---------------------------------------| +| 1 mapping | no | seconds | +| 2 dst index | no | < 15 min (stop and report if longer) | +| 3 copy+verify | YES | ~5.8 h (176 GiB @ 8.65 MiB/s) | +| 4 reconcile | no | minutes (1% spot re-hash on DST) | +| 5 jellyfin | no | minutes + scan time | + +## 1. Measured facts (Phase 0, do not re-derive) +``` +SRC /Volumes/MicroSD-512 ExFAT, SD slot, 477 GiB, 185 GiB used +DST /Volumes/photos library iCloud APFS, USB, 1.8 TiB, 1.2 TiB free +BUNDLE $DST/photos-restore.photoslibrary 108,274 files / 201.4 GiB (READ-ONLY) + +SRC payload: 1,833 files / 175.8 GiB / mtime 2015-09 .. 2026-06 +SRC composition: 455 .mp4 (150.9 GB) + 100 .mkv (37.9 GB) = ~99.6% of bytes + 872 .torrent (13.6 MB), 268 .srt, 50 .jpg (1.9 MB) + 901 AppleDouble ._* files, 6 .DS_Store -> EXCLUDED +SRC top dirs: archive/z-New folder 333f/87.0GB (2024-04..2026-05) + Jellyfin/Movie 447f/66.5GB (2015-09..2025-12) + Jellyfin/TV 118f/35.3GB (2025-01..2025-09) + archive/Twitch 8f/30MB + QBT/0-torrents 872f/13.6MB +Bundles on SRC: NONE (b6 validated - the only bundle is $BUNDLE on DST) +Overlap: size-only 1,396 files but 0.0 GiB; name+size 472 files 0.0 GiB + => NONE of the ~176 GiB of video exists at DST yet +Read speed: 8.65 MiB/s sequential (small-file bench invalid, n=3) +Projection: ~5.8 h for one full read pass +DST split: in-bundle 108,274f / 201.4 GiB; loose 3,926f / 462.8 GiB +Jellyfin: 10.11.11, stopped, datadir 329 MB + root/default/Movies/Movie.mblink -> /Volumes/MicroSD-512/Jellyfin/Movie + root/default/TV/TV1.mblink -> /Volumes/MicroSD-512/Jellyfin/TV + (video.mblink and collections.mblink are unrelated, do not touch) +Scan errors: none (find/stat error logs empty) +``` + +## 2. Resolved decisions (do not re-ask the operator) +``` +GATE-0 imaging DECLINED. Only 37% of the card is occupied and the + payload is large sequential video; a full-device image + would read 477 GiB (~15.7 h) vs 176 GiB (~5.8 h) of + real data. Read SRC directly, concurrency 1. +Q-NEW bundle-only dupes SKIP + log to in-bundle-only.tsv. Moot in practice: + only 50 tiny JPEGs could possibly be affected. +a4 already-copied data AVOID recopy via content hash, but the measured overlap + is ~0 GiB, so expect ~176 GiB of genuine new data. +c8 move vs copy COPY. Never delete from SRC. Emit delete-candidates only. +c9 layout MIRROR. Reorg is deferred to PLAN-REORG. +c10 hashing b3sum > xxh128sum > shasum -a 256. Pick once, record it. +c11 collisions Quarantine to $DST/_CONFLICTS/<relpath>. +b7 sidecars .srt/.nfo/.jpg follow their media, exempt from dedup. +e16 watch state NOT required. No DB surgery under any circumstances. +f18 execution AGENT EXECUTES the scripts itself and reports a <=40-line + digest per phase. Agent checks with the operator BEFORE + starting each phase and before moving to the next phase. + Operator may override any proposed action. +``` + +## 3. Non-negotiable rules +``` +R1 SRC is READ-ONLY for this entire plan. No write, move, rename, delete, chmod, + chown, or touch on SRC. Deletion is a separate future task, operator-only. +R2 $BUNDLE is READ-ONLY and is used ONLY as a dedup oracle. Never write into it, + never open it with Photos.app during migration, never make it a copy target, + never point a Jellyfin library at it or at the $DST root. +R3 Each SRC byte is read once. Hash in-stream while copying. Forbidden: + rsync --checksum, any "hash pass then copy pass", any second verification + read of SRC. Verification re-reads happen on the DST side only. +R4 Concurrency against SRC is exactly 1. Parallelism against DST/internal is fine. +R5 Bulk output goes to $AUDIT files. Chat receives digests of <=40 lines only. + Never paste manifests, file lists, or hash tables into the conversation. +R6 Every phase is reuse-first and resumable. If a ledger entry exists for a file, + that file is NEVER re-read from SRC. Rebuilding requires an explicit --force. +R7 I/O errors are logged and retried once at end-of-phase, never fatal mid-run. +R8 Jellyfin: NEVER edit *.db, *.db-wal, *.db-shm, or *.db.bak* with a text editor + or with sqlite. 10.11 replaced library.db with jellyfin.db and a new schema; + all pre-10.11 path-rewrite recipes are invalid. Path changes go through + .mblink files or the web UI only. +``` + +## 4. Script hygiene (binding on every generated script) +``` +S1 Generated scripts are PURE ASCII. No em/en dashes, ellipses, curly quotes, + or non-breaking spaces, including in comments and print strings. +S2 Every awk/sed/perl program is a SINGLE LINE. If too long, write it to + $AUDIT/prog.awk and call awk -f. +S3 No apostrophes inside single-quoted strings. +S4 Operator runs `zsh -n