psingletary.com / _shared
This repository has no description
Star
Fork 0
Watch
_shared docs UNIFIED-BACKUP-MAPPING.md
4.7 kB
Markdown
at main

UNIFIED X10 BACKUP MAPPING — review draft (2026-08-20) #

Goal: one reusable backup pipeline for all three sources (Mac Studio, MacBook Air, Crucial 2TB) that writes INTO the canonical X10 tree. X10 canonical structure is authoritative. Provenance in _ledger/, not in paths. True global dedup: one physical copy per unique sha256 across all sources.

Status: REVIEW DRAFT — no scripts changed yet. Awaiting operator approval.

1. Canonical X10 target tree (authoritative, just built) #

Media/{Movies,TV,Twitch,Music-Videos,Clips,Rips,Other-Video}/
Music/  Pictures/  Documents/  Desktop/  Downloads/  QBT/{torrents,incomplete}/
Archive/  z-New folder/  photos-restore.photoslibrary/  _ledger/  _CONFLICTS/

2. Semantics #

  • Machines (Studio + Air): MOVE (frees local disk), dedup-aware. For each source file: sha256. - if sha already in _ledger/provenance.csv (content canonical on X10): record this machine as an additional provenance source, then remove the local file (content is already safely on X10). Frees disk, NO duplicate. - else: rsync verify-then-delete (--checksum --remove-source-files) into the canonical target, add provenance row.
  • Crucial 2TB: COPY (never repoint Jellyfin/qBittorrent; source stays live). Dedup-aware too: skip files whose sha is already canonical; copy only unique ones.

3. Machine (Studio + Air) source -> canonical target #

~/Music        -> X10/Music/
~/Pictures     -> X10/Pictures/
~/Documents    -> X10/Documents/
~/Desktop      -> X10/Desktop/
~/Movies       -> X10/Media/Movies/        (iMovie Library -> Media/Movies/ or Other-Video/)
~/Downloads:
   stacher/    -> X10/Media/Music-Videos/  (music-video downloads; adult content policy:
                    if flagged adult, X10/z-New folder/ instead — operator confirm)
   Media/      -> X10/Media/Other-Video/   (generic download-media)
   Music/      -> X10/Music/
   *.dmg/pkg/exe/msi/iso/zip/tar/gz/tgz/bz2/xz/7z -> X10/Archive/installers/
   *.pdf       -> X10/Documents/
   other       -> X10/Downloads/
No per-machine subdirs: both machines write into the SAME canonical dirs; dedup +
provenance in _ledger separates them.

4. Crucial 2TB source -> canonical target #

Media/Movies   -> X10/Media/Movies/
Media/TV       -> X10/Media/TV/
Media/Twitch   -> X10/Media/Twitch/
Media/...      -> X10/Media/<same-subdir>
Music/         -> X10/Music/
Pictures/      -> X10/Pictures/
QBT/           -> X10/QBT/torrents/
Archive/       -> X10/Archive/
photos-restore.photoslibrary -> X10/photos-restore.photoslibrary/  (OPAQUE, R5)

5. What goes away (complexity/redundancy reduced) #

  • X10/archive/// per-machine buckets -> canonical top-levels (dedup)
  • X10/_photos-library-mirror/ whole-volume root -> absorbed into canonical top-levels
  • Per-machine subdirs in target paths -> provenance column in _ledger

6. Provenance #

_ledger/provenance.csv (index, comma, 4 cols: sha|canonical_path|size|topdir) = authoritative canonical index from the reorg. The unified backup appends NEW machine/2TB provenance to a separate file _ledger/provenance-machines.csv (sha|canonical_path|size|machine|source_relpath), so the reorg index stays untouched. Dedup authority = union of shas in BOTH files. A file present on Studio + 2TB = 1 physical copy, 2 machine-provenance rows.

7. Pipeline (launchd on each machine when X10 attached) #

  • unified-backup.zsh runs under launchd (com.psingletary.backup3, 60s poll) on EACH machine.
  • When X10 is attached to a machine, that machine runs its own machine-MOVE phase.
  • On the Studio (which also hosts the 2TB), the 2TB COPY phase runs after the machine phase.
  • Exits 0 in <1s when X10 absent (launchd-safe). Dry-run default; --apply for real.

8. Resolved decisions (2026-08-20, operator) #

  • Machine semantics: MOVE (free local disk) + 2TB: COPY. Confirmed.
  • Adult stacher content -> top-level z-New folder/. (routed in route_downloads)
  • OneDrive routing DISABLED by default (2026-08-20): writing to OneDrive from launchd context deadlocks ("Resource deadlock avoided" — File Provider). All files go to X10 canonical. Re-enable with USE_ONEDRIVE=yes for manual runs only.
  • When a machine file's sha is already canonical, delete the local copy (X10 has it).
  • 2TB phase uses rsync --ignore-existing (path-based skip) — NOT per-file shasum, so a launchd fire does not re-read the whole 2TB. Efficient, correct for 1:1 mapping.
  • Provenance: machine/2TB rows go to _ledger/provenance-machines.csv (reorg index untouched).