diff --git a/CLAUDE.md b/CLAUDE.md index a3fb197..dd12791 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,7 +1,7 @@ # CLAUDE.md Working notes for an agent picking this up. Read `flit-spec.md` §13 before -changing anything — it encodes nine change guards and sixteen standing rules, +changing anything — it encodes nine change guards and seventeen standing rules, each derived from a defect that actually occurred here. ## The rule that outranks everything else here diff --git a/README.md b/README.md index 04fd8d4..3779766 100644 --- a/README.md +++ b/README.md @@ -201,6 +201,15 @@ are most of the bytes, and a `node_modules` built on an arm64 laptop is the wrong one for an amd64 server anyway. A project needing more exclusions lists them, one pattern per line, in a `.flit-push-exclude` file in its own root. +That list matches directory names, and a name can be wrong: a repository is free +to track a source file under `build/` or `dist/`. Anything git tracks is carried +regardless, so a blunt pattern cannot strand it. Your own `.flit-push-exclude` +still outranks that — it is an explicit decision about this project, and a +repository may well track a data set the server has its own copy of. When one of +your patterns does leave a tracked file behind, the push says which files and +what to do about it, because the server's checkout reads as dirty from then on +and pushing again will not fix it. + The push is one-way, like everything else here: the laptop is authoritative and the server never reaches back. It is not `--delete` by default, so a file that exists only on the server survives; pass `--delete` to make the server an exact diff --git a/flit-spec.md b/flit-spec.md index b351693..cbcf3d0 100644 --- a/flit-spec.md +++ b/flit-spec.md @@ -4,7 +4,7 @@ **Audience:** An agent executing the build, with a human available to approve interactive steps. > **Amending this spec? Read §13 first.** It encodes seven review techniques, two checks that -> only work once an implementation exists, and sixteen standing rules — each derived from a defect +> only work once an implementation exists, and seventeen standing rules — each derived from a defect > that actually occurred here. Several of those defects were introduced *by fixes to earlier > ones*, so amendments carry the same risk as the original. @@ -285,6 +285,26 @@ different reason — both hold absolute laptop paths that name nothing on the se needing more adds patterns to a `.flit-push-exclude` file in its own root; the committed list stays general. +**A name is not a fact, so git overrules the general list.** `build` in that list means "output some +command regenerates", but nothing stops a repository from tracking a source file at +`apps/publisher/build/docs.ts`, and one does. Dropping it makes the server's checkout permanently +dirty with a deletion no later push can heal, because the exclusion that caused it also blocks the +file that would repair it — and every push from then on stops to ask about overwriting changes +nobody made. So `git_protect_list()` (`lib/common.sh`) names every tracked file, and every ancestor +directory of one, ahead of the general patterns; rsync takes the first rule that matches. Ancestors +are not optional: rsync prunes an excluded directory during the walk and never looks inside it, so +protecting the file without protecting `apps/publisher/build/` protects nothing. + +The three tiers, in the order the filters are assembled: a project's own `.flit-push-exclude`, then +git's tracked files, then the general list. **Explicit intent beats git, and git beats a guess about +a name.** A project's file outranks the protection deliberately — it is written by someone looking +at that project who has decided a path does not belong on the server, and that decision covers a +tracked file as readily as an untracked one, since a repository may track a data set the server has +its own copy of. That is the one tier that can still strand a tracked file, so after the transfer +the run asks the server for `git ls-files --deleted` and reports what is missing. git on the far +side is the check rather than a second pass over the patterns, because it reports the state that +resulted instead of predicting it. + **Session transcripts move with the project.** Claude Code keys them by the project's absolute path with the slashes replaced by dashes, so the laptop's key and the server's are different strings and the copy is re-keyed on the way. This transfer never deletes, whatever `--delete` was given for the @@ -2218,6 +2238,18 @@ Violating any of these has broken something before: process was given. Writing the prompt to `/dev/tty` is a separate matter and still required, because `run_log_start` routes fd 2 through an awk filter that emits only complete lines and a prompt has no trailing newline. `lib/common.sh` `confirm()` is the one implementation. +17. **An exclusion list matches names; git knows facts. Let git overrule it.** A pattern like + `build` or `dist` is a guess that a directory holds regenerated output, and a repository is + free to track a source file inside one — `apps/publisher/build/docs.ts` is a real case. The + damage is not the missing file but that it cannot be repaired: the server's checkout reads as + dirty from then on, every later push stops to ask about overwriting changes nobody made, and + pushing again cannot fix it, because the exclusion that caused the deletion also blocks the + file. Name git's tracked files, and every ancestor directory of one, ahead of any general + pattern — `git_protect_list()` in `lib/common.sh`. Ancestors matter because rsync prunes an + excluded directory during the walk and never descends into it. Where a filter is still allowed + to strand a tracked file — a project's own `.flit-push-exclude`, which outranks the protection + on purpose — the run reports what went missing rather than leaving the next push to discover + it, and it asks the far side's git for that answer instead of re-deriving it from the patterns. ### When an amendment is large diff --git a/laptop/push-project.sh b/laptop/push-project.sh index e1838e1..099fef1 100755 --- a/laptop/push-project.sh +++ b/laptop/push-project.sh @@ -21,6 +21,13 @@ # server's linux amd64. EXCLUDES below is that list; a project needing more # adds them to a .flit-push-exclude file in its own root. # +# EXCLUDES matches on a directory's name, which is a guess and not a fact: a +# repository may track a source file under build/ or dist/, and dropping it +# leaves the server's checkout dirty with a deletion no later push can heal. +# So git's tracked files are protected ahead of that list. Three tiers of +# filter, first match winning: .flit-push-exclude, then git's files, then +# EXCLUDES — explicit intent beats git, and git beats a guess about a name. +# # Session transcripts move too, unless --no-sessions. Claude Code keys them by # the project's absolute path with the slashes turned into dashes, so the # laptop's directory name for a project and the server's are different keys — @@ -137,18 +144,51 @@ EXCLUDES=( .cache ) -RSYNC_EXCLUDE=() -for e in "${EXCLUDES[@]}"; do RSYNC_EXCLUDE+=(--exclude "$e"); done +GENERAL_EXCLUDE=() +for e in "${EXCLUDES[@]}"; do GENERAL_EXCLUDE+=(--exclude "$e"); done + # A project's own additions, one pattern per line. Comments and blank lines are # dropped so the file can explain itself. +# +# These are kept in their own array because they outrank the tracked-file +# protection below, where the general list does not. The two are different kinds +# of statement: EXCLUDES is a guess from a directory's name, made once for every +# project that will ever be pushed, while .flit-push-exclude is written by +# someone looking at this project who has decided this path does not belong on +# the server. That decision covers a tracked file as readily as an untracked +# one — a repository can track a data set the server has its own copy of — so +# protecting git's files from it would override the only explicit instruction +# in the whole filter chain. +PROJECT_EXCLUDE=() if [[ -r $SRC/.flit-push-exclude ]]; then while IFS= read -r line; do line=${line%%#*}; line=${line## }; line=${line%% } - [[ -n $line ]] && RSYNC_EXCLUDE+=(--exclude "$line") + [[ -n $line ]] && PROJECT_EXCLUDE+=(--exclude "$line") done <"$SRC/.flit-push-exclude" info "honouring $NAME/.flit-push-exclude" fi +# ------------------------------------------------------------------- protect + +# git's tracked files, named ahead of the general list so a blunt pattern +# cannot drop a file the repository actually carries. See git_protect_list() +# in lib/common.sh for what goes wrong without this. +PROTECT=() +protect_file="" +# An explicit template rather than `mktemp -t NAME`: BSD mktemp accepts a bare +# prefix there, GNU mktemp rejects it for having too few X's, and this script +# runs on whichever laptop the operator has. +if protect_file=$(mktemp "${TMPDIR:-/tmp}/flit-push-protect.XXXXXX") \ + && git_protect_list "$SRC" >"$protect_file"; then + if [[ -s $protect_file ]]; then + PROTECT=(--include-from="$protect_file") + fi +else + info "$NAME is not a git work tree; the exclusion list is the only filter" +fi +# shellcheck disable=SC2064 +[[ -n $protect_file ]] && trap "rm -f '$protect_file'" EXIT + # ------------------------------------------------------------ reachability # Resolving is not reaching (standing rule 15): $FLIT_SERVER is a tailnet name, @@ -189,7 +229,13 @@ fi # -i itemizes, so the run says what moved rather than only that it moved. # openrsync is the /usr/bin/rsync on a current macOS and every flag used here # is one it implements — --info=stats1 is not, and neither is -F. -RSYNC=(rsync -az -i "${RSYNC_EXCLUDE[@]}") +# Filter order IS the policy, and rsync takes the first rule that matches: +# the project's own exclusions, then git's tracked files, then the general +# list. Explicit intent beats git, and git beats a guess about a name. +RSYNC=(rsync -az -i) +[[ ${#PROJECT_EXCLUDE[@]} -gt 0 ]] && RSYNC+=("${PROJECT_EXCLUDE[@]}") +[[ ${#PROTECT[@]} -gt 0 ]] && RSYNC+=("${PROTECT[@]}") +RSYNC+=("${GENERAL_EXCLUDE[@]}") [[ $DELETE -eq 1 ]] && RSYNC+=(--delete) [[ $PRINT_ONLY -eq 1 ]] && RSYNC+=(--dry-run) @@ -203,6 +249,27 @@ fi "${RSYNC[@]}" -e "ssh ${SSH_OPTS[*]}" "$SRC/" "$FLIT_USER@$FLIT_SERVER:$DEST/" \ || die "rsync failed; the server's copy of $NAME may be half-updated" +# --------------------------------------------------- what the filters dropped + +# A filter that drops a tracked file is not wrong — .flit-push-exclude exists to +# do exactly that — but it is never invisible: the server's checkout reads as +# dirty from then on, every later push stops to ask about overwriting changes +# nobody made, and the deletion cannot be fixed by pushing again, because the +# filter that caused it also blocks the repair. So the run says which files, and +# says it here rather than leaving it to be discovered by the next push. +# +# git on the server is the check rather than a second pass over the filters, +# because it reports the state that actually resulted instead of predicting it. +if [[ $PRINT_ONLY -eq 0 ]] && remote "test -d ~/$DEST/.git"; then + gone=$(remote "cd ~/$DEST && git ls-files --deleted" 2>/dev/null) + if [[ -n $gone ]]; then + n=$(printf '%s\n' "$gone" | grep -c '') + warn "$n tracked file(s) missing from the server's checkout, most likely dropped by a filter:" + printf '%s\n' "$gone" >&2 + info "narrow the pattern in $NAME/.flit-push-exclude, or run: ssh $FLIT_ALIAS 'cd ~/$DEST && git restore .'" + fi +fi + # ------------------------------------------------------------------ sessions if [[ $SESSIONS -eq 1 ]]; then diff --git a/lib/common.sh b/lib/common.sh index 2a9b099..91cb582 100644 --- a/lib/common.sh +++ b/lib/common.sh @@ -356,6 +356,46 @@ Run this from a shell, or pass --non-interactive to assert rather than ask." need_cmd() { command -v "$1" >/dev/null 2>&1 || die "required command not found: $1"; } +# ------------------------------------------------------- protecting git's own +# git_protect_list — write, on stdout, an rsync filter list that protects +# every file git tracks in , one anchored pattern per line. Empty output, +# and a non-zero return, when is not a git work tree. +# +# This exists because a general exclusion list is a heuristic about names, and a +# name is not a fact about a file. `build` in laptop/push-project.sh's EXCLUDES +# means "output some command regenerates", but a repository is free to track a +# source file at apps/publisher/build/docs.ts, and one does. Dropping it makes +# the server's checkout permanently dirty with a deletion the next push cannot +# heal, because the exclusion that caused it also prevents the fix. +# +# git already knows which files are neither rebuildable nor disposable: the +# tracked ones. Naming them ahead of the heuristic lets the heuristic stay blunt +# and still be safe. What this does NOT override is a project's own +# .flit-push-exclude — see push-project.sh for why that ordering is deliberate. +# +# Every ancestor directory of a tracked file is emitted too. rsync prunes an +# excluded directory during traversal and never looks inside it, so protecting +# apps/publisher/build/docs.ts without also protecting apps/publisher/build/ +# protects nothing: the walk stops one level above the file. +git_protect_list() { + local dir=$1 f d + git -C "$dir" rev-parse --is-inside-work-tree >/dev/null 2>&1 || return 1 + # -z, and a null-delimited read, because a newline is a legal character in a + # filename and a line-delimited list would silently split one file into two + # patterns that match nothing. + git -C "$dir" ls-files -z 2>/dev/null | { + while IFS= read -r -d '' f; do + [[ -n $f ]] || continue + printf '/%s\n' "$f" + d=$f + while [[ $d == */* ]]; do + d=${d%/*} + printf '/%s/\n' "$d" + done + done + } | sort -u +} + # ------------------------------------------------------------ --replace flag # wants_replace — 0 if NAME appears in the space-separated FLIT_REPLACE # list, exact match only: "restic-password" must not match a FLIT_REPLACE diff --git a/lib/common.test.sh b/lib/common.test.sh index b09cb3e..7b2272e 100755 --- a/lib/common.test.sh +++ b/lib/common.test.sh @@ -601,5 +601,95 @@ chmod_paths=$(printf '%s\n' "$chmod_line" | sed -E 's/^for p in //' \ t "drill-restore.sh's chmod list matches server/bin/backup.sh's restic backup set" \ "" "$(diff <(printf '%s\n' "$backup_paths") <(printf '%s\n' "$chmod_paths"))" +# ---- git_protect_list: git names what a general exclusion pattern must not drop +# +# git and rsync are the subject of these cases, not incidental to them: what is +# being pinned is that BOTH userlands read an --include-from the same way, since +# /usr/bin/rsync is openrsync on the laptop and GNU rsync on the server. A bare +# container has neither tool, and four cases failing for that reason read as a +# defect in the filter ordering, so say which it is. +for _needed in git rsync; do + command -v "$_needed" >/dev/null 2>&1 \ + || { printf '\nerror: %s is not installed; these cases test it directly.\n' "$_needed" + printf ' In a container: apt-get update && apt-get install -y git rsync\n' + exit 2; } +done + +# +# The defect this pins: EXCLUDES in laptop/push-project.sh matches on a +# directory's NAME, so `build` drops apps/publisher/build/docs.ts — a file the +# repository tracks. The server's checkout then reads as dirty forever, and no +# later push repairs it, because the exclusion that caused the deletion also +# blocks the file that would fix it. +PROJ=$TD/protect/proj +mkdir -p "$PROJ/apps/publisher/build" "$PROJ/junk/build" "$PROJ/node_modules" +: >"$PROJ/main.ts" +: >"$PROJ/apps/publisher/build/docs.ts" +: >"$PROJ/junk/build/generated.js" +: >"$PROJ/node_modules/dep.js" +: >"$PROJ/.env" + +t "git_protect_list: a directory git does not know is not a work tree" \ + "1" "$(git_protect_list "$PROJ" >/dev/null 2>&1; echo $?)" +t "git_protect_list: and it names nothing" \ + "0" "$(git_protect_list "$PROJ" 2>/dev/null | count)" + +git -C "$PROJ" init -q >/dev/null 2>&1 +git -C "$PROJ" add main.ts apps/publisher/build/docs.ts >/dev/null 2>&1 + +PROTECT_OUT=$TD/protect/list +git_protect_list "$PROJ" >"$PROTECT_OUT" 2>/dev/null + +t "git_protect_list: a tracked file is named, anchored to the transfer root" \ + "1" "$(grep -cx '/apps/publisher/build/docs.ts' "$PROTECT_OUT")" +t "git_protect_list: every ancestor directory is named too" \ + "3" "$(grep -cE '^/(apps|apps/publisher|apps/publisher/build)/$' "$PROTECT_OUT")" +t "git_protect_list: an untracked file is not named" \ + "0" "$(grep -c 'generated.js' "$PROTECT_OUT")" +t "git_protect_list: a gitignorable working file is not named either" \ + "0" "$(grep -c '\.env' "$PROTECT_OUT")" + +# ---- the three filter tiers behave as push-project.sh relies on them to +# +# rsync takes the first rule that matches, and this asserts that on THIS +# userland: /usr/bin/rsync is openrsync on macOS and GNU rsync on the server, +# and an --include-from that the two read differently would silently restore +# the defect above. The ordering under test is push-project.sh's own: the +# project's exclusions, then git's tracked files, then the general list. +: >"$TD/protect/project-exclude" +printf '/apps/publisher/\n' >"$TD/protect/project-exclude" + +SENT=$TD/protect/sent +rsync -a -n -i --include-from="$PROTECT_OUT" --exclude build --exclude node_modules \ + "$PROJ/" "$TD/protect/dst/" 2>/dev/null >"$SENT" + +t "filters: a tracked file under an excluded name survives" \ + "1" "$(grep -c 'apps/publisher/build/docs.ts' "$SENT")" +t "filters: an untracked file under the same excluded name does not" \ + "0" "$(grep -c 'junk/build/generated.js' "$SENT")" +t "filters: the general list still drops what nothing protects" \ + "0" "$(grep -c 'node_modules/dep.js' "$SENT")" +t "filters: an untracked working file is still carried" \ + "1" "$(grep -c '\.env' "$SENT")" + +SENT2=$TD/protect/sent2 +rsync -a -n -i --exclude-from="$TD/protect/project-exclude" \ + --include-from="$PROTECT_OUT" --exclude build --exclude node_modules \ + "$PROJ/" "$TD/protect/dst2/" 2>/dev/null >"$SENT2" + +t "filters: a project's own exclusion outranks git's tracked files" \ + "0" "$(grep -c 'apps/publisher/build/docs.ts' "$SENT2")" + +# ---- push-project.sh builds the tiers in that order +# +# The rsync semantics above are worth nothing if the caller assembles them +# backwards, and the ordering is three lines apart in the file, so it is exactly +# the kind of thing a later edit reorders without noticing. +ORDER=$(grep -n 'RSYNC+=(\("\${PROJECT_EXCLUDE\|"\${PROTECT\|"\${GENERAL_EXCLUDE\)' laptop/push-project.sh \ + | sed -e 's/:.*PROJECT_EXCLUDE.*/ project/' -e 's/:.*PROTECT.*/ protect/' -e 's/:.*GENERAL_EXCLUDE.*/ general/' \ + | sed 's/^[0-9]* //' | tr '\n' ' ') +t "push-project.sh orders the filters project, then git, then general" \ + "project protect general " "$ORDER" + printf '\n%d passed, %d failed\n' "$PASS" "$FAIL" [[ $FAIL -eq 0 ]]