From a83d9afbbea8fc33d2dea159d6ab8a65a63fa832 Mon Sep 17 00:00:00 2001 From: dietrich ayala Date: Sat, 8 Aug 2026 22:24:50 +0200 Subject: [PATCH] let git overrule an exclusion list that matches on names alone MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `build` in push-project.sh's EXCLUDES means "output some command regenerates", but a repository is free to track a source file under that name, and one does: apps/publisher/build/docs.ts never reached the server. The damage is not the missing file, it is that pushing again cannot repair it — the exclusion that caused the deletion also blocks the file that would fix it, so the checkout reads as dirty from then on and every later push stops to ask about overwriting changes nobody made. git_protect_list() names every tracked file, and every ancestor directory of one, ahead of the general patterns. Ancestors are not optional: rsync prunes an excluded directory during the walk and never descends into it, so protecting the file without protecting its parent protects nothing. Three filter tiers, first match winning: a project's own .flit-push-exclude, then git's tracked files, then the general list. Explicit intent beats git, and git beats a guess about a name — a project's file outranks the protection on purpose, since a repository may track a data set the server has its own copy of. That tier can still strand a tracked file, so after the transfer the run asks the server for `git ls-files --deleted` and reports what is missing, rather than leaving the next push to discover it. Twelve cases pin this, and they assert rsync's filter semantics directly because /usr/bin/rsync is openrsync on the laptop and GNU rsync on the server; an --include-from the two read differently would restore the defect in silence. 93 passed on both. Standing rule 17. --- CLAUDE.md | 2 +- README.md | 9 +++++ flit-spec.md | 34 +++++++++++++++- laptop/push-project.sh | 75 +++++++++++++++++++++++++++++++++-- lib/common.sh | 40 +++++++++++++++++++ lib/common.test.sh | 90 ++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 244 insertions(+), 6 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index a3fb197..dd12791 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,7 +1,7 @@ # CLAUDE.md Working notes for an agent picking this up. Read `flit-spec.md` §13 before -changing anything — it encodes nine change guards and sixteen standing rules, +changing anything — it encodes nine change guards and seventeen standing rules, each derived from a defect that actually occurred here. ## The rule that outranks everything else here diff --git a/README.md b/README.md index 04fd8d4..3779766 100644 --- a/README.md +++ b/README.md @@ -201,6 +201,15 @@ are most of the bytes, and a `node_modules` built on an arm64 laptop is the wrong one for an amd64 server anyway. A project needing more exclusions lists them, one pattern per line, in a `.flit-push-exclude` file in its own root. +That list matches directory names, and a name can be wrong: a repository is free +to track a source file under `build/` or `dist/`. Anything git tracks is carried +regardless, so a blunt pattern cannot strand it. Your own `.flit-push-exclude` +still outranks that — it is an explicit decision about this project, and a +repository may well track a data set the server has its own copy of. When one of +your patterns does leave a tracked file behind, the push says which files and +what to do about it, because the server's checkout reads as dirty from then on +and pushing again will not fix it. + The push is one-way, like everything else here: the laptop is authoritative and the server never reaches back. It is not `--delete` by default, so a file that exists only on the server survives; pass `--delete` to make the server an exact diff --git a/flit-spec.md b/flit-spec.md index b351693..cbcf3d0 100644 --- a/flit-spec.md +++ b/flit-spec.md @@ -4,7 +4,7 @@ **Audience:** An agent executing the build, with a human available to approve interactive steps. > **Amending this spec? Read §13 first.** It encodes seven review techniques, two checks that -> only work once an implementation exists, and sixteen standing rules — each derived from a defect +> only work once an implementation exists, and seventeen standing rules — each derived from a defect > that actually occurred here. Several of those defects were introduced *by fixes to earlier > ones*, so amendments carry the same risk as the original. @@ -285,6 +285,26 @@ different reason — both hold absolute laptop paths that name nothing on the se needing more adds patterns to a `.flit-push-exclude` file in its own root; the committed list stays general. +**A name is not a fact, so git overrules the general list.** `build` in that list means "output some +command regenerates", but nothing stops a repository from tracking a source file at +`apps/publisher/build/docs.ts`, and one does. Dropping it makes the server's checkout permanently +dirty with a deletion no later push can heal, because the exclusion that caused it also blocks the +file that would repair it — and every push from then on stops to ask about overwriting changes +nobody made. So `git_protect_list()` (`lib/common.sh`) names every tracked file, and every ancestor +directory of one, ahead of the general patterns; rsync takes the first rule that matches. Ancestors +are not optional: rsync prunes an excluded directory during the walk and never looks inside it, so +protecting the file without protecting `apps/publisher/build/` protects nothing. + +The three tiers, in the order the filters are assembled: a project's own `.flit-push-exclude`, then +git's tracked files, then the general list. **Explicit intent beats git, and git beats a guess about +a name.** A project's file outranks the protection deliberately — it is written by someone looking +at that project who has decided a path does not belong on the server, and that decision covers a +tracked file as readily as an untracked one, since a repository may track a data set the server has +its own copy of. That is the one tier that can still strand a tracked file, so after the transfer +the run asks the server for `git ls-files --deleted` and reports what is missing. git on the far +side is the check rather than a second pass over the patterns, because it reports the state that +resulted instead of predicting it. + **Session transcripts move with the project.** Claude Code keys them by the project's absolute path with the slashes replaced by dashes, so the laptop's key and the server's are different strings and the copy is re-keyed on the way. This transfer never deletes, whatever `--delete` was given for the @@ -2218,6 +2238,18 @@ Violating any of these has broken something before: process was given. Writing the prompt to `/dev/tty` is a separate matter and still required, because `run_log_start` routes fd 2 through an awk filter that emits only complete lines and a prompt has no trailing newline. `lib/common.sh` `confirm()` is the one implementation. +17. **An exclusion list matches names; git knows facts. Let git overrule it.** A pattern like + `build` or `dist` is a guess that a directory holds regenerated output, and a repository is + free to track a source file inside one — `apps/publisher/build/docs.ts` is a real case. The + damage is not the missing file but that it cannot be repaired: the server's checkout reads as + dirty from then on, every later push stops to ask about overwriting changes nobody made, and + pushing again cannot fix it, because the exclusion that caused the deletion also blocks the + file. Name git's tracked files, and every ancestor directory of one, ahead of any general + pattern — `git_protect_list()` in `lib/common.sh`. Ancestors matter because rsync prunes an + excluded directory during the walk and never descends into it. Where a filter is still allowed + to strand a tracked file — a project's own `.flit-push-exclude`, which outranks the protection + on purpose — the run reports what went missing rather than leaving the next push to discover + it, and it asks the far side's git for that answer instead of re-deriving it from the patterns. ### When an amendment is large diff --git a/laptop/push-project.sh b/laptop/push-project.sh index e1838e1..099fef1 100755 --- a/laptop/push-project.sh +++ b/laptop/push-project.sh @@ -21,6 +21,13 @@ # server's linux amd64. EXCLUDES below is that list; a project needing more # adds them to a .flit-push-exclude file in its own root. # +# EXCLUDES matches on a directory's name, which is a guess and not a fact: a +# repository may track a source file under build/ or dist/, and dropping it +# leaves the server's checkout dirty with a deletion no later push can heal. +# So git's tracked files are protected ahead of that list. Three tiers of +# filter, first match winning: .flit-push-exclude, then git's files, then +# EXCLUDES — explicit intent beats git, and git beats a guess about a name. +# # Session transcripts move too, unless --no-sessions. Claude Code keys them by # the project's absolute path with the slashes turned into dashes, so the # laptop's directory name for a project and the server's are different keys — @@ -137,18 +144,51 @@ EXCLUDES=( .cache ) -RSYNC_EXCLUDE=() -for e in "${EXCLUDES[@]}"; do RSYNC_EXCLUDE+=(--exclude "$e"); done +GENERAL_EXCLUDE=() +for e in "${EXCLUDES[@]}"; do GENERAL_EXCLUDE+=(--exclude "$e"); done + # A project's own additions, one pattern per line. Comments and blank lines are # dropped so the file can explain itself. +# +# These are kept in their own array because they outrank the tracked-file +# protection below, where the general list does not. The two are different kinds +# of statement: EXCLUDES is a guess from a directory's name, made once for every +# project that will ever be pushed, while .flit-push-exclude is written by +# someone looking at this project who has decided this path does not belong on +# the server. That decision covers a tracked file as readily as an untracked +# one — a repository can track a data set the server has its own copy of — so +# protecting git's files from it would override the only explicit instruction +# in the whole filter chain. +PROJECT_EXCLUDE=() if [[ -r $SRC/.flit-push-exclude ]]; then while IFS= read -r line; do line=${line%%#*}; line=${line## }; line=${line%% } - [[ -n $line ]] && RSYNC_EXCLUDE+=(--exclude "$line") + [[ -n $line ]] && PROJECT_EXCLUDE+=(--exclude "$line") done <"$SRC/.flit-push-exclude" info "honouring $NAME/.flit-push-exclude" fi +# ------------------------------------------------------------------- protect + +# git's tracked files, named ahead of the general list so a blunt pattern +# cannot drop a file the repository actually carries. See git_protect_list() +# in lib/common.sh for what goes wrong without this. +PROTECT=() +protect_file="" +# An explicit template rather than `mktemp -t NAME`: BSD mktemp accepts a bare +# prefix there, GNU mktemp rejects it for having too few X's, and this script +# runs on whichever laptop the operator has. +if protect_file=$(mktemp "${TMPDIR:-/tmp}/flit-push-protect.XXXXXX") \ + && git_protect_list "$SRC" >"$protect_file"; then + if [[ -s $protect_file ]]; then + PROTECT=(--include-from="$protect_file") + fi +else + info "$NAME is not a git work tree; the exclusion list is the only filter" +fi +# shellcheck disable=SC2064 +[[ -n $protect_file ]] && trap "rm -f '$protect_file'" EXIT + # ------------------------------------------------------------ reachability # Resolving is not reaching (standing rule 15): $FLIT_SERVER is a tailnet name, @@ -189,7 +229,13 @@ fi # -i itemizes, so the run says what moved rather than only that it moved. # openrsync is the /usr/bin/rsync on a current macOS and every flag used here # is one it implements — --info=stats1 is not, and neither is -F. -RSYNC=(rsync -az -i "${RSYNC_EXCLUDE[@]}") +# Filter order IS the policy, and rsync takes the first rule that matches: +# the project's own exclusions, then git's tracked files, then the general +# list. Explicit intent beats git, and git beats a guess about a name. +RSYNC=(rsync -az -i) +[[ ${#PROJECT_EXCLUDE[@]} -gt 0 ]] && RSYNC+=("${PROJECT_EXCLUDE[@]}") +[[ ${#PROTECT[@]} -gt 0 ]] && RSYNC+=("${PROTECT[@]}") +RSYNC+=("${GENERAL_EXCLUDE[@]}") [[ $DELETE -eq 1 ]] && RSYNC+=(--delete) [[ $PRINT_ONLY -eq 1 ]] && RSYNC+=(--dry-run) @@ -203,6 +249,27 @@ fi "${RSYNC[@]}" -e "ssh ${SSH_OPTS[*]}" "$SRC/" "$FLIT_USER@$FLIT_SERVER:$DEST/" \ || die "rsync failed; the server's copy of $NAME may be half-updated" +# --------------------------------------------------- what the filters dropped + +# A filter that drops a tracked file is not wrong — .flit-push-exclude exists to +# do exactly that — but it is never invisible: the server's checkout reads as +# dirty from then on, every later push stops to ask about overwriting changes +# nobody made, and the deletion cannot be fixed by pushing again, because the +# filter that caused it also blocks the repair. So the run says which files, and +# says it here rather than leaving it to be discovered by the next push. +# +# git on the server is the check rather than a second pass over the filters, +# because it reports the state that actually resulted instead of predicting it. +if [[ $PRINT_ONLY -eq 0 ]] && remote "test -d ~/$DEST/.git"; then + gone=$(remote "cd ~/$DEST && git ls-files --deleted" 2>/dev/null) + if [[ -n $gone ]]; then + n=$(printf '%s\n' "$gone" | grep -c '') + warn "$n tracked file(s) missing from the server's checkout, most likely dropped by a filter:" + printf '%s\n' "$gone" >&2 + info "narrow the pattern in $NAME/.flit-push-exclude, or run: ssh $FLIT_ALIAS 'cd ~/$DEST && git restore .'" + fi +fi + # ------------------------------------------------------------------ sessions if [[ $SESSIONS -eq 1 ]]; then diff --git a/lib/common.sh b/lib/common.sh index 2a9b099..91cb582 100644 --- a/lib/common.sh +++ b/lib/common.sh @@ -356,6 +356,46 @@ Run this from a shell, or pass --non-interactive to assert rather than ask." need_cmd() { command -v "$1" >/dev/null 2>&1 || die "required command not found: $1"; } +# ------------------------------------------------------- protecting git's own +# git_protect_list — write, on stdout, an rsync filter list that protects +# every file git tracks in , one anchored pattern per line. Empty output, +# and a non-zero return, when is not a git work tree. +# +# This exists because a general exclusion list is a heuristic about names, and a +# name is not a fact about a file. `build` in laptop/push-project.sh's EXCLUDES +# means "output some command regenerates", but a repository is free to track a +# source file at apps/publisher/build/docs.ts, and one does. Dropping it makes +# the server's checkout permanently dirty with a deletion the next push cannot +# heal, because the exclusion that caused it also prevents the fix. +# +# git already knows which files are neither rebuildable nor disposable: the +# tracked ones. Naming them ahead of the heuristic lets the heuristic stay blunt +# and still be safe. What this does NOT override is a project's own +# .flit-push-exclude — see push-project.sh for why that ordering is deliberate. +# +# Every ancestor directory of a tracked file is emitted too. rsync prunes an +# excluded directory during traversal and never looks inside it, so protecting +# apps/publisher/build/docs.ts without also protecting apps/publisher/build/ +# protects nothing: the walk stops one level above the file. +git_protect_list() { + local dir=$1 f d + git -C "$dir" rev-parse --is-inside-work-tree >/dev/null 2>&1 || return 1 + # -z, and a null-delimited read, because a newline is a legal character in a + # filename and a line-delimited list would silently split one file into two + # patterns that match nothing. + git -C "$dir" ls-files -z 2>/dev/null | { + while IFS= read -r -d '' f; do + [[ -n $f ]] || continue + printf '/%s\n' "$f" + d=$f + while [[ $d == */* ]]; do + d=${d%/*} + printf '/%s/\n' "$d" + done + done + } | sort -u +} + # ------------------------------------------------------------ --replace flag # wants_replace — 0 if NAME appears in the space-separated FLIT_REPLACE # list, exact match only: "restic-password" must not match a FLIT_REPLACE diff --git a/lib/common.test.sh b/lib/common.test.sh index b09cb3e..7b2272e 100755 --- a/lib/common.test.sh +++ b/lib/common.test.sh @@ -601,5 +601,95 @@ chmod_paths=$(printf '%s\n' "$chmod_line" | sed -E 's/^for p in //' \ t "drill-restore.sh's chmod list matches server/bin/backup.sh's restic backup set" \ "" "$(diff <(printf '%s\n' "$backup_paths") <(printf '%s\n' "$chmod_paths"))" +# ---- git_protect_list: git names what a general exclusion pattern must not drop +# +# git and rsync are the subject of these cases, not incidental to them: what is +# being pinned is that BOTH userlands read an --include-from the same way, since +# /usr/bin/rsync is openrsync on the laptop and GNU rsync on the server. A bare +# container has neither tool, and four cases failing for that reason read as a +# defect in the filter ordering, so say which it is. +for _needed in git rsync; do + command -v "$_needed" >/dev/null 2>&1 \ + || { printf '\nerror: %s is not installed; these cases test it directly.\n' "$_needed" + printf ' In a container: apt-get update && apt-get install -y git rsync\n' + exit 2; } +done + +# +# The defect this pins: EXCLUDES in laptop/push-project.sh matches on a +# directory's NAME, so `build` drops apps/publisher/build/docs.ts — a file the +# repository tracks. The server's checkout then reads as dirty forever, and no +# later push repairs it, because the exclusion that caused the deletion also +# blocks the file that would fix it. +PROJ=$TD/protect/proj +mkdir -p "$PROJ/apps/publisher/build" "$PROJ/junk/build" "$PROJ/node_modules" +: >"$PROJ/main.ts" +: >"$PROJ/apps/publisher/build/docs.ts" +: >"$PROJ/junk/build/generated.js" +: >"$PROJ/node_modules/dep.js" +: >"$PROJ/.env" + +t "git_protect_list: a directory git does not know is not a work tree" \ + "1" "$(git_protect_list "$PROJ" >/dev/null 2>&1; echo $?)" +t "git_protect_list: and it names nothing" \ + "0" "$(git_protect_list "$PROJ" 2>/dev/null | count)" + +git -C "$PROJ" init -q >/dev/null 2>&1 +git -C "$PROJ" add main.ts apps/publisher/build/docs.ts >/dev/null 2>&1 + +PROTECT_OUT=$TD/protect/list +git_protect_list "$PROJ" >"$PROTECT_OUT" 2>/dev/null + +t "git_protect_list: a tracked file is named, anchored to the transfer root" \ + "1" "$(grep -cx '/apps/publisher/build/docs.ts' "$PROTECT_OUT")" +t "git_protect_list: every ancestor directory is named too" \ + "3" "$(grep -cE '^/(apps|apps/publisher|apps/publisher/build)/$' "$PROTECT_OUT")" +t "git_protect_list: an untracked file is not named" \ + "0" "$(grep -c 'generated.js' "$PROTECT_OUT")" +t "git_protect_list: a gitignorable working file is not named either" \ + "0" "$(grep -c '\.env' "$PROTECT_OUT")" + +# ---- the three filter tiers behave as push-project.sh relies on them to +# +# rsync takes the first rule that matches, and this asserts that on THIS +# userland: /usr/bin/rsync is openrsync on macOS and GNU rsync on the server, +# and an --include-from that the two read differently would silently restore +# the defect above. The ordering under test is push-project.sh's own: the +# project's exclusions, then git's tracked files, then the general list. +: >"$TD/protect/project-exclude" +printf '/apps/publisher/\n' >"$TD/protect/project-exclude" + +SENT=$TD/protect/sent +rsync -a -n -i --include-from="$PROTECT_OUT" --exclude build --exclude node_modules \ + "$PROJ/" "$TD/protect/dst/" 2>/dev/null >"$SENT" + +t "filters: a tracked file under an excluded name survives" \ + "1" "$(grep -c 'apps/publisher/build/docs.ts' "$SENT")" +t "filters: an untracked file under the same excluded name does not" \ + "0" "$(grep -c 'junk/build/generated.js' "$SENT")" +t "filters: the general list still drops what nothing protects" \ + "0" "$(grep -c 'node_modules/dep.js' "$SENT")" +t "filters: an untracked working file is still carried" \ + "1" "$(grep -c '\.env' "$SENT")" + +SENT2=$TD/protect/sent2 +rsync -a -n -i --exclude-from="$TD/protect/project-exclude" \ + --include-from="$PROTECT_OUT" --exclude build --exclude node_modules \ + "$PROJ/" "$TD/protect/dst2/" 2>/dev/null >"$SENT2" + +t "filters: a project's own exclusion outranks git's tracked files" \ + "0" "$(grep -c 'apps/publisher/build/docs.ts' "$SENT2")" + +# ---- push-project.sh builds the tiers in that order +# +# The rsync semantics above are worth nothing if the caller assembles them +# backwards, and the ordering is three lines apart in the file, so it is exactly +# the kind of thing a later edit reorders without noticing. +ORDER=$(grep -n 'RSYNC+=(\("\${PROJECT_EXCLUDE\|"\${PROTECT\|"\${GENERAL_EXCLUDE\)' laptop/push-project.sh \ + | sed -e 's/:.*PROJECT_EXCLUDE.*/ project/' -e 's/:.*PROTECT.*/ protect/' -e 's/:.*GENERAL_EXCLUDE.*/ general/' \ + | sed 's/^[0-9]* //' | tr '\n' ' ') +t "push-project.sh orders the filters project, then git, then general" \ + "project protect general " "$ORDER" + printf '\n%d passed, %d failed\n' "$PASS" "$FAIL" [[ $FAIL -eq 0 ]] -- 2.51.2