From 8e7c1e712cc9c9aa92bd67f469acc4cf6992a0ec Mon Sep 17 00:00:00 2001 From: Graham Barber Date: Wed, 15 Jul 2026 10:45:22 -0700 Subject: [PATCH] propose csv-backfill-import change Co-Authored-By: Claude Opus 4.8 --- .../csv-backfill-import/.openspec.yaml | 2 + .../changes/csv-backfill-import/design.md | 281 ++++++++++++++++++ .../changes/csv-backfill-import/proposal.md | 93 ++++++ .../specs/csv-import/spec.md | 236 +++++++++++++++ .../specs/reporting/spec.md | 32 ++ openspec/changes/csv-backfill-import/tasks.md | 141 +++++++++ 6 files changed, 785 insertions(+) create mode 100644 openspec/changes/csv-backfill-import/.openspec.yaml create mode 100644 openspec/changes/csv-backfill-import/design.md create mode 100644 openspec/changes/csv-backfill-import/proposal.md create mode 100644 openspec/changes/csv-backfill-import/specs/csv-import/spec.md create mode 100644 openspec/changes/csv-backfill-import/specs/reporting/spec.md create mode 100644 openspec/changes/csv-backfill-import/tasks.md diff --git a/openspec/changes/csv-backfill-import/.openspec.yaml b/openspec/changes/csv-backfill-import/.openspec.yaml new file mode 100644 index 0000000..4f63482 --- /dev/null +++ b/openspec/changes/csv-backfill-import/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-07-15 diff --git a/openspec/changes/csv-backfill-import/design.md b/openspec/changes/csv-backfill-import/design.md new file mode 100644 index 0000000..38dc4dd --- /dev/null +++ b/openspec/changes/csv-backfill-import/design.md @@ -0,0 +1,281 @@ +## Context + +Quantum's ingestion spine is: fetch bytes → archive them verbatim in `raw_syncs` +→ normalize as a pure, replayable function of the archived payload → upsert. That +shape is load-bearing and this change preserves it. + +The relevant current state: + +- `transactions` is keyed `UNIQUE (account_id, sfin_id)` with `sfin_id TEXT NOT + NULL`. Identity comes from the provider; nothing in the codebase synthesizes it. +- `ingestTransactions` (`src/lib/server/services/sync.ts:183`) is an + **authoritative** writer. It assumes its input is the complete truth for the + account over a window: it reconciles new posted rows against pending rows within + ±5 days, and it soft-removes every pending row absent from the feed + (`sync.ts:291-301`). +- `accounts.connection_id` is `NOT NULL`, and `account-management` requires that + accounts originate only from sync. +- `normalizePayload` (`normalize.ts:61`) is pure and emits `NormalizedAccount[]`. +- `applyRulesToUncategorized` (`rules.ts:168`) is unscoped and database-wide. +- Reports scope on `t.pending = 0 AND t.removed_at IS NULL` (`reports.ts:7`) and + order by `COALESCE(posted, transacted_at, created_at)`. + +The user-facing problem is one account whose SimpleFIN feed reaches back only +about a week. Critically, its synced history still **accumulates**: each daily sync +tops up the last seven days, so over time the account holds synced rows across a +widening stretch. A backfill CSV and the synced data therefore overlap across a +*region*, not at a boundary line. + +## Goals / Non-Goals + +**Goals:** + +- Close a historical gap in one existing account from a bank's CSV export. +- Never corrupt synced data. An import must not be able to delete, reconcile, or + overwrite anything sync owns. +- Re-importing the same or an overlapping file must be a no-op, not a duplication. +- Surface plausible cross-source duplicates for human judgment before commit. +- Make a botched import reversible as a unit. +- Preserve the archive-then-replay property for CSV payloads. + +**Non-Goals:** + +- Creating accounts from a CSV. `account-management` forbids it, and a + connection-less account would have no balance snapshots, which would silently + break the net-worth report. Out of scope, deliberately. +- Importing a category column. Provenance for "Mint thought so in 2024" is an + unresolved question and `categorization_events.source` is a closed enum. Rules + cover the need for now. +- Saved per-account mapping profiles. Worth it only if imports recur; this is a + one-time gap fill. +- Fuzzy merchant matching. Duplicate detection stays on amount and date. + +## Decisions + +### 1. CSV is an additive writer, not a reuse of `ingestTransactions` + +**Decision:** A new, insert-only ingestion path. It never reconciles, never +sweeps, never updates an existing row. + +**Why:** Reusing `ingestTransactions` looks attractive — it's the same table and +the same `NormalizedTransaction` shape — but it would be a live bug. Its stale +sweep soft-deletes every pending row in the account not present in the feed; a +2023 backfill contains none of today's pending rows, so importing would silently +remove all of them. Its ±5-day reconciliation could also let an old CSV row hijack +a live pending row. + +The seam is one level below the function: sync and CSV share the table and the +insert, but not the authority. Sync is authoritative for a window; CSV is additive +into a gap. Encoding that distinction as two writers is both safer and less code +than parameterizing one writer with a `mode` flag, which would leave the dangerous +paths one boolean away from a backfill. + +**Alternative considered:** Add `authoritative: boolean` to `ingestTransactions`. +Rejected — the flag makes the hazardous branches reachable from the CSV path by +mistake, and the shared code would be only the INSERT statement. + +### 2. Synthetic identity: content hash + occurrence index + +**Decision:** `sfin_id = 'csv:' + hash(account_id, date, amount_cents, +description, occurrence_index)`, where `occurrence_index` is the row's ordinal +among identical rows within the same file. + +**Why:** `sfin_id` is `NOT NULL` under a unique key, so CSV must invent one. A +plain content hash would collapse two genuinely distinct identical transactions +(two $4.75 coffees on one Tuesday) into a single row. The occurrence index keeps +them distinct while staying deterministic. + +This yields idempotency with a pleasing property: because the grouping key *is* +the row's content, rows within a group are interchangeable. If a second export +lists the same group in a different order, or contains a superset of it, the set +of derived ids is unchanged for the rows that already exist, and only genuinely +new rows insert. Re-import of an identical or overlapping file is therefore a +no-op that never even reaches duplicate review. + +**Alternative considered:** Rename `sfin_id` to `source_id` and add a `source` +discriminator. Honest, but it touches every query in the codebase, and `import_id` +(decision 5) already records true origin. Namespacing the value with a `csv:` +prefix keeps the lie small and greppable. + +### 3. Archive on upload, decide later + +**Decision:** The verbatim file is written to a new `imports` row at upload time +with `status = 'draft'`, before any parsing. The column mapping and the duplicate +decisions are stored on that same row. Wizard steps re-parse the archived bytes. + +**Why:** This does double duty. It honors the existing "archive before +normalization" spine, and it solves where a multi-step wizard keeps its state +without stuffing a CSV into a session. Replay determinism requires the mapping and +the decisions to be archived alongside the bytes — bytes alone do not determine +the outcome, since a human chose the mapping and adjudicated the duplicates. + +**Alternative considered:** A separate `raw_imports` (bytes) plus `imports` +(record). Rejected as over-normalization for a two-person app; the payload column +inside `imports` is written once and never mutated, which preserves the property +that matters. + +### 4. Duplicate detection: amount and date, never description + +**Decision:** A CSV row is flagged when the target account already holds a +non-removed transaction with the identical `amount_cents` and an effective date +within ±1 day. Descriptions are displayed side by side but never matched on. +Flagged rows default to **skip**. + +**Why:** The two sources render the same merchant differently — SimpleFIN gives +the bank's raw string (`AMZN Mktp US*2K4LM9QR3`), the CSV gives its own rendering +(`Amazon`). Matching on description would catch almost nothing, which is the worst +outcome: a duplicate check that appears to run and silently passes everything. So +description is evidence for the human, not a filter. + +Amount + date alone is deliberately loose and will throw occasional false +positives (a second identical coffee). That is what the review step is for; a +loose matcher plus a human beats a tight matcher that misses. + +The ±1 day window rather than same-day: a CSV's single date column is often the +transaction date while SimpleFIN's `posted` can trail it, so strict same-day would +miss the very rows most at risk. + +**Skip as the default is principled, not arbitrary.** Where both sources claim a +transaction, the synced row is strictly better: it carries a real SimpleFIN id so +it stays idempotent under future syncs, it holds the bank's own description, and +it already carries categorization history. The CSV row is a photocopy of a record +already held. Declining it loses nothing. The human's job in the wizard is to +catch the false positives and flip those few to keep — a much lighter chore than +adjudicating every row from scratch. + +### 5. `import_id` on transactions: undo and origin in one column + +**Decision:** `transactions.import_id INTEGER NULL REFERENCES imports (id)`. +NULL means synced. + +**Why:** One nullable column serves three needs. Undo becomes "soft-remove where +`import_id = ?`". The ledger's source filter becomes `import_id IS NULL` / +`IS NOT NULL`. The origin line in the history panel joins through it. Retrofitting +this later would mean re-deriving hashes to work out which rows came from which +file. + +Undo **soft-removes** (`removed_at = now`) rather than hard-deleting, matching the +existing stale-pending precedent and the house rule that history is never deleted. +Hard deletion would also collide with the append-only categorization event log, +since events reference `transaction_id`. + +Undo sets `imports.status = 'undone'`. Re-importing a corrected file after an undo +generally produces different hashes (a fixed mapping parses different values), so +new rows insert cleanly. In the case where the same mapping is re-imported, the +insert path finds the soft-removed row by unique key and revives it +(`removed_at = NULL`, new `import_id`) rather than erroring. + +### 6. Which date column to write + +**Decision:** Map the CSV's date to `posted`, and set `pending = 0`. If the file +exposes a distinct transaction date, map it to `transacted_at`; otherwise leave it +NULL. + +**Why:** Reports scope on `pending = 0` (`reports.ts:7`), not on `posted IS NOT +NULL`, so either choice lands in reports. But a row with `pending = 0` and `posted +IS NULL` is a state sync has never produced, and inventing novel states for +downstream queries to encounter is the riskier path. A CSV export is by definition +a statement of settled history, so `pending = 0` with `posted` set matches the +shape sync emits for posted rows, and every existing query behaves identically. + +The residual imprecision — the file's single date column may be the transaction +date rather than the post date — is absorbed by the ±1 day duplicate window. + +### 7. Amount parsing: a CSV pre-pass, not a change to `parseAmountToCents` + +**Decision:** Normalize CSV negative conventions (`(12.34)`, `12.34-`, currency +symbols) to a signed decimal string, then hand off to the existing +`parseAmountToCents`. Support a single signed column, and separate debit/credit +columns, as distinct mapping modes. + +**Why:** `parseAmountToCents` (`normalize.ts:48`) is carefully float-free and its +regex `^([+-]?)(\d*)(?:\.(\d+))?$` is a deliberate contract for SimpleFIN's +numeric strings. Loosening it to accept accountant's parentheses would weaken +validation on the sync path to serve the CSV path. A pre-pass keeps each source's +tolerances where they belong. + +### 8. Categorization comes free + +**Decision:** Call `applyRulesToUncategorized(db)` once after commit. + +**Why:** It's already unscoped and database-wide (`rules.ts:168`), and it already +skips transactions whose latest event is `manual`. Backfilled history categorizes +itself against existing rules with no new code and no new event source. + +### 9. Surfacing: filter and popover, no ledger badge + +**Decision:** `LedgerFilters` gains `source?: 'synced' | 'imported'`; `LedgerRow` +gains origin; the existing history panel gains an origin line. No per-row badge. +The import record lives in Settings, not the main nav. + +**Why:** Within a backfilled range every row is imported and above it none are, so +a per-row badge would be present on 100% of rows where it appears — carrying no +information exactly where it's densest, while competing with the existing +categorization badge, which answers a different question (*who chose this +category*, not *where did this row come from*). + +A seam marker at the import boundary was considered and rejected on stronger +grounds: there is no boundary. Because the short-history account accumulates +synced rows daily, synced and imported rows genuinely interleave across the same +dates. A seam would be a drawn line where none exists. + +Origin interest is episodic — it matters during and shortly after an import, then +it is just history. A filter answers it on demand; the popover answers it for a +specific row where the question is actually asked. + +## Risks / Trade-offs + +- **A wrong column mapping silently imports garbage** (e.g. dates off by a + century, amounts inverted) → Undo via `import_id`, plus a preview step that + states the parsed date range and row count before commit, where an inverted or + misparsed column is obvious at a glance. + +- **The overlap region is wide, so duplicate review could have real volume** → + Skip-by-default means clicking through without reading yields the safe outcome + (synced data retained, no double-counting). The cost of inattention is a missing + photocopy, not corrupted reporting. + +- **A false positive silently drops a real transaction** (two genuinely distinct + identical charges on one day, one already synced) → This is the flip side of + skip-by-default and is the accepted trade. The wizard shows both rows side by + side specifically to make it catchable, and the import record retains the raw + file, so a missed row can be recovered by re-importing with a corrected + decision. + +- **`sfin_id` now holds values that are not SimpleFIN ids** → Namespaced `csv:` + prefix makes them greppable, and `import_id` is the authoritative origin field. + Accepted as a small, contained dishonesty rather than a codebase-wide rename. + +- **Abandoned drafts accumulate** → Low volume by nature; drafts are invisible to + every surface except the import record. Not worth a reaper. + +- **Two writers to `transactions` could drift apart** → The CSV writer is + insert-only and deliberately shares no code with `ingestTransactions` beyond the + row shape. The risk is a future schema change updating one and not the other; + mitigated by both paths being covered by tests over the same table. + +## Migration Plan + +One additive migration (`005_csv_import.sql`): + +- `CREATE TABLE imports` — `id`, `account_id` (FK, NOT NULL), `filename`, + `uploaded_at`, `payload` (verbatim, never mutated), `mapping` (JSON), `decisions` + (JSON), `status` CHECK in (`draft`, `committed`, `undone`), `committed_at`, + `undone_at`. +- `ALTER TABLE transactions ADD COLUMN import_id INTEGER REFERENCES imports (id)` + — nullable, NULL meaning synced, so every existing row is correct without a + backfill. +- Index on `transactions (import_id)` for undo and the source filter. + +No changes to existing columns and no data rewrite, so the migration is +forward-only and safe to re-run against a populated database. Rollback is dropping +the table and column; imported rows would need removal first, which the undo flow +already performs. + +## Open Questions + +- Should the preview step hard-refuse a file whose parsed range extends into the + future or predates the account's opening, as a cheap guard against a + catastrophically wrong date mapping — or is stating the range enough? +- Does the import record in Settings need to list the individual skipped rows, or + is a count sufficient? diff --git a/openspec/changes/csv-backfill-import/proposal.md b/openspec/changes/csv-backfill-import/proposal.md new file mode 100644 index 0000000..08b6af5 --- /dev/null +++ b/openspec/changes/csv-backfill-import/proposal.md @@ -0,0 +1,93 @@ +## Why + +One connected account only returns about a week of transaction history from +SimpleFIN, while another returns the full 90 days. The short-history account +therefore has a permanent hole in its past that no sync will ever fill — the +lookback window advances, it does not reach back. Reports that span more than a +week are silently wrong for that account, and no amount of waiting fixes it. + +The bank does publish that history as a CSV export. This change lets a user pour +that file into the account it belongs to, once, to close the gap. + +## What Changes + +- A CSV import flow, reached from Settings: upload a file, attribute it to one + existing account, confirm the column mapping, review potential duplicates, + commit. +- Imports are **additive only**. A CSV never reconciles pending transactions, + never marks accounts inactive, never removes rows, and never overwrites a + synced transaction. It inserts, or it does nothing. +- Imported transactions get a **synthetic stable id** derived from their content, + so re-importing the same file (or an overlapping range from a second file) is + a no-op rather than a duplication. +- **Duplicate review**: rows matching an existing transaction in the same account + by amount and near-identical date are surfaced for human judgment before + commit, defaulting to skip. Descriptions are shown side by side to inform the + decision but are not used to match, because the two sources render the same + merchant differently. +- **Undo**: every imported transaction records which import produced it, so a + botched mapping can be reversed as a unit. +- Raw uploaded bytes are archived verbatim before any normalization, alongside + the mapping and duplicate decisions, preserving the existing "normalization is + a replayable pure function of an archived payload" property. +- The ledger gains a **source filter** (synced / imported), and a transaction's + origin appears in its existing history panel. No new per-row ledger chrome: + within a backfilled range every row is imported, so a per-row badge would carry + no information where it appeared most. +- Rules run over imported transactions after commit, so backfilled history + categorizes itself against rules that already exist. + +Not in this change: creating accounts from a CSV, importing a category column +from a CSV, and saved per-account mapping profiles. See Impact. + +## Capabilities + +### New Capabilities +- `csv-import`: Uploading a CSV of historical transactions, archiving it + verbatim, mapping its columns, detecting and adjudicating potential duplicates + against existing data, additively ingesting the survivors into one existing + account, recording the import as a reversible unit, and undoing it. + +### Modified Capabilities +- `reporting`: The transaction ledger requirement gains a source filter (synced + vs. imported) that composes with the existing filters, and the ledger surfaces + a transaction's origin in its detail/history panel. + +## Impact + +**Affected specs**: new `csv-import`; modified `reporting` (transaction ledger +requirement). + +**Deliberately unaffected**: `account-management` requires that "the system SHALL +create account records only from sync data, never via in-app creation." This +change honors that rule rather than repealing it — a CSV must be attributed to an +account that sync already discovered. Consequently no CSV-only account exists, no +balance snapshots are invented, and the net-worth report is untouched. +`simplefin-sync` is likewise unchanged; its reconciliation and stale-sweep +behavior remain the exclusive authority of sync. + +**Affected code**: +- `migrations/` — new migration: a `raw_imports` archive table; a nullable + `import_id` FK on `transactions`. +- `src/lib/server/services/sync.ts` — `ingestTransactions` is *not* reused. It + treats its input as authoritative for the account (its stale-pending sweep would + soft-delete every pending row absent from a backfill CSV, and its ±5-day + reconciliation could let an old CSV row hijack a live pending row). CSV needs a + separate, insert-only writer. +- `src/lib/server/services/normalize.ts` — `parseAmountToCents` is reusable but + its regex rejects the parenthesized-negative and trailing-minus conventions + common in bank CSV exports; it needs a CSV-flavored pre-pass rather than a + change to its own contract. +- `src/lib/server/services/ledger.ts` — `LedgerFilters` gains `source`; + `LedgerRow` gains origin. +- `src/lib/server/services/rules.ts` — `applyRulesToUncategorized` is called + post-commit. It is already unscoped and database-wide, so it needs no change. +- New service for CSV parsing/mapping/duplicate detection; new Settings routes for + the import wizard and the import record. + +**Risk**: The overlap between a CSV's range and existing synced data is a region, +not a boundary — the short-history account accumulates synced rows daily, so both +sources can claim the same stretch of time. Duplicate review is therefore the +central safeguard, not an edge case. Where both sources claim a transaction, the +synced row wins: it carries a real SimpleFIN id, reconciles correctly under future +syncs, and already holds categorization history. diff --git a/openspec/changes/csv-backfill-import/specs/csv-import/spec.md b/openspec/changes/csv-backfill-import/specs/csv-import/spec.md new file mode 100644 index 0000000..484d04d --- /dev/null +++ b/openspec/changes/csv-backfill-import/specs/csv-import/spec.md @@ -0,0 +1,236 @@ +## ADDED Requirements + +### Requirement: Import attribution to an existing account + +The system SHALL require every CSV import to be attributed to exactly one account +that already exists in the database. The system SHALL NOT create, rename, or +otherwise modify an account as a result of an import, preserving the +`account-management` rule that accounts originate only from sync data. Accounts in +state `HIDDEN` SHALL NOT be offered as import targets. + +#### Scenario: User selects a target account + +- **WHEN** a user begins a CSV import +- **THEN** the system requires them to choose one existing non-hidden account, and + every transaction ingested from that file is attached to that account + +#### Scenario: No account selected + +- **WHEN** a user attempts to proceed without choosing a target account +- **THEN** the import does not proceed and no rows are ingested + +### Requirement: Verbatim archival before normalization + +The system SHALL store the uploaded file's bytes verbatim in an `imports` record +before any parsing, mapping, or normalization occurs. The archived payload SHALL +never be mutated or deleted by the application. The system SHALL store the chosen +column mapping and the duplicate decisions on the same record, so that the +outcome of an import is a deterministic function of the archived record alone. + +#### Scenario: File archived on upload + +- **WHEN** a user uploads a CSV file +- **THEN** an `imports` row is committed containing the exact uploaded bytes with + status `draft`, before any row is parsed + +#### Scenario: Import is replayable + +- **WHEN** the import logic is re-run over an archived record with its stored + mapping and decisions +- **THEN** it produces the same set of transactions as the original run + +#### Scenario: Unparseable file + +- **WHEN** an uploaded file cannot be parsed as CSV +- **THEN** the archived record is retained, the failure is stated plainly with the + reason, and no transactions are ingested + +### Requirement: Column mapping + +The system SHALL detect a CSV's date, amount, and description columns from its +header row where possible, and SHALL require the user to confirm or correct the +mapping before commit. The system SHALL support amounts expressed as a single +signed column and as separate debit and credit columns. The system SHALL accept +the negative conventions common to bank exports, including parenthesized values +and trailing minus signs, and SHALL convert amounts to integer cents without +floating-point arithmetic. The system SHALL map the file's date to the +transaction's posted timestamp and record imported transactions as not pending; +when the file exposes a distinct transaction date, the system SHALL map it to the +transaction date. + +#### Scenario: Headers auto-detected + +- **WHEN** a CSV's header row contains recognizable date, amount, and description + columns +- **THEN** the system pre-selects them and presents the mapping for confirmation + +#### Scenario: User corrects a wrong guess + +- **WHEN** the auto-detected mapping is wrong +- **THEN** the user can reassign any field to any column before committing + +#### Scenario: Parenthesized negative + +- **WHEN** an amount column contains `(12.34)` +- **THEN** it is ingested as `-1234` integer cents + +#### Scenario: Separate debit and credit columns + +- **WHEN** a file expresses amounts as separate debit and credit columns +- **THEN** the user can map both, and each row resolves to a single signed integer + cent amount + +#### Scenario: Imported rows appear in reports + +- **WHEN** an imported transaction is committed +- **THEN** it is recorded as not pending with its posted timestamp set, and + appears in monthly reports for the month of that timestamp + +### Requirement: Stable synthetic identity + +The system SHALL derive a stable, deterministic identifier for each imported +transaction from its content — the target account, date, amount, and description — +combined with an occurrence index distinguishing rows that are otherwise +identical within the same file. Identifiers SHALL be namespaced so they are +distinguishable from provider-supplied identifiers. Importing a file whose rows +have already been ingested under the same identifiers SHALL NOT create duplicate +rows. + +#### Scenario: Same file imported twice + +- **WHEN** a user imports a file and then imports the identical file again +- **THEN** no new transactions are created and the second import reports that + every row already exists + +#### Scenario: Overlapping files + +- **WHEN** a user imports a January–March file and then a February–April file into + the same account +- **THEN** the February–March rows are recognized as already ingested and only the + April rows are added + +#### Scenario: Genuinely identical transactions + +- **WHEN** a file contains two rows with the same date, amount, and description, + representing two real transactions +- **THEN** both are ingested as separate transactions + +### Requirement: Potential duplicate review + +The system SHALL, before commit, identify each candidate row that would create a +transaction resembling one the target account already holds — matching on +identical amount and an effective date within one day, against non-removed +transactions regardless of their origin. The system SHALL NOT match on +description, because the same transaction is rendered differently by different +sources. The system SHALL present each flagged pair to the user with both +descriptions shown, and SHALL default the flagged row to being skipped. The user +SHALL be able to override any flagged row to be imported. + +#### Scenario: Cross-source duplicate flagged + +- **WHEN** a candidate row has the same amount and date as an existing synced + transaction in the target account, but a differently worded description +- **THEN** the row is flagged for review, both descriptions are shown side by + side, and it defaults to skipped + +#### Scenario: Default skip retains the synced row + +- **WHEN** a user commits an import without changing any duplicate decision +- **THEN** every flagged row is skipped, the existing transactions are left + untouched, and no duplicate is created + +#### Scenario: User keeps a false positive + +- **WHEN** a flagged row is in fact a distinct transaction and the user marks it to + be imported +- **THEN** it is ingested as a new transaction alongside the existing one + +#### Scenario: Preview before commit + +- **WHEN** a mapped file is ready for review +- **THEN** the system states the parsed date range, the number of rows to be + imported, the number already present, and the number flagged as potential + duplicates + +### Requirement: Additive-only ingestion + +An import SHALL only insert transactions. The system SHALL NOT, as a result of an +import, modify or remove any existing transaction, reconcile pending transactions, +change any account's state, or write balance snapshots. Reconciliation and +feed-authority behavior SHALL remain exclusive to sync. + +#### Scenario: Pending transactions untouched + +- **WHEN** an import is committed into an account that holds pending transactions + absent from the CSV +- **THEN** those pending transactions remain unchanged and are not removed + +#### Scenario: Existing transaction not overwritten + +- **WHEN** a candidate row resolves to an identifier already present in the account +- **THEN** the existing transaction is left exactly as it was + +#### Scenario: No balance snapshots + +- **WHEN** an import is committed +- **THEN** no balance snapshot is written and the net worth report is unaffected + +#### Scenario: Account state unaffected + +- **WHEN** an import is committed into an `ACTIVE` account +- **THEN** the account's state and `last_successful_data_at` are unchanged + +### Requirement: Rule application after import + +The system SHALL apply existing categorization rules to imported transactions once +the import is committed, using the same rule precedence and the same `rule` +categorization event source as sync. Imported transactions SHALL NOT introduce a +new categorization event source. + +#### Scenario: Backfilled history categorizes itself + +- **WHEN** an import is committed and existing rules match some of the imported + transactions +- **THEN** those transactions are categorized with `rule` events recording the + winning rule, exactly as if they had arrived by sync + +#### Scenario: Manual decisions respected + +- **WHEN** rules are applied after an import +- **THEN** transactions whose latest categorization event is `manual` are not + re-categorized + +### Requirement: Import record and undo + +The system SHALL record every import and surface the record in Settings, showing +the file name, target account, parsed date range, counts of rows imported and +skipped, and the commit time. Each imported transaction SHALL record which import +produced it. The system SHALL allow a committed import to be undone as a unit, +removing the transactions it produced without deleting any categorization event +history. An undone import SHALL be re-importable after correcting its mapping. + +#### Scenario: Import listed in settings + +- **WHEN** a user opens the import record in Settings +- **THEN** each import is listed with its file name, account, date range, counts, + and commit time + +#### Scenario: Undo removes only that import's rows + +- **WHEN** a user undoes an import +- **THEN** exactly the transactions that import produced are removed from the + ledger and reports, no synced transaction is affected, and the import is marked + undone + +#### Scenario: Event history survives undo + +- **WHEN** an imported transaction had been categorized and its import is undone +- **THEN** the transaction no longer appears in the ledger or reports, and its + categorization events remain in the append-only log + +#### Scenario: Re-import after a corrected mapping + +- **WHEN** a user undoes an import made with a wrong mapping and re-imports the + same file with a corrected mapping +- **THEN** the corrected transactions are ingested and the undone import's rows do + not reappear diff --git a/openspec/changes/csv-backfill-import/specs/reporting/spec.md b/openspec/changes/csv-backfill-import/specs/reporting/spec.md new file mode 100644 index 0000000..3c36c22 --- /dev/null +++ b/openspec/changes/csv-backfill-import/specs/reporting/spec.md @@ -0,0 +1,32 @@ +## MODIFIED Requirements + +### Requirement: Transaction ledger +The system SHALL provide a ledger view of transactions filterable by account, category (including uncategorized), month, pending status, and source (synced vs. imported), and searchable by free text. Text search SHALL match case-insensitively against the raw description, payee, and memo fields and against the effective displayed name produced by rule display-name overlays, so search finds what the user sees. Search SHALL compose with all structured filters, and the source filter SHALL compose with all other filters. The ledger shows date, account, displayed name (rule overlay applied when present), amount, category, and a provenance indicator (rule vs. person). The ledger SHALL NOT mark a transaction's source on the row itself; a transaction's origin — synced from a connection, or imported from a named file — SHALL be shown in its history panel. The ledger is the surface for manual categorization and for creating rules from transactions. + +#### Scenario: Filter to uncategorized +- **WHEN** a user filters the ledger to uncategorized transactions +- **THEN** only transactions with no current category are listed, ready for manual assignment + +#### Scenario: Provenance indicator +- **WHEN** a categorized transaction is displayed +- **THEN** the row indicates whether the category came from a rule, a person (with their identity), or reconciliation carry-forward + +#### Scenario: Search matches a renamed transaction +- **WHEN** a rule renames "ACH TRANSFER 4417" to display as "Rent" and a user searches the ledger for "rent" +- **THEN** the transaction is found, even though no raw field contains "rent" + +#### Scenario: Search composes with filters +- **WHEN** a user searches for "netflix" with a month filter active +- **THEN** only that month's transactions matching the text (raw fields or displayed name) are listed + +#### Scenario: Filter to imported transactions +- **WHEN** a user filters the ledger by source to imported transactions +- **THEN** only transactions produced by a CSV import are listed + +#### Scenario: Source filter composes +- **WHEN** a user filters by source to imported with an account and month filter active +- **THEN** only that account's imported transactions in that month are listed + +#### Scenario: Origin shown in history +- **WHEN** a user opens the history panel of a transaction that came from a CSV import +- **THEN** the panel states that it was imported and names the file and import date, distinguishing it from a transaction synced from a connection diff --git a/openspec/changes/csv-backfill-import/tasks.md b/openspec/changes/csv-backfill-import/tasks.md new file mode 100644 index 0000000..fff8033 --- /dev/null +++ b/openspec/changes/csv-backfill-import/tasks.md @@ -0,0 +1,141 @@ +## 1. Schema + +- [ ] 1.1 Add `migrations/005_csv_import.sql` creating the `imports` table: `id`, + `account_id` (NOT NULL, FK to accounts), `filename`, `uploaded_at`, `payload` + (verbatim bytes, never mutated), `mapping` (JSON, nullable), `decisions` (JSON, + nullable), `status` NOT NULL DEFAULT `'draft'` CHECK in (`draft`, `committed`, + `undone`), `committed_at`, `undone_at`. +- [ ] 1.2 In the same migration, `ALTER TABLE transactions ADD COLUMN import_id + INTEGER REFERENCES imports (id)` (nullable; NULL means synced) and add an index + on `transactions (import_id)`. +- [ ] 1.3 Extend `src/lib/server/db.test.ts` to assert the migration applies to a + populated database and that existing transactions read back with + `import_id IS NULL`. + +## 2. CSV parsing and mapping (pure) + +- [ ] 2.1 Add `@std/csv` (JSR) as a dependency. Do not hand-roll parsing — + quoted delimiters, embedded newlines, and BOMs are the failure modes. +- [ ] 2.2 Create `src/lib/server/services/csv-import.ts` with a pure + `parseCsv(payloadText)` returning header row plus data rows, tolerating a BOM + and CRLF line endings. +- [ ] 2.3 Implement pure `detectMapping(headers)` returning best-guess column + assignments for date, amount (single signed column or debit/credit pair), and + description, plus optional payee/memo and a distinct transaction-date column. +- [ ] 2.4 Implement pure `normalizeCsvAmount(raw)` handling parenthesized + negatives `(12.34)`, trailing minus `12.34-`, currency symbols, and thousands + separators, emitting a signed decimal string for the existing + `parseAmountToCents`. Do not loosen `parseAmountToCents` itself — its strict + regex is the sync path's contract. +- [ ] 2.5 Implement pure `normalizeCsv(payloadText, mapping, accountId)` producing + candidate rows with `posted` set from the file's date, `pending = 0`, + `transacted_at` set only when the mapping names a distinct transaction-date + column, and amounts as integer cents. +- [ ] 2.6 Implement the synthetic identity: `csv:` + hash over (accountId, date, + amountCents, description) plus an occurrence index among identical rows within + the file. +- [ ] 2.7 Unit-test 2.2–2.6 in `csv-import.test.ts`: BOM/CRLF, both amount modes, + each negative convention, date parsing, and that two identical rows receive + distinct ids while a reordered or superset file re-derives the same id set. + +## 3. Duplicate detection + +- [ ] 3.1 Implement `findPotentialDuplicates(db, accountId, candidates)` matching + each candidate against non-removed transactions in the account by identical + `amount_cents` and effective date within ±1 day (compare against + `COALESCE(posted, transacted_at)`). Never match on description. +- [ ] 3.2 Return, per candidate, one of: `new`, `already-present` (its synthetic + id already exists — never surfaced to the user), or `flagged` with the existing + transaction's date, amount, and description for side-by-side display. +- [ ] 3.3 Test: a cross-source duplicate with differently worded descriptions is + flagged; a row one day off is flagged; a row two days off is not; an + already-ingested row reports `already-present` rather than `flagged`. + +## 4. Import lifecycle and additive ingestion + +- [ ] 4.1 Implement `createDraftImport(db, accountId, filename, payload)` writing + the `imports` row with verbatim payload and `status = 'draft'` before any + parsing. +- [ ] 4.2 Implement `saveMapping` and `saveDecisions` persisting to the draft row, + so the archived record alone determines the outcome. +- [ ] 4.3 Implement `commitImport(db, importId)` — an **insert-only** writer. + It MUST NOT reconcile, MUST NOT sweep stale pending rows, MUST NOT update + existing transactions, MUST NOT write balance snapshots, and MUST NOT touch + account state. Do not call or extend `ingestTransactions`; its stale-pending + sweep (`sync.ts:291-301`) would soft-delete every pending row absent from the + CSV. +- [ ] 4.4 In `commitImport`, skip candidates marked skipped and those whose + synthetic id already exists; revive a soft-removed row matching the unique key + by clearing `removed_at` and reassigning `import_id`; stamp `import_id` on every + inserted row; set `status = 'committed'` and `committed_at`. Run the whole + commit in one transaction. +- [ ] 4.5 Call `applyRulesToUncategorized(db)` after the commit transaction + succeeds. No new categorization event source. +- [ ] 4.6 Test: importing into an account holding pending rows leaves them + untouched; no snapshots are written; account state and + `last_successful_data_at` are unchanged; re-importing the same file inserts + nothing; an overlapping file inserts only the new rows; committed rows carry + `import_id` and are picked up by rules. + +## 5. Undo + +- [ ] 5.1 Implement `undoImport(db, importId)` soft-removing (`removed_at = now`) + every transaction with that `import_id`, setting `status = 'undone'` and + `undone_at`, in one transaction. Never hard-delete — categorization events + reference `transaction_id` and the event log is append-only. +- [ ] 5.2 Test: undo removes exactly that import's rows from ledger and report + scope, leaves synced rows untouched, preserves categorization events for the + removed rows, and a corrected re-import afterwards does not resurrect the old + rows. + +## 6. Ledger surfacing + +- [ ] 6.1 Add `source?: 'synced' | 'imported'` to `LedgerFilters` in + `src/lib/server/services/ledger.ts`, translating to `t.import_id IS NULL` / + `IS NOT NULL`, composing with every existing filter. +- [ ] 6.2 Add origin to `LedgerRow` (import id, file name, import date; null when + synced) via a join through `import_id`. +- [ ] 6.3 Test in `ledger.test.ts`: the source filter composes with account, + month, category, pending, and text search. +- [ ] 6.4 Add the source filter control to the ledger page alongside the existing + filters. Add no per-row badge — within a backfilled range every row is imported, + so a per-row mark carries no information where it appears. +- [ ] 6.5 Add an origin line to the top of the existing history panel: "Imported + from `` · ``" or "Synced from SimpleFIN". + +## 7. Import wizard (Settings) + +- [ ] 7.1 Add a Settings route for the import wizard. Step 1: file upload plus + target-account picker (non-hidden accounts only), archiving on submit and + redirecting to the draft's id. +- [ ] 7.2 Step 2: mapping confirmation, pre-filled from `detectMapping`, every + field reassignable, with a few parsed sample rows rendered so a wrong guess is + visible. +- [ ] 7.3 Step 3: preview stating the parsed date range, rows to import, rows + already present, and rows flagged — the guard against a catastrophically wrong + date mapping. +- [ ] 7.4 Step 4: duplicate review listing each flagged row against its existing + match, both descriptions shown, defaulting to skip, each flippable to keep. +- [ ] 7.5 Commit action, then a result summary linking to the ledger filtered to + that account and source. +- [ ] 7.6 Style per DESIGN.md: tabular numerals on money and dates, calm empty and + error states (one sentence plus one action), no new red unless data is at risk. + +## 8. Import record (Settings) + +- [ ] 8.1 Add a Settings section listing imports: file name, account, parsed date + range, rows imported, rows skipped, commit time, status. Not in the main nav. +- [ ] 8.2 Add the undo action with a confirmation stating exactly how many + transactions will be removed. +- [ ] 8.3 Render an empty state for the no-imports-yet case. + +## 9. Verification + +- [ ] 9.1 Run `deno task test` and `deno task check`. +- [ ] 9.2 Drive the wizard end to end against a real bank CSV export in dev: + confirm the gap closes in the ledger, the month reports for backfilled months + change as expected, and the net worth chart is unchanged (no snapshots written). +- [ ] 9.3 Verify a sync runs cleanly after an import: pending rows still + reconcile, and no imported row is disturbed. +- [ ] 9.4 Exercise undo on a real import and confirm the ledger and reports return + to their pre-import state. -- 2.51.2