diff --git a/AGENTS.md b/AGENTS.md index 9714ccb..b4ff143 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -15,8 +15,8 @@ array. Keep that framing in mind: every read path is record-by-record. - Rust 1.80+, edition 2021. Single binary, no workspace. - `cargo build` / `cargo build --release` (binary at `target/release/czsplicer`). -- `cargo test` — 175 passed, 1 ignored (2 suites; integration in `tests/integration.rs`, - unit in `src/mailbox.rs` + `src/mermaid.rs` + `src/csv.rs`), all synthetic. +- `cargo test` — 189 passed, 1 ignored (2 suites; integration in `tests/integration.rs`, + unit in `src/mailbox.rs` + `src/mermaid.rs` + `src/csv.rs` + `src/secrets.rs`), all synthetic. - `cargo fmt --check` is enforced. The pre-commit hook (`hooks/pre-commit`, enable with `git config core.hooksPath hooks`) runs `fmt --check` + `cargo test` when `.rs`/`.toml`/`tests/` files are staged. @@ -24,7 +24,7 @@ array. Keep that framing in mind: every read path is record-by-record. export data and must never be committed. Note: the README and architecture.md agree with the live `cargo test` count -(175 passed, 1 ignored, 2 suites). Keep them in sync when the count changes. +(189 passed, 1 ignored, 2 suites). Keep them in sync when the count changes. ## Repository layout @@ -34,6 +34,7 @@ src/ commands.rs one *Args (clap) struct + cmd_* fn per subcommand. ~2/3 of code. filter.rs Filter + FilterArgs, shared by all selection commands format.rs CBOR<->JSON bridge, RecordStream, ZstdPacker, redact/search, field accessors + redact.rs shared redaction preset table + compile_redact_regexes (consumed by edit/thread/report/secrets) thread.rs conversation-thread reconstruction (trie over message-content hashes) + RecordMeta render.rs shared helpers for the HTML renderers (escape_html, truncate, sender_color, best_record_id, ...) + clip_chars markdown.rs minimal safe Markdown->HTML subset for the built-in renderer @@ -51,7 +52,7 @@ vendor/ highlight.js (BSD-3-Clause) + CSS themes, embedded via inclu ``` No `mod.rs` under `src/`; `main.rs` declares -`mod builtin; mod commands; mod csv; mod filter; mod format; mod mailbox; mod markdown; mod md_thread; mod mermaid; mod render; mod theme; mod thread;`. +`mod builtin; mod commands; mod csv; mod filter; mod format; mod mailbox; mod markdown; mod md_thread; mod mermaid; mod redact; mod render; mod secrets; mod theme; mod thread;`. ## Architecture & data flow diff --git a/README.md b/README.md index bfc37ba..d72bd93 100644 --- a/README.md +++ b/README.md @@ -162,6 +162,13 @@ controls mbox/maildir body rendering: `plain`, `html` (multipart/alternative, default), `html-only`. Redaction runs on message bodies and tool text *before* rendering, so secrets never reach the output file. +> **Heuristic secrets warning.** When you emit HTML or Markdown *without* +> `--redact*`, czsplicer scans the output for common secret shapes (API keys, +> bearer tokens, JWTs, AWS keys, credit cards, SSNs) and prints a stderr +> warning listing the hits. It's a best-effort check, not a guarantee — custom +> token shapes aren't caught. Pass `--i-know` to suppress it, or re-run with +> `--redact-preset all` to scrub. + ### Failure analysis See when errors happen and which models are responsible. The default view shows a @@ -230,7 +237,7 @@ Directory arguments are expanded to their sorted `*.cbor.zstd` contents, so ## Development ```sh -cargo test # 175 passed, 1 ignored (2 suites; unit tests in mailbox/mermaid/csv) +cargo test # 189 passed, 1 ignored (2 suites; unit tests in mailbox/mermaid/csv/secrets) ``` The repository includes a pre-commit hook (`hooks/pre-commit`) that runs diff --git a/ROADMAP.md b/ROADMAP.md index 97d6ed3..1bc2089 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -83,19 +83,22 @@ content share the same Markdown document. section; branch points are noted inline and shared prefixes are not re-rendered. Mirrors `builtin.rs`. -### Secrets-safety net - -- When emitting any human-readable output (HTML/Markdown/mbox/EPUB) **without** - `--redact*`, run the canned preset regexes (`bearer`, `apikey`, `jwt`, …) as - a **detector** over the to-be-written bytes. -- On a hit: print a stderr warning naming the offending field/record pointer, - quoting the matched pattern (not the secret), and pointing at - `--redact-preset all`. Opt out with `--i-know`. -- **Detection only — never mutates output.** Reuses `compile_redact_regexes` + - the preset table. -- Warning wording must make clear it's a **best-effort heuristic**, not a - guarantee — custom token shapes (`x-ts-internal-…`) will not be caught. The - goal is to prevent the embarrassing day-one leak, not promise safety. +### Secrets-safety net (shipped) + +- When emitting HTML/Markdown output **without** `--redact*`, scan the rendered + bytes for likely-secret patterns and print a stderr warning. Opt out with + `--i-know`; silent when `--redact*` was applied (output already scrubbed). +- **Detection only — never mutates output.** Reads the final rendered bytes. +- **High-precision subset only** of `REDACT_PRESETS`: jwt, apikey, bearer, aws, + secretkey, creditcard, ssn. Deliberately **excludes email/ipv4/uuid** — those + appear routinely in legitimate metadata and would cry wolf. Patterns are + sourced from the single shared `REDACT_PRESETS` table (no duplication). +- **Scope (v1): HTML (builtin + themed) and Markdown.** mbox/maildir are *not* + scanned — their renderers stream to files/dirs without a single in-memory + buffer, and the warning is best-effort anyway. EPUB inherits the same + constraint when it lands. +- Warning wording explicitly calls it a **best-effort heuristic** — custom token + shapes are NOT caught — so it never creates false confidence. ### EPUB + Kindle (azw3) output — later release @@ -161,21 +164,26 @@ because the proprietary-compression half is already built in `~/src/huffcomp`. behavior. **Convention going forward: any PR that changes the command surface or test count updates the README in the same change.** -### Phase 2 — foundations (ship as standalone flags) +### Phase 2 — foundations (ship as standalone flags) — COMPLETE Each piece is individually useful and unblocks Phase 3. -2. **Mermaid emitters** — `pie`, `xychart-beta`, `timeline`, `flowchart`, +2. ✅ **Mermaid emitters** — `pie`, `xychart-beta`, `timeline`, `flowchart`, `sequenceDiagram`. Pure string emission over buckets `stats`/`failures` already compute. Add top-N/bucketing policy. Expose via `stats --format mermaid`, `failures --format mermaid`. -3. **Markdown thread renderer** — `thread --format md`. Trie walker → - **linear root-to-leaf path flattening** (the only scheme surviving - depth-749 trees; nested headings/lists cap at 6). Each path is one section; - branch points noted inline. User turns as blockquotes, tool calls fenced. -4. **CSV emitter** — `stats --format csv`, `failures --format csv`, `ls --csv`. - Tiny; tabular sibling to `--json`. -5. **Secrets-safety net** — detector on all human-readable output paths. +2. ✅ **Mermaid emitters** — `pie`, `xychart-beta`, `timeline` (the analysis + set). Pure string emission over buckets `stats`/`failures` already compute, + plus a top-N/bucketing policy. Expose via `stats --format mermaid`, + `failures --format mermaid`. (Structure diagrams `flowchart`/ + `sequenceDiagram` are deferred — see Phase 4.) +3. ✅ **Markdown thread renderer** — `thread --format md`. Trie walker → + linear path flattening (the only scheme surviving depth-749 trees; see + resolved item above). Headings + fenced tool blocks + user-as-blockquote. +4. ✅ **CSV emitter** — `stats --format csv`, `failures --format csv`, + `ls --format csv`. Tabular sibling to `--json`. +5. ✅ **Secrets-safety net** — detector on HTML/Markdown output paths + (see "Secrets-safety net (shipped)" above). ### Phase 3 — the centerpiece diff --git a/src/commands.rs b/src/commands.rs index fcdc066..0d7b664 100644 --- a/src/commands.rs +++ b/src/commands.rs @@ -5,6 +5,7 @@ use crate::format::{self, RecordStream}; use crate::mailbox; use crate::md_thread; use crate::mermaid; +use crate::secrets; use crate::theme; use crate::thread::{conversation_root, ThreadBuilder}; use anyhow::{anyhow, Result}; @@ -1444,88 +1445,10 @@ fn short_error(e: &str) -> String { } // --------------------------------------------------------------------------- -// redaction presets +// redaction (presets + helpers live in the `redact` module) // --------------------------------------------------------------------------- -/// Named canned regex patterns for `edit --redact-preset`. -pub const REDACT_PRESETS: &[(&str, &str, &str)] = &[ - ( - "email", - r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b", - "email addresses", - ), - ( - "jwt", - r"eyJ[A-Za-z0-9_-]+\.eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+", - "JSON Web Tokens", - ), - ( - "apikey", - r"sk-[A-Za-z0-9]{20,}|sk-ant-[A-Za-z0-9_-]{20,}|xai-[A-Za-z0-9]{20,}", - "API keys (OpenAI/Anthropic/xAI)", - ), - ("bearer", r"(?i:bearer\s+[A-Za-z0-9._-]+)", "Bearer tokens"), - ("aws", r"AKIA[0-9A-Z]{16}", "AWS access key IDs"), - ( - "secretkey", - r#"(?i)secret(?:\s+access)?\s+key\b[*`:='"\s]*[A-Za-z0-9/+=]{20,}"#, - "Labeled secret access keys (any 'Secret key:' / 'Secret access key:' block)", - ), - ("ipv4", r"\b(?:\d{1,3}\.){3}\d{1,3}\b", "IPv4 addresses"), - ( - "uuid", - r"\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\b", - "UUIDs", - ), - ( - "creditcard", - r"\b(?:\d[ -]?){13,16}\b", - "credit card numbers", - ), - ( - "ssn", - r"\b\d{3}-\d{2}-\d{4}\b", - "US Social Security numbers", - ), -]; - -/// Expand preset names into regex patterns. `all` expands to every preset. -pub fn expand_presets(names: &[String]) -> Result> { - let mut out = Vec::new(); - for name in names { - if name == "all" { - for (_, pat, _) in REDACT_PRESETS { - out.push((*pat).to_string()); - } - continue; - } - match REDACT_PRESETS.iter().find(|(n, _, _)| n == name) { - Some((_, pat, _)) => out.push(pat.to_string()), - None => { - let valid: Vec<&str> = REDACT_PRESETS.iter().map(|(n, _, _)| *n).collect(); - return Err(anyhow!( - "unknown redact preset `{name}`; valid: {}, all", - valid.join(", ") - )); - } - } - } - Ok(out) -} - -/// Merge explicit `--redact` patterns with expanded `--redact-preset` names, -/// drop empties (an empty regex would match everywhere and redact the whole -/// body), and compile each into a `regex::Regex`. Shared by `edit` and `thread`. -fn compile_redact_regexes(redact: &[String], presets: &[String]) -> Result> { - let mut all_patterns = redact.to_vec(); - all_patterns.extend(expand_presets(presets)?); - all_patterns.retain(|p| !p.is_empty()); - all_patterns - .iter() - .map(|p| regex::Regex::new(p)) - .collect::>() - .map_err(|e| anyhow!("invalid redact regex: {e}")) -} +use crate::redact::compile_redact_regexes; // --------------------------------------------------------------------------- // merge @@ -1806,6 +1729,12 @@ pub struct ThreadArgs { /// Replacement text for redacted spans (default `[REDACTED]`). #[arg(long, value_name = "TOKEN", default_value = "[REDACTED]")] pub redact_replacement: String, + /// Suppress the un-redacted-secrets warning for HTML/Markdown output. + /// The warning fires when output is emitted without `--redact*` and a + /// likely secret (API key, bearer token, JWT, …) is detected. This flag + /// acknowledges the risk; it does not disable detection for other paths. + #[arg(long = "i-know", default_value_t = false)] + pub i_know: bool, #[command(flatten)] pub filter: FilterArgs, } @@ -1824,6 +1753,23 @@ pub fn cmd_thread(args: &ThreadArgs) -> Result<()> { cur }; + // Secrets-safety net: when emitting human-readable output (HTML, Markdown) + // without `--redact*`, scan the rendered bytes for likely-secret patterns + // and warn on stderr. Skipped when redaction was applied (already scrubbed + // for valid-UTF-8 bodies) or when the user passed `--i-know`. mbox/maildir + // are not scanned here (their renderers stream to files/dirs without a + // single in-memory buffer); the warning is a best-effort heuristic anyway. + let warn_secrets = |bytes: &[u8]| { + if do_redact || args.i_know { + return; + } + if let Ok(text) = std::str::from_utf8(bytes) { + if let Some(w) = secrets::scan(text).warning() { + eprintln!("{w}"); + } + } + }; + let mut builder = ThreadBuilder::new(); let mut total = 0u64; let mut with_messages = 0u64; @@ -1844,6 +1790,7 @@ pub fn cmd_thread(args: &ThreadArgs) -> Result<()> { // Built-in long-form HTML renderer (no external theme bundle). if args.format == ThreadFormat::Html && args.theme.is_none() { let html = builtin::render_html(&j, args.dark)?; + warn_secrets(html.as_bytes()); write_output(args.output.as_ref(), html.as_bytes(), "html")?; eprintln!( "{}; built-in HTML{}", @@ -1856,6 +1803,7 @@ pub fn cmd_thread(args: &ThreadArgs) -> Result<()> { // HTML (themed) path: render the forest through an Adium message style. if let Some(theme_path) = &args.theme { let html = theme::render_forest(&j, theme_path, args.variant.as_deref())?; + warn_secrets(html.as_bytes()); write_output(args.output.as_ref(), html.as_bytes(), "html")?; eprintln!( "{}; themed -> {}", @@ -1904,6 +1852,7 @@ pub fn cmd_thread(args: &ThreadArgs) -> Result<()> { if args.format == ThreadFormat::Md { let md = md_thread::render_md(&j); + warn_secrets(md.as_bytes()); write_output(args.output.as_ref(), md.as_bytes(), "md")?; eprintln!("{}; markdown", thread_summary(total, with_messages, &j)); return Ok(()); diff --git a/src/main.rs b/src/main.rs index 872600f..b43acd3 100644 --- a/src/main.rs +++ b/src/main.rs @@ -7,7 +7,9 @@ mod mailbox; mod markdown; mod md_thread; mod mermaid; +mod redact; mod render; +mod secrets; mod theme; mod thread; diff --git a/src/redact.rs b/src/redact.rs new file mode 100644 index 0000000..0fe2fdd --- /dev/null +++ b/src/redact.rs @@ -0,0 +1,94 @@ +//! Shared redaction presets and regex compilation. +//! +//! Single source of truth for the canned secret-shape patterns used by: +//! - `edit --redact-preset` (mutating scrub of capture bodies / whole records), +//! - `thread`/`report` redaction (mutating scrub before rendering), +//! - `secrets` safety net (read-only detection over rendered bytes). +//! +//! All three consume the same `REDACT_PRESETS` table so a pattern added here is +//! immediately available to every path. The redaction invariant (this program +//! never silently mutates raw CBOR except on explicit `edit`/redact paths) is +//! preserved: the detector in `secrets.rs` reads only, and mutating callers +//! invoke `compile_redact_regexes` explicitly. + +use anyhow::{anyhow, Result}; + +/// Named canned regex patterns: `(name, regex, human description)`. +pub const REDACT_PRESETS: &[(&str, &str, &str)] = &[ + ( + "email", + r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b", + "email addresses", + ), + ( + "jwt", + r"eyJ[A-Za-z0-9_-]+\.eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+", + "JSON Web Tokens", + ), + ( + "apikey", + r"sk-[A-Za-z0-9]{20,}|sk-ant-[A-Za-z0-9_-]{20,}|xai-[A-Za-z0-9]{20,}", + "API keys (OpenAI/Anthropic/xAI)", + ), + ("bearer", r"(?i:bearer\s+[A-Za-z0-9._-]+)", "Bearer tokens"), + ("aws", r"AKIA[0-9A-Z]{16}", "AWS access key IDs"), + ( + "secretkey", + r#"(?i)secret(?:\s+access)?\s+key\b[*`:='"\s]*[A-Za-z0-9/+=]{20,}"#, + "Labeled secret access keys (any 'Secret key:' / 'Secret access key:' block)", + ), + ("ipv4", r"\b(?:\d{1,3}\.){3}\d{1,3}\b", "IPv4 addresses"), + ( + "uuid", + r"\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\b", + "UUIDs", + ), + ( + "creditcard", + r"\b(?:\d[ -]?){13,16}\b", + "credit card numbers", + ), + ( + "ssn", + r"\b\d{3}-\d{2}-\d{4}\b", + "US Social Security numbers", + ), +]; + +/// Expand preset names into regex patterns. `all` expands to every preset. +pub fn expand_presets(names: &[String]) -> Result> { + let mut out = Vec::new(); + for name in names { + if name == "all" { + for (_, pat, _) in REDACT_PRESETS { + out.push((*pat).to_string()); + } + continue; + } + match REDACT_PRESETS.iter().find(|(n, _, _)| n == name) { + Some((_, pat, _)) => out.push(pat.to_string()), + None => { + let valid: Vec<&str> = REDACT_PRESETS.iter().map(|(n, _, _)| *n).collect(); + return Err(anyhow!( + "unknown redact preset `{name}`; valid: {}, all", + valid.join(", ") + )); + } + } + } + Ok(out) +} + +/// Merge explicit `--redact` patterns with expanded `--redact-preset` names, +/// drop empties (an empty regex would match everywhere and redact the whole +/// body), and compile each into a `regex::Regex`. +pub fn compile_redact_regexes(redact: &[String], presets: &[String]) -> Result> { + let mut all_patterns = redact.to_vec(); + all_patterns.extend(expand_presets(presets)?); + all_patterns.retain(|p| !p.is_empty()); + all_patterns + .iter() + .map(|p| regex::Regex::new(p)) + .collect::>() + .map_err(|e| anyhow!("invalid redact regex: {e}")) +} diff --git a/src/secrets.rs b/src/secrets.rs new file mode 100644 index 0000000..b052171 --- /dev/null +++ b/src/secrets.rs @@ -0,0 +1,236 @@ +//! Secrets-safety net: a detection-only heuristic that warns when +//! human-readable output (HTML, Markdown) is emitted *without* `--redact*` +//! yet appears to contain likely secrets. +//! +//! This is NOT a guarantee — it runs the high-precision subset of +//! `REDACT_PRESETS` (the low-false-positive secret shapes: API keys, bearer +//! tokens, JWTs, AWS keys, credit cards, SSNs) as detectors over the rendered +//! bytes. Noisy patterns (email, IPv4, UUID) are deliberately excluded: they +//! appear routinely in legitimate metadata and would cry wolf. A custom token +//! shape the presets don't cover will sail through unflagged; the warning text +//! says so explicitly so it never creates false confidence. +//! +//! Detection never mutates output. The redaction invariant (see AGENTS.md, +//! "Critical invariants" §2) is preserved: this module reads only. + +use regex::Regex; +use std::sync::OnceLock; + +/// High-precision preset names treated as "likely secrets" for the warning. +/// Deliberately excludes email / ipv4 / uuid (routine in metadata → false +/// positives). Kept as names so the patterns stay sourced from +/// `REDACT_PRESETS` (single source of truth). +const LIKELY_SECRET_PRESETS: &[&str] = &[ + "jwt", + "apikey", + "bearer", + "aws", + "secretkey", + "creditcard", + "ssn", +]; + +/// Compiled (preset-name, regex) set, built once per process. +static SECRET_REGEXES: OnceLock> = OnceLock::new(); + +fn secret_regexes() -> &'static Vec<(&'static str, Regex)> { + SECRET_REGEXES.get_or_init(|| { + let mut out = Vec::new(); + for &name in LIKELY_SECRET_PRESETS { + if let Some((_, pat, _)) = crate::redact::REDACT_PRESETS + .iter() + .find(|(n, _, _)| *n == name) + { + if let Ok(re) = Regex::new(pat) { + out.push((name, re)); + } + } + } + out + }) +} + +/// Result of scanning rendered output for likely secrets. +#[derive(Default)] +pub struct Report { + /// (preset_name, hit_count) for each preset with ≥1 hit, in preset order. + pub hits: Vec<(&'static str, usize)>, +} + +impl Report { + pub fn total_hits(&self) -> usize { + self.hits.iter().map(|(_, n)| *n).sum() + } + + pub fn is_clean(&self) -> bool { + self.hits.is_empty() + } + + /// Comma-separated list of preset names that fired with counts, e.g. "bearer×1, apikey×2". + fn types(&self) -> String { + self.hits + .iter() + .map(|(name, n)| format!("{name}×{n}")) + .collect::>() + .join(", ") + } + + /// The stderr warning text (no trailing newline). Returns None if clean. + pub fn warning(&self) -> Option { + if self.is_clean() { + return None; + } + Some(format!( + "warning: output may contain un-redacted secrets; pattern-based check found {} likely hit(s) across {} type(s) ({}). \ + This is a best-effort heuristic — custom token shapes are NOT caught. \ + Re-run with `--redact-preset all` to scrub, or pass `--i-know` to suppress this warning.", + self.total_hits(), + self.hits.len(), + self.types() + )) + } +} + +/// Scan rendered output text for likely-secret patterns. Reads only; never +/// mutates. The `text` should be the final bytes about to be written. +pub fn scan(text: &str) -> Report { + let mut hits = Vec::new(); + for (name, re) in secret_regexes() { + // The creditcard preset matches any 13-16 digit run, which includes hex + // dumps, zero-runs, and UUID fragments. Require a valid Luhn checksum + // so the warning doesn't cry wolf on those (real card numbers are + // Luhn-valid); other presets are counted as-is. + let n = if *name == "creditcard" { + re.find_iter(text) + .filter(|m| luhn_ok(&m.as_str().replace([' ', '-'], ""))) + .count() + } else { + re.find_iter(text).count() + }; + if n > 0 { + hits.push((*name, n)); + } + } + Report { hits } +} + +/// Luhn (mod-10) checksum over a digit string. Returns true for valid card +/// numbers and false for the digit-only fragments the bare creditcard regex also +/// matches (e.g. "3030303030303030", "1111111111111111"). +fn luhn_ok(digits: &str) -> bool { + let mut sum = 0u32; + let mut dbl = false; + for b in digits.bytes().rev() { + if !b.is_ascii_digit() { + return false; + } + let mut d = (b - b'0') as u32; + if dbl { + d *= 2; + if d > 9 { + d -= 9; + } + } + sum += d; + dbl = !dbl; + } + !digits.is_empty() && sum % 10 == 0 +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn clean_text_reports_nothing() { + let r = scan("just a normal conversation about rust and cookies"); + assert!(r.is_clean()); + assert!(r.warning().is_none()); + } + + #[test] + fn detects_openai_api_key() { + let r = scan("Authorization: sk-abcdefghijklmnopqrstuvwxyz1234567890"); + assert!(r.hits.iter().any(|(n, _)| *n == "apikey")); + assert!(r.warning().is_some()); + } + + #[test] + fn detects_bearer_token() { + let r = scan("bearer eyJhbGc.somepayload.sig"); + assert!(r.hits.iter().any(|(n, _)| *n == "bearer")); + } + + #[test] + fn detects_jwt() { + let r = scan("eyJhbGci.eyJzdWIi.e30signature"); + assert!(r.hits.iter().any(|(n, _)| *n == "jwt")); + } + + #[test] + fn detects_aws_key() { + let r = scan("role arn with AKIAIOSFODNN7EXAMPLE key"); + assert!(r.hits.iter().any(|(n, _)| *n == "aws")); + } + + #[test] + fn detects_ssn() { + let r = scan("ssn on file: 123-45-6789"); + assert!(r.hits.iter().any(|(n, _)| *n == "ssn")); + } + + #[test] + fn ignores_routine_metadata() { + // Email, IPv4, UUID appear in normal logs and must NOT trigger the + // secret warning (they're excluded from LIKELY_SECRET_PRESETS). + let r = + scan("from user@example.com at 192.168.1.1 (id 550e8400-e29b-41d4-a716-446655440000)"); + assert!( + r.is_clean(), + "routine metadata should not warn: {:?}", + r.hits + ); + } + + #[test] + fn counts_multiple_hits() { + let r = scan( + "sk-abcdefghijklmnopqrstuvwxyz1234567890 and sk-abcdefghijklmnopqrstuvwxyz0987654321", + ); + assert_eq!(r.total_hits(), 2); + } + + #[test] + fn warning_lists_types_and_counts() { + let r = scan("sk-abcdefghijklmnopqrstuvwxyz1234567890"); + let w = r.warning().unwrap(); + assert!( + w.contains("apikey×1"), + "warning should name type+count: {w}" + ); + assert!( + w.contains("best-effort heuristic"), + "warning must disclaim: {w}" + ); + assert!(w.contains("--i-know"), "warning must mention opt-out: {w}"); + } + + #[test] + fn creditcard_requires_luhn_checksum() { + // Hex/UUID fragments match the bare 13-16 digit regex but fail Luhn: + // they must NOT trigger the warning. + let r = scan("card: 3030 3030 3030 3030"); + assert!( + !r.hits.iter().any(|(n, _)| *n == "creditcard"), + "non-Luhn digit run should not warn: {:?}", + r.hits + ); + // A Luhn-valid test card number should be flagged. + let r2 = scan("card: 4111 1111 1111 1111"); + assert!( + r2.hits.iter().any(|(n, _)| *n == "creditcard"), + "Luhn-valid card should warn: {:?}", + r2.hits + ); + } +} diff --git a/tests/integration.rs b/tests/integration.rs index bd3318e..abb6f8d 100644 --- a/tests/integration.rs +++ b/tests/integration.rs @@ -3333,6 +3333,140 @@ fn thread_md_writes_to_file_with_minus_o() { ); } +// =========================================================================== +// thread secrets-safety net (HTML / Markdown warning) +// =========================================================================== + +/// NDJSON with an OpenAI-style API key embedded in a user message. +const SECRET_NDJSON: &str = "{\"id\":1,\"timestamp\":\"2026-06-20T08:00:00Z\",\"model\":\"m\",\"path\":\"/v1/messages\",\"status_code\":200,\"capture\":{\"requestBody\":\"{\\\"messages\\\":[{\\\"role\\\":\\\"system\\\",\\\"content\\\":\\\"S\\\"},{\\\"role\\\":\\\"user\\\",\\\"content\\\":\\\"my key is sk-abcdefghijklmnopqrstuvwxyz1234567890\\\"}],\\\"model\\\":\\\"m\\\"}\"}}"; + +#[test] +fn secrets_warning_fires_on_html_with_secret() { + let f = Fixture::from_ndjson(SECRET_NDJSON); + let out = f + .cmd() + .arg("thread") + .arg(&f.cbor_zstd) + .arg("--format") + .arg("html") + .arg("-o") + .arg(f.dir.join("o.html")) + .assert() + .success() + .get_output() + .stderr + .clone(); + let stderr = String::from_utf8(out).unwrap(); + assert!( + stderr.contains("un-redacted secrets"), + "should warn: {stderr}" + ); + assert!(stderr.contains("apikey"), "should name the type: {stderr}"); + assert!( + stderr.contains("--i-know"), + "should mention opt-out: {stderr}" + ); +} + +#[test] +fn secrets_warning_fires_on_markdown_with_secret() { + let f = Fixture::from_ndjson(SECRET_NDJSON); + let out = f + .cmd() + .arg("thread") + .arg(&f.cbor_zstd) + .arg("--format") + .arg("md") + .arg("-o") + .arg(f.dir.join("o.md")) + .assert() + .success() + .get_output() + .stderr + .clone(); + let stderr = String::from_utf8(out).unwrap(); + assert!( + stderr.contains("un-redacted secrets"), + "md should warn: {stderr}" + ); +} + +#[test] +fn secrets_warning_suppressed_by_i_know() { + let f = Fixture::from_ndjson(SECRET_NDJSON); + let out = f + .cmd() + .arg("thread") + .arg(&f.cbor_zstd) + .arg("--format") + .arg("html") + .arg("--i-know") + .arg("-o") + .arg(f.dir.join("o.html")) + .assert() + .success() + .get_output() + .stderr + .clone(); + let stderr = String::from_utf8(out).unwrap(); + assert!( + !stderr.contains("un-redacted secrets"), + "--i-know should suppress: {stderr}" + ); +} + +#[test] +fn secrets_warning_silent_when_redacted() { + // With --redact-preset apikey the key is scrubbed before rendering, so the + // detector finds nothing and stays silent. + let f = Fixture::from_ndjson(SECRET_NDJSON); + let out = f + .cmd() + .arg("thread") + .arg(&f.cbor_zstd) + .arg("--format") + .arg("html") + .arg("--redact-preset") + .arg("apikey") + .arg("-o") + .arg(f.dir.join("o.html")) + .assert() + .success() + .get_output() + .stderr + .clone(); + let stderr = String::from_utf8(out).unwrap(); + assert!( + !stderr.contains("un-redacted secrets"), + "redacted output should not warn: {stderr}" + ); +} + +#[test] +fn secrets_warning_silent_on_clean_input() { + // No secret in the conversation → no warning. + let nd = "{\"id\":1,\"timestamp\":\"2026-06-20T08:00:00Z\",\"model\":\"m\",\"path\":\"/v1/messages\",\"status_code\":200,\"capture\":{\"requestBody\":\"{\\\"messages\\\":[{\\\"role\\\":\\\"user\\\",\\\"content\\\":\\\"just a normal chat\\\"}],\\\"model\\\":\\\"m\\\"}\"}}"; + let f = Fixture::from_ndjson(nd); + let out = f + .cmd() + .arg("thread") + .arg(&f.cbor_zstd) + .arg("--format") + .arg("html") + .arg("-o") + .arg(f.dir.join("o.html")) + .assert() + .success() + .get_output() + .stderr + .clone(); + let stderr = String::from_utf8(out).unwrap(); + assert!( + !stderr.contains("un-redacted secrets"), + "clean input should not warn: {stderr}" + ); +} + // =========================================================================== // tree --html (built-in long-form renderer) // ===========================================================================