diff --git a/AGENTS.md b/AGENTS.md index 4b66cc9..8a9c945 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -15,8 +15,8 @@ array. Keep that framing in mind: every read path is record-by-record. - Rust 1.80+, edition 2021. Single binary, no workspace. - `cargo build` / `cargo build --release` (binary at `target/release/czsplicer`). -- `cargo test` — 158 tests total (146 integration in `tests/integration.rs` + - 12 unit in `src/mailbox.rs` + `src/mermaid.rs`), all synthetic. 1 ignored. +- `cargo test` — 164 tests total (146 integration in `tests/integration.rs` + + 18 unit across `src/mailbox.rs` + `src/mermaid.rs`), all synthetic. 1 ignored. - `cargo fmt --check` is enforced. The pre-commit hook (`hooks/pre-commit`, enable with `git config core.hooksPath hooks`) runs `fmt --check` + `cargo test` when `.rs`/`.toml`/`tests/` files are staged. @@ -24,7 +24,7 @@ array. Keep that framing in mind: every read path is record-by-record. export data and must never be committed. Note: the README and architecture.md agree with the live `cargo test` count -(158 passed, 1 ignored, 2 suites). Keep them in sync when the count changes. +(164 passed, 1 ignored, 2 suites). Keep them in sync when the count changes. ## Repository layout @@ -38,6 +38,7 @@ src/ render.rs shared helpers for the HTML renderers (escape_html, truncate, sender_color, best_record_id, ...) + clip_chars markdown.rs minimal safe Markdown->HTML subset for the built-in renderer mermaid.rs Mermaid diagram emitters (pie/xychart/timeline) for `stats --format mermaid` / `failures --format mermaid` + md_thread.rs Markdown thread renderer (`thread --format md`) — linear path flattening, mirrors builtin.rs builtin.rs built-in long-form HTML renderer (wide column, markdown, status/tool chips) theme.rs Adium .AdiumMessageStyle loader + renderer (--theme, optional) mailbox.rs mbox/Maildir export (RFC822 + threading via Message-ID/In-Reply-To) @@ -50,7 +51,7 @@ vendor/ highlight.js (BSD-3-Clause) + CSS themes, embedded via inclu ``` No `mod.rs` under `src/`; `main.rs` declares -`mod builtin; mod commands; mod filter; mod format; mod mailbox; mod markdown; mod mermaid; mod render; mod theme; mod thread;`. +`mod builtin; mod commands; mod filter; mod format; mod mailbox; mod markdown; mod md_thread; mod mermaid; mod render; mod theme; mod thread;`. ## Architecture & data flow diff --git a/README.md b/README.md index 4fc8bab..6d709ca 100644 --- a/README.md +++ b/README.md @@ -147,11 +147,14 @@ czsplicer thread prod/ --format mbox -o threads.mbox # Maildir (one file per message) with plain-text bodies instead of HTML. czsplicer thread prod/ --format maildir --body plain -o maildir/ +# Markdown (one document, pasteable into GitHub/Notion/wikis; branch-safe). +czsplicer thread prod/ --format md -o threads.md + # Redact secrets in the rendered output (same presets as `edit`). czsplicer thread prod/ --format html --redact-preset all -o threads.html ``` -Formats: `json` (default), `html` (built-in), `mbox`, `maildir`. `--body` +Formats: `json` (default), `html` (built-in), `mbox`, `maildir`, `md`. `--body` controls mbox/maildir body rendering: `plain`, `html` (multipart/alternative, default), `html-only`. Redaction runs on message bodies and tool text *before* rendering, so secrets never reach the output file. @@ -224,7 +227,7 @@ Directory arguments are expanded to their sorted `*.cbor.zstd` contents, so ## Development ```sh -cargo test # 158 tests (146 integration + 12 unit, synthetic fixtures) +cargo test # 164 tests (146 integration + 18 unit, synthetic fixtures) ``` The repository includes a pre-commit hook (`hooks/pre-commit`) that runs diff --git a/ROADMAP.md b/ROADMAP.md index 1c6c8dd..97d6ed3 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -19,7 +19,10 @@ taken and the phased plan for implementation. Items below this line are ## The two graph problems (kept distinct) 1. **Structure graphs** — the conversation trie (branches, turns, tool calls). - Solved by Mermaid flowchart + sequenceDiagram. + Rendered as **flattened root-to-leaf path prose** (the only scheme that + survives depth-749 trees — see the resolved item below). Mermaid + `flowchart`/`sequenceDiagram` renderings of structure are deferred (they cap + out well below real conversation depth). 2. **Analysis graphs** — aggregates over records (cost/day, tokens by model, status by provider, error rate by hour). Computed by `stats`/`failures`, rendered by Mermaid pie/xychart/timeline + emitted as CSV. @@ -37,11 +40,13 @@ content share the same Markdown document. - **CSV/TSV** is added for the spreadsheet / own-tool audience. Tabular commands (`stats`, `failures`, `ls`) gain `--format csv`. JSON stays; CSV is a new sibling, not a replacement. -- **Mermaid diagram types to support:** - - Structure: `flowchart` (conversation tree outline with branch points), - `sequenceDiagram` (turn-by-turn request→response→tool). - - Analysis: `pie` (model / status share), `xychart-beta` (cost or tokens over - time), `timeline` (incident bursts). +- **Mermaid diagram types:** + - Analysis (v0.4): `pie` (model / status share), `xychart-beta` (cost or + tokens over time), `timeline` (incident bursts). + - Structure (deferred to Phase 4): `flowchart` (conversation tree outline + with branch points), `sequenceDiagram` (turn-by-turn + request→response→tool). Both cap out well below real conversation depth, so + v0.4 renders structure as flattened prose instead. - *Not* doing Mermaid `gitGraph` for v1 — see "deferred." - **Aggregation policy for scale:** top-N + "other" collapse for pies; per-day or per-hour bucketing for xychart. Naive full-resolution Mermaid chokes past @@ -59,7 +64,7 @@ content share the same Markdown document. ## Failures — Mermaid timeline (incident bursts) + status breakdown ## Conversations ### - + ``` - The underlying Mermaid emitters, Markdown thread renderer, and CSV emitter @@ -68,13 +73,15 @@ content share the same Markdown document. ### Markdown thread renderer -- Headings per turn; fenced code blocks for tool-call bodies; branch points - as nested sections with anchors. +- Headings per turn; fenced code blocks for tool-call bodies. - Reuses the existing trie walker (`thread::all_paths`); conceptually the reverse flavor of the existing `markdown.rs` md→HTML parser. -- *[open]* Branch representation within Markdown: linear-with-anchors vs. - per-conversation "## Branches" outline at top with anchor links. Pick before - implementing. +- *[resolved 2026-07-01]* **Linear root-to-leaf path flattening.** Real + `days/` data contains depth-749 trees and 20-way branches; Markdown headings + cap at 6 and nested lists get unreadable past ~6, so headings-per-turn / + nested-section / anchor schemes are impossible. Each root-to-leaf path is one + section; branch points are noted inline and shared prefixes are not + re-rendered. Mirrors `builtin.rs`. ### Secrets-safety net @@ -162,9 +169,10 @@ Each piece is individually useful and unblocks Phase 3. `sequenceDiagram`. Pure string emission over buckets `stats`/`failures` already compute. Add top-N/bucketing policy. Expose via `stats --format mermaid`, `failures --format mermaid`. -3. **Markdown thread renderer** — `thread --format md`. Trie walker → headings - + fenced tool blocks + branch anchors. Resolve the [open] branch - representation first. +3. **Markdown thread renderer** — `thread --format md`. Trie walker → + **linear root-to-leaf path flattening** (the only scheme surviving + depth-749 trees; nested headings/lists cap at 6). Each path is one section; + branch points noted inline. User turns as blockquotes, tool calls fenced. 4. **CSV emitter** — `stats --format csv`, `failures --format csv`, `ls --csv`. Tiny; tabular sibling to `--json`. 5. **Secrets-safety net** — detector on all human-readable output paths. @@ -172,8 +180,8 @@ Each piece is individually useful and unblocks Phase 3. ### Phase 3 — the centerpiece 6. **`report` command** — composes summary + stats-mermaid + failures-mermaid + - per-thread (flowchart + prose) into one `.md`. This is the OSS-launch - headline artifact. + per-thread flattened prose into one `.md`. This is the OSS-launch headline + artifact. 7. **HTML report (stretch)** — same composition, rendered: vendored mermaid.js + existing builtin thread HTML. Opt-in, not the default. @@ -186,8 +194,14 @@ Each piece is individually useful and unblocks Phase 3. ## Open questions to resolve before each phase -- *Phase 2, item 3:* Markdown branch representation (linear-with-anchors vs. - top-of-conversation outline). +- *Phase 2, item 3:* **RESOLVED 2026-07-01 against real `days/` data.** A + single day (2026-06-21) contains a tree of depth **749** (752 nodes) and + 20/21 trees have branches. Markdown headings cap at 6 levels and indented + lists get unreadable past ~6, so **headings-per-turn / nested-list schemes + are impossible.** The renderer mirrors `builtin.rs`: each root-to-leaf path + (`thread::all_paths`) becomes one linear section. Branch points are marked + inline and cross-referenced via anchors ("→ continues from path N at turn + M"); shared prefixes are noted, not re-rendered. - *Phase 2, item 2:* **RESOLVED 2026-07-01 against real `days/` data (17,484 records, $1,199, 25 models, ~110-day span):** - **Top-N = 8 + "other"** for model aggregation (top-8 ≈ 95% of cost; the diff --git a/src/commands.rs b/src/commands.rs index 6347413..56cd418 100644 --- a/src/commands.rs +++ b/src/commands.rs @@ -2,6 +2,7 @@ use crate::builtin; use crate::filter::{Filter, FilterArgs}; use crate::format::{self, RecordStream}; use crate::mailbox; +use crate::md_thread; use crate::mermaid; use crate::theme; use crate::thread::{conversation_root, ThreadBuilder}; @@ -1649,6 +1650,7 @@ pub enum ThreadFormat { Html, Mbox, Maildir, + Md, } #[derive(clap::Args)] @@ -1658,7 +1660,7 @@ pub struct ThreadArgs { /// Write JSON/HTML/MBOX/Maildir to this path instead of stdout (`-` for stdout). #[arg(short, long)] pub output: Option, - /// Output format: json (default), html (built-in renderer), mbox, maildir. + /// Output format: json (default), html (built-in renderer), mbox, maildir, md. #[arg(long, value_enum, default_value = "json")] pub format: ThreadFormat, /// Body rendering for mbox/maildir: plain, html (multipart/alternative), html-only. @@ -1781,6 +1783,13 @@ pub fn cmd_thread(args: &ThreadArgs) -> Result<()> { return Ok(()); } + if args.format == ThreadFormat::Md { + let md = md_thread::render_md(&j); + write_output(args.output.as_ref(), md.as_bytes(), "md")?; + eprintln!("{}; markdown", thread_summary(total, with_messages, &j)); + return Ok(()); + } + let pretty = serde_json::to_string_pretty(&j)?; let bytes = pretty.into_bytes(); write_output(args.output.as_ref(), &bytes, "json")?; diff --git a/src/main.rs b/src/main.rs index c627eca..69b7ad3 100644 --- a/src/main.rs +++ b/src/main.rs @@ -4,6 +4,7 @@ mod filter; mod format; mod mailbox; mod markdown; +mod md_thread; mod mermaid; mod render; mod theme; diff --git a/src/md_thread.rs b/src/md_thread.rs new file mode 100644 index 0000000..664a202 --- /dev/null +++ b/src/md_thread.rs @@ -0,0 +1,310 @@ +//! Markdown thread renderer (`thread --format md`). +//! +//! Mirrors `builtin.rs`'s linear path-flattening approach. Each root-to-leaf +//! path (`thread::all_paths`) becomes one section; branch points are marked +//! inline and shared prefixes are noted rather than re-rendered. This is the +//! only representation that survives arbitrary-depth trees — real captures +//! contain depth-749 conversations and 20-way branches, far beyond Markdown's +//! 6-level heading/list cap. + +use crate::render::{best_record_id, model_of, tool_event_records}; +use crate::thread::all_paths; +use serde_json::Value as Json; + +/// Render the full thread forest as a single Markdown document. +pub fn render_md(forest: &Json) -> String { + let mut out = String::new(); + out.push_str("# Conversation threads\n\n"); + out.push_str(&summary_block(forest)); + out.push('\n'); + + let records = forest.get("records").and_then(|r| r.as_object()); + let paths = all_paths(forest); + + // Group paths by their root node so each conversation tree is one section. + // `all_paths` walks trees in `forest["trees"]` order, so paths sharing a + // root are contiguous; group them by the first node's hash. + let mut path_idx = 0usize; + for group in group_paths_by_root(&paths) { + out.push_str(&render_tree_section(&group, records, path_idx)); + path_idx += group.len(); + out.push('\n'); + } + out +} + +fn summary_block(forest: &Json) -> String { + let total = forest + .get("records_total") + .and_then(|v| v.as_u64()) + .unwrap_or(0); + let with = forest + .get("records_with_messages") + .and_then(|v| v.as_u64()) + .unwrap_or(0); + let roots = forest + .get("root_count") + .and_then(|v| v.as_u64()) + .unwrap_or(0); + let branches = forest + .get("branch_count") + .and_then(|v| v.as_u64()) + .unwrap_or(0); + format!( + "> **{total} records** ({with} with messages) → **{roots} thread(s)**, **{branches} branch point(s)**.\n\n" + ) +} + +/// Group contiguous paths sharing the same root node hash. Each group is one +/// conversation tree (its paths are the distinct root-to-leaf walks). +fn group_paths_by_root<'a>(paths: &[Vec<&'a Json>]) -> Vec>> { + let mut groups: Vec>> = Vec::new(); + for path in paths { + let key = path + .first() + .and_then(|n| n.get("hash")) + .and_then(|h| h.as_str()) + .unwrap_or(""); + match groups.last_mut() { + Some(g) if !g.is_empty() => { + let prev_key = g[0] + .first() + .and_then(|n| n.get("hash")) + .and_then(|h| h.as_str()) + .unwrap_or(""); + if prev_key == key { + g.push(path.clone()); + continue; + } + } + _ => {} + } + groups.push(vec![path.clone()]); + } + groups +} + +fn render_tree_section( + group: &[Vec<&Json>], + records: Option<&serde_json::Map>, + start_idx: usize, +) -> String { + let mut out = String::new(); + let root = match group.first().and_then(|p| p.first()) { + Some(r) => r, + None => return out, + }; + + let label = root_label(root); + let npaths = group.len(); + out.push_str(&format!("## {label}\n\n")); + out.push_str(&format!( + "_depth {max_depth}, {npaths} path(s)_\n\n", + max_depth = max_depth_of(group), + )); + + for (i, path) in group.iter().enumerate() { + out.push_str(&render_path(path, records, start_idx + i + 1)); + out.push('\n'); + } + out +} + +fn max_depth_of(group: &[Vec<&Json>]) -> usize { + group.iter().map(|p| p.len()).max().unwrap_or(0) +} + +/// Derive a readable section label from the root node (its preview). +fn root_label(root: &Json) -> String { + let preview = root.get("preview").and_then(|v| v.as_str()).unwrap_or(""); + let single = preview.lines().next().unwrap_or("").trim(); + if single.is_empty() { + "(empty conversation)".into() + } else if single.chars().count() > 80 { + format!("{}…", single.chars().take(79).collect::()) + } else { + single.to_string() + } +} + +fn render_path( + path: &[&Json], + records: Option<&serde_json::Map>, + path_num: usize, +) -> String { + let mut out = String::new(); + out.push_str(&format!("### Path {path_num}\n\n")); + + for (i, node) in path.iter().enumerate() { + let role = node.get("role").and_then(|v| v.as_str()).unwrap_or(""); + + // Metadata anchor: record whose request first included this node's + // message. Mirrors builtin.rs (intro_rid of the next node, with a + // best_record_id fallback for the terminal node). + let meta_rid = if i + 1 < path.len() { + best_record_id(path[i + 1], records) + } else { + best_record_id(node, records) + }; + let meta_rec = meta_rid.and_then(|id| records.and_then(|rmap| rmap.get(&id.to_string()))); + + let (label, is_user) = match role { + "user" => ("you".to_string(), true), + "assistant" => ( + model_of(meta_rec).unwrap_or_else(|| "assistant".into()), + false, + ), + "system" => ("system".to_string(), false), + other => (other.to_string(), false), + }; + + let body = node + .get("content") + .and_then(|v| v.as_str()) + .or_else(|| node.get("preview").and_then(|v| v.as_str())) + .unwrap_or(""); + + let time = meta_rec + .and_then(|r| r.get("timestamp")) + .and_then(|v| v.as_str()) + .unwrap_or(""); + let status = meta_rec + .and_then(|r| r.get("status_code")) + .and_then(|v| v.as_u64()); + + out.push_str(&format!("**{label}**")); + if !time.is_empty() { + out.push_str(&format!(" · _{time}_")); + } + if let Some(code) = status { + out.push_str(&format!(" · `{code}`")); + } + out.push_str("\n\n"); + + // Body: user messages as blockquotes (preserve verbatim); system / + // assistant as markdown body (so rendered prose). The node content is + // already markdown for assistant turns. + if !body.is_empty() { + if is_user { + for line in body.lines() { + out.push_str("> "); + out.push_str(line); + out.push('\n'); + } + out.push('\n'); + } else { + out.push_str(body); + if !body.ends_with('\n') { + out.push('\n'); + } + out.push('\n'); + } + } + + // Tool calls / results for assistant turns. + if role == "assistant" { + let (call_rec, result_rec) = tool_event_records(node, records); + let tools = tool_events_md(call_rec, result_rec); + if !tools.is_empty() { + out.push_str(&tools); + out.push('\n'); + } + } + + out.push_str("---\n\n"); + } + out +} + +/// A Markdown code fence long enough that no run of backticks in `content` +/// can close it: one backtick longer than the longest backtick run (minimum 3, +/// the shortest legal fence). Per CommonMark a closing fence must be at least as +/// long as the opening fence, so a backtick run in the payload can't prematurely +/// end a block fenced with one more. +fn fence(content: &str) -> String { + let mut longest = 0usize; + let mut run = 0usize; + for b in content.bytes() { + if b == b'`' { + run += 1; + if run > longest { + longest = run; + } + } else { + run = 0; + } + } + "`".repeat((longest + 1).max(3)) +} + +/// Render tool events as fenced Markdown code blocks. Calls carry the tool +/// name (language hint) and input; results follow. Mirrors the HTML rendering +/// order: call-then-result. +fn tool_events_md(call_rec: Option<&Json>, result_rec: Option<&Json>) -> String { + let mut out = String::new(); + + if let Some(events) = call_rec + .and_then(|r| r.get("tool_events")) + .and_then(|v| v.as_array()) + { + for ev in events { + if ev.get("kind").and_then(|v| v.as_str()) == Some("call") { + let name = ev.get("name").and_then(|v| v.as_str()).unwrap_or("tool"); + let input = ev.get("input").and_then(|v| v.as_str()).unwrap_or(""); + out.push_str(&format!("**tool call: `{name}`**\n\n")); + let f = fence(input); + out.push_str(&format!("{f}json\n")); + out.push_str(input); + if !input.ends_with('\n') { + out.push('\n'); + } + out.push_str(&format!("{f}\n\n")); + } + } + } + + if let Some(events) = result_rec + .and_then(|r| r.get("tool_events")) + .and_then(|v| v.as_array()) + { + for ev in events { + if ev.get("kind").and_then(|v| v.as_str()) == Some("result") { + let content = ev.get("content").and_then(|v| v.as_str()).unwrap_or(""); + if !content.is_empty() { + out.push_str("**tool result**\n\n"); + let f = fence(content); + out.push_str(&format!("{f}\n")); + out.push_str(content); + if !content.ends_with('\n') { + out.push('\n'); + } + out.push_str(&format!("{f}\n\n")); + } + } + } + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn fence_is_at_least_three_backticks() { + assert_eq!(fence(""), "```"); + assert_eq!(fence("no backticks here"), "```"); + // Runs shorter than 3 still floor to a 3-backtick fence. + assert_eq!(fence("a ` b"), "```"); + assert_eq!(fence("a `` b"), "```"); + } + + #[test] + fn fence_one_longer_than_longest_run() { + // A 3-backtick run in the payload needs a 4-backtick fence to stay open. + assert_eq!(fence("```"), "````"); + assert_eq!(fence("text\n```\nmore"), "````"); + // A 5-backtick run needs a 6-backtick fence. + assert_eq!(fence("`````"), "``````"); + } +} diff --git a/tests/integration.rs b/tests/integration.rs index 5e4ff86..18c4fb8 100644 --- a/tests/integration.rs +++ b/tests/integration.rs @@ -3108,6 +3108,137 @@ fn theme_loads_lowercase_dirs_and_styles_base_css() { ); } +// =========================================================================== +// thread --format md (Markdown renderer) +// =========================================================================== + +/// Run `thread --format md` on an NDJSON corpus and return the Markdown output. +fn thread_md(ndjson: &str) -> String { + let f = Fixture::from_ndjson(ndjson); + let out = Command::cargo_bin("czsplicer") + .unwrap() + .arg("thread") + .arg(&f.cbor_zstd) + .arg("--format") + .arg("md") + .assert() + .success() + .get_output() + .stdout + .clone(); + String::from_utf8(out).expect("valid utf8 markdown") +} + +#[test] +fn thread_md_renders_summary_header() { + let nd = format!( + "{}\n", + rec(1, &body_with_messages("S", &[("user", "hello")])) + ); + let md = thread_md(&nd); + assert!( + md.starts_with("# Conversation threads"), + "missing title: {md}" + ); + assert!(md.contains("**1 records**"), "missing record count: {md}"); + assert!(md.contains("**1 thread(s)**"), "missing thread count: {md}"); +} + +#[test] +fn thread_md_user_messages_are_blockquotes() { + let nd = format!( + "{}\n", + rec(1, &body_with_messages("S", &[("user", "hello world")])) + ); + let md = thread_md(&nd); + // User content rendered as a blockquote. + assert!( + md.contains("> hello world"), + "user content not blockquote: {md}" + ); + // System content rendered as body prose (NOT a blockquote). + assert!(md.contains("S\n"), "system content missing: {md}"); + // User turn labelled "you". + assert!(md.contains("**you**"), "user not labelled 'you': {md}"); +} + +#[test] +fn thread_md_linear_chain_single_path() { + let nd = format!( + "{}\n{}\n", + rec(1, &body_with_messages("S", &[("user", "hello")])), + rec( + 2, + &body_with_messages("S", &[("user", "hello"), ("assistant", "hi")]) + ), + ); + let md = thread_md(&nd); + assert!(md.contains("### Path 1"), "missing path heading: {md}"); + // A linear chain yields exactly one path. + assert!( + !md.contains("### Path 2"), + "linear chain should have one path: {md}" + ); +} + +#[test] +fn thread_md_renders_each_branch_as_a_path() { + // Two records diverging at the first user message → two paths. + let nd = format!( + "{}\n{}\n", + rec(1, &body_with_messages("S", &[("user", "hello")])), + rec(2, &body_with_messages("S", &[("user", "goodbye")])) + ); + let md = thread_md(&nd); + assert!(md.contains("### Path 1"), "missing path 1: {md}"); + assert!(md.contains("### Path 2"), "missing path 2: {md}"); + // Both user messages should appear (one per path). + assert!(md.contains("> hello"), "missing branch A content: {md}"); + assert!(md.contains("> goodbye"), "missing branch B content: {md}"); +} + +#[test] +fn thread_md_section_title_from_root_preview() { + let nd = format!( + "{}\n", + rec( + 1, + &body_with_messages("SystemPrompt", &[("user", "How do I bake bread?")]) + ) + ); + let md = thread_md(&nd); + // Section title comes from the root preview (system prompt first line). + assert!( + md.contains("## SystemPrompt"), + "section title from root: {md}" + ); +} + +#[test] +fn thread_md_writes_to_file_with_minus_o() { + let nd = format!( + "{}\n", + rec(1, &body_with_messages("S", &[("user", "hello")])) + ); + let f = Fixture::from_ndjson(&nd); + let tmp = f.dir.join("out.md"); + Command::cargo_bin("czsplicer") + .unwrap() + .arg("thread") + .arg(&f.cbor_zstd) + .arg("--format") + .arg("md") + .arg("-o") + .arg(&tmp) + .assert() + .success(); + let written = std::fs::read_to_string(&tmp).unwrap(); + assert!( + written.contains("# Conversation threads"), + "file output: {written}" + ); +} + // =========================================================================== // tree --html (built-in long-form renderer) // ===========================================================================