Academic paper knowledge organizer
Something went wrong. Try again.
18 kB · 140 lines
JSON
at claude
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140{ "what_we_decided": [ "2026-01-06 - Primary framing: personal research library with knowledge graph, not just study synthesis tool", "2026-01-06 - Aggressive deduplication: preprints/postprints and unannotated/annotated treated as version updates", "2026-01-06 - Auto-process with flagging for manual review (maximize laziness/seamlessness)", "2026-01-06 - Will rely on OCR for scanned PDFs and book scans (including two-page spreads)", "2026-01-06 - Use GROBID for slow/comprehensive processing (~30s per paper, worth it for capabilities)", "2026-01-06 - Use OCRmyPDF + Unpaper for scanned PDFs and two-page spread handling", "2026-01-06 - Citation parsing: extract and store in MVP, defer retrieval to Phase 2", "2026-01-06 - MVP metadata: authors, year, title, venue, volume/pages, DOI, abstract, page count", "2026-01-06 - Development order: Start with arXiv papers (high quality, no annotations) as v0.1.0 MVP, expand to undergrad papers (mixed quality, annotations) in v0.1.5", "2026-01-06 - MVP synthesis: summaries + 'why did I care' context, flashcards deferred, connections fuzzy/deferred", "2026-01-06 - Acquisition sources: cast wide net including gray-market (user has legitimate access, just poor quality copies)", "2026-01-07 - Use Obsidian as primary output interface - fast generation, excellent search, graph view for connections", "2026-01-08 - Two-tier synthesis: quick preview (MVP) + detailed section-by-section summary (on-demand per paper)", "2026-01-08 - Link all citations in paper notes, defer author pages to post-MVP", "2026-01-08 - Use full author list for citation disambiguation (Smith, Jones & Chen 2023 vs Smith, Lee & Park 2023)", "2026-01-08 - Use uv for Python package management instead of pip/venv", "2026-01-08 - Target Python 3.14+ (latest features, no backwards compatibility needed)", "2026-01-08 - Use real GROBID calls for testing (not mocks) - easier integration debugging", "2026-01-10 - Citation and PaperMetadata should share a common BibliographicEntry base class - they're the same type of object (bibliographic references) with different use contexts", "2026-01-10 - Garbage detection should use soft suspicion modifiers for years, not hard boundaries - avoid breaking humanities citations or future cutting-edge work (2028+ thesis prep)", "2026-01-10 - Garbage detection caps should be raised when multiple signals from same category fire (e.g., 4 bio signals = 80 point cap instead of 60)", "2026-01-10 - Move from test_*.py scripts to proper pytest test suite (test_arxiv_fetcher.py, test_grobid_processor.py, test_synthesis_generator.py, test_markdown_writer.py) - pytest auto-discovery, better reporting, CI/CD ready, supports marking integration vs unit tests", "2026-01-10 - CLI entry point via pyproject.toml [project.scripts] instead of test_pipeline.py - cleaner, more discoverable, proper command-line tool" ], "what_we_built": [ "2026-01-08 - config.py - config management from env vars (ANTHROPIC_API_KEY, GROBID_URL, VAULT_PATH, derived paths)", "2026-01-08 - models.py - Pydantic data models (Citation, PaperMetadata, ArticleMetadata, Synthesis, ProcessingState)", "2026-01-08 - state.py - StateManager for processing_state.json tracking (is_processed, mark_processed, save/load)", "2026-01-08 - docker-compose.yml - GROBID 0.8.0 service setup (4G memory, pool_size=4)", "2026-01-08 - env.example - template for ANTHROPIC_API_KEY, GROBID_URL, VAULT_PATH configuration", "2026-01-08 - Package structure - paper_library/ with __init__, config, models, state; pyproject.toml for uv; README.md", "2026-01-08 - grobid_processor.py - Parse GROBID XML to PaperMetadata, extract title/authors/year/abstract/venue/citations with full XPath parsing", "2026-01-08 - synthesis_generator.py - Claude Haiku integration with structured prompts, XML parsing, token cost tracking", "2026-01-08 - markdown_writer.py - Obsidian formatter with YAML frontmatter, structured sections, wikilinks, filename generation", "2026-01-08 - arxiv_fetcher.py - Fetch papers from arXiv API, parse IDs (multiple formats), download PDFs, extract metadata from Atom XML", "2026-01-08 - orchestrator.py - Main pipeline coordination: fetch, GROBID, synthesis, markdown, state tracking", "2026-01-08 - test_grobid.py - Test script for GROBID processor with sample PDF, displays extracted metadata", "2026-01-08 - test_synthesis.py - Test script for Claude synthesis with sample Transformer paper data", "2026-01-08 - test_markdown.py - Test script for Obsidian note generation, creates sample_note.md", "2026-01-08 - test_arxiv.py - Test script for arXiv fetching, demonstrates ID parsing and PDF download", "2026-01-08 - test_pipeline.py - End-to-end integration test with pre-flight checks, processes complete paper to Obsidian", "2026-01-08 - README.md - Comprehensive repository documentation with quick start, architecture decisions, troubleshooting", "2026-01-08 - v0.1.0 MVP working end-to-end - successfully processed first paper (Attention Is All You Need) from arXiv to Obsidian", "2026-01-08 - v0.1.0 MVP formatting polish complete - clean titles, proper citations, casual summaries, improved fallback", "2026-01-08 - Title case conversion - smart handling of ALL-CAPS titles while preserving acronyms (BERT, GPT, etc)", "2026-01-08 - batch_process.py - Simple script to process papers from text file (one identifier per line)", "2026-01-08 - Filename sanitization - removes quotes, replaces periods (except decimals), smart truncation at boundaries, single ellipsis", "2026-01-08 - YAML quote escaping - properly escapes quotes in title field to prevent malformed frontmatter", "2026-01-09 - Citation filtering - skip citations >500 chars, <10 chars, or missing both authors and title (catches figure captions, bios)", "2026-01-10 - Biographical garbage detection - filters citations containing 'the person attended', 'graduated from', etc (catches author bios)", "2026-01-10 - Filename period handling - all periods become dashes (3.1 \u2192 3-1, U.S.A. \u2192 U-S-A-) for Windows compatibility", "2026-01-10 - Figure/table caption detection - filters citations with 'N=10000', 'bit / param', 'Figure 1', hyperparameter notation (k-D, -C1-, etc)", "2026-01-10 - Mathematical notation detection - filters citations with >10 equals signs or 3:1 digit-to-letter ratio (catches parameter lists)", "2026-01-10 - Citation garbage detection refactored to scoring-based heuristic (as grobid_processor._calculate_garbage_score)", "2026-01-10 - BibliographicEntry base class - shared by Citation and PaperMetadata (as models.py refactor)", "2026-01-10 - Citation extraction now captures venue/volume/issue/pages from GROBID (grobid_processor.py with new helper methods)", "2026-01-10 - Enhanced garbage detection: soft year boundaries, raised caps for multiple signals (bio=80 if 3+ signals), hyperparameter density tracking, venue trust bonus", "2026-01-10 - Citation formatting now displays venue/volume/issue/pages in Full Citation List (markdown_writer.py)", "2026-01-10 - Fixed Pydantic validation errors: title/authors/year now Optional in BibliographicEntry base, required again in PaperMetadata override", "2026-01-10 - Fixed markdown rendering: blank lines now properly separate all sections (headers render correctly in Obsidian)", "2026-01-10 - Fixed UTF-8 encoding: explicitly set response.encoding='utf-8' when receiving GROBID XML to prevent mojibake (\u00e9\u2192\u00c3\u00a9)", "2026-01-10 - Updated synthesis prompt to preserve quoted text and technical notation - don't convert '\"A is B\"' to <placeholders>", "2026-01-10 - Escape angle brackets in synthesis output - prevents markdown from treating <placeholders> as HTML tags and stripping them", "2026-01-10 - Fixed YAML frontmatter tags - append each tag as separate line to prevent malformed YAML (was joining into single string with embedded newlines)", "2026-01-10 - Bibliography removal heuristic in text extraction - detects common 'References' headers and truncates if found in latter 50% of paper", "2026-01-10 - Increased synthesis text cutoff from 50k to 100k chars (~25k tokens) - plenty of room for long papers after bibliography removal", "2026-01-10 - tests/test_arxiv_fetcher.py - comprehensive pytest tests for arXiv fetching (ID parsing, metadata extraction, PDF download)", "2026-01-10 - tests/test_grobid_processor.py, test_synthesis_generator.py, test_markdown_writer.py - full pytest test suites for all major components (670+ lines total)", "2026-01-10 - paper_library/cli.py - command-line interface with alcanzai ingest/batch/stats/validate commands - replaces test_pipeline.py as entry point", "2026-01-10 - .pre-commit-config.yaml - git pre-commit hooks configuration (black, ruff, trailing whitespace, YAML validation)", "2026-01-10 - .devcontainer/devcontainer.json + postCreateCommand.sh - complete isolated development environment (Python 3.11, Docker-in-Docker, GROBID + CouchDB ports, VS Code extensions pre-configured)", "2026-01-10 - DEVELOPER.md - comprehensive developer guide covering pytest, black/ruff, pre-commit hooks, devcontainer setup and workflow" ], "what_we_need_to_decide": [ "2026-01-08 - Obsidian output formatting - needs troubleshooting/verification", "2026-01-08 - Batch file format sophistication - markdown bullet handling, inline comments, multi-format support (Obsidian notes as input?)", "2026-01-08 - Synthesis caching/optimization - separate system prompt from per-paper instructions, enable chat-with-paper without regenerating base summary", "2026-01-08 - Deduplication strategy - skip re-synthesis if note exists, only regenerate for new content (section-by-section summaries, annotations)", "2026-01-10 - Should we add logging for borderline citations (score 40-60) to help tune the threshold, or is 60 good enough for now?" ], "important_files": [ "2026-01-08 - paper_library/__init__.py - Package exports and version", "2026-01-08 - paper_library/orchestrator.py - Main entry point for paper processing", "2026-01-08 - tests/test_pipeline.py - End-to-end integration test, start here for testing", "2026-01-08 - pyproject.toml - Package dependencies and configuration", "2026-01-08 - .env.example - Template for API keys and paths" ], "context": [ "2026-01-06 - Annotation sources: Android tablet PDF readers (variable apps), future Kindle Scribe", "2026-01-06 - Legacy materials scattered across formats from undergrad", "2026-01-06 - Will include scanned hardcopy materials and book chapters", "2026-01-06 - OCR is new area, needs deeper exploration", "2026-01-06 - Using Python script for state management instead of bash+jq", "2026-01-06 - Need to handle resolved questions in state tracking", "2026-01-06 - Two paper datasets available: (1) arXiv AI papers - high quality, no annotations yet, (2) undergrad papers - mixed quality, existing highlights, cross-discipline", "2026-01-06 - Has ~24+ arXiv AI/LLM papers ready for testing, mostly with arXiv IDs", "2026-01-08 - Using CouchDB for Obsidian sync on homelab server", "2026-01-08 - Vault structure designed to accommodate future lecture recordings and session notes", "2026-01-08 - Token cost not a concern - can err on side of sending full papers to Haiku", "2026-01-08 - Processing on homelab server (K3s), writes to vault on same server, Livesync handles mobile sync", "2026-01-08 - Code should include ELI5 comments for Python learning opportunity", "2026-01-08 - Two-tier synthesis: MVP is quick-preview only (3-4 sent), detailed per-section summary deferred to v0.2 - no on-demand trigger needed for MVP", "2026-01-08 - Web content in MVP is confirmed (not v0.2): Unified 'Clear Your Tabs' workflow handles arXiv+DOI+web articles+clean PDFs all together", "2026-01-08 - Citation wikilink format: [[Author, Author & Author (Year) - Title]] with full author list for disambiguation - links are 'future notes', no cross-linking until papers processed", "2026-01-08 - File naming: 'FirstAuthor et al (Year) - Title' (80 char limit). PDFs stored as arxiv_{id}, doi_{slug}, or local_{slug} in vault/PDFs/", "2026-01-08 - Note template sections: Title+byline \u2192 Quote \u2192 Quick Refresh \u2192 Why You Cared \u2192 Key Concepts \u2192 Cites \u2192 Cited By \u2192 Details \u2192 Abstract \u2192 Full Citation List", "2026-01-08 - Missing modules identified (11 total): 5 blocking (grobid_processor, synthesis_generator, markdown_writer, arxiv_fetcher, orchestrator), 6 supporting. Sequential build order recommended.", "2026-01-08 - env.example exists but not in project files due to hidden file upload limitation", "2026-01-08 - Phase A complete (all 5 blocking modules): grobid_processor, synthesis_generator, markdown_writer, arxiv_fetcher, orchestrator", "2026-01-08 - v0.1.0 MVP Phase A complete and ready for testing - all 5 core modules + 5 test scripts + comprehensive README", "2026-01-08 - All individual files presented as separate downloads due to UI limitations - organized into paper_library/, root, and tests/ directories", "2026-01-08 - Model name fix required: claude-haiku-20250514 \u2192 claude-3-5-haiku-20241022 (actual current Haiku model)", "2026-01-08 - Virtual environment setup: need to run 'source .venv/bin/activate' before each session - added to README", "2026-01-08 - First test revealed formatting issues: copyright in title, title as quote, citation whitespace - all fixed", "2026-01-08 - Pricing updated: Haiku 4.5 is $1.00 input / $5.00 output per million tokens (Jan 2026)", "2026-01-08 - Citation formatting improved: now builds from parsed fields (Authors Year Title DOI) with proper punctuation, fallback to cleaned raw text", "2026-01-10 - Phase B roadmap: doi_fetcher.py (CrossRef + Unpaywall), web_fetcher.py (HTML\u2192markdown), mixed batch testing \u2192 v0.1.5", "2026-01-10 - Phase C roadmap: cli.py (proper CLI), enhanced error handling, logging framework \u2192 v0.2.0", "2026-01-10 - v0.1.0 MVP COMPLETE - ready to process 24 arXiv papers, all core functionality working, comprehensive filtering in place", "2026-01-10 - Scoring system uses: baseline trust from parsed fields (author+year), mutually-exclusive garbage categories (algo/math/figure/bio), strong weighting for + and = operators", "2026-01-10 - Current Citation model only stores authors/title/year/doi - GROBID extracts venue/volume/pages but we're throwing that data away during citation extraction", "2026-01-10 - User background: Victorianist (19th century lit) - citations routinely go back 50-100+ years, classics/medieval even older. System must handle humanities citation patterns, not just recent CS/AI papers", "2026-01-10 - Test markers: @pytest.mark.integration for slow/network tests (skip with: pytest -m 'not integration'), enables fast unit test feedback during development", "2026-01-10 - Pre-commit workflow: Prevents bad commits, improves code quality automatically, integrates with git (pre-commit install = one-time setup)", "2026-01-10 - Devcontainer = complete isolated dev environment in Docker - benefits: reproducibility, consistency across team, CI/CD ready, easy cleanup. Accessible via VS Code 'Reopen in Container' command" ], "what_we_resolved": [ "2026-01-06 - Paper ingestion pipeline architecture \u00e2\u2020\u2019 Decided: GROBID full processing with OCRmyPDF preprocessing for scanned materials, citation extraction deferred to Phase 2", "2026-01-07 - Output interface: Notion vs faster alternatives? \u00e2\u2020\u2019 Decided: Obsidian for fast generation, excellent search, and graph view", "2026-01-08 - Database vs JSON/Markdown for MVP - what are the tradeoffs? \u00e2\u2020\u2019 Decided: Use Obsidian frontmatter + simple JSON processing state, no SQLite for MVP", "2026-01-08 - SQLite vs Obsidian-native metadata (YAML frontmatter + Dataview) - which for MVP? \u00e2\u2020\u2019 Decided: Obsidian frontmatter for metadata, processing_state.json for deduplication tracking", "2026-01-08 - MVP input priority: arXiv+DOI first, then PDFs+web content? Or include web content in MVP? \u00e2\u2020\u2019 Decided: Include web content in MVP: arXiv + DOI + web articles + clean PDFs. Defer scanned/annotated PDFs to v0.2", "2026-01-08 - When to implement paper acquisition/retrieval stack (*arr-style)? \u00e2\u2020\u2019 Decided: Implement in v0.1.5 after MVP - use for quality upgrades of poor scans", "2026-01-08 - Obsidian output format specification - YAML vs JSON frontmatter, file naming, directory structure, citation link format? \u00e2\u2020\u2019 Decided: YAML frontmatter, 'Author et al (Year) - Title' naming, wikilink citations [[Author & Author (Year) - Title]]", "2026-01-08 - GROBID integration timeline - OCRmyPDF before/after/parallel? Fallback strategy for scanned PDFs? \u00e2\u2020\u2019 Decided: GROBID for all MVP papers; OCRmyPDF preprocessing deferred to v0.2 with scanned PDFs", "2026-01-08 - ArXiv paper source handling - fetch from API each time or expect local files? Mixed approach? \u00e2\u2020\u2019 Decided: Fetch from arXiv API by ID; no local file assumption for MVP" ]}