From e183696273081c8b25244a97cb95f71daa5ee4d5 Mon Sep 17 00:00:00 2001 From: Michael Granitzer Date: Sat, 3 Jan 2026 09:09:26 +0100 Subject: [PATCH] docs: add py4lexis4-cmd branch documentation with current status analysis Analysis includes: - cmd package structure (320KB total, base.py 1783 lines) - Dual CLI framework issue (Click + partial Typer) - Custom SubCommand system problems - Parameter parsing inconsistencies - Proposed Typer-based architecture - Command inventory for all 6 command classes --- docs/branch/py4lexis4-cmd.md | 228 +++++++++++++++++++++++++++++++++++ 1 file changed, 228 insertions(+) create mode 100644 docs/branch/py4lexis4-cmd.md diff --git a/docs/branch/py4lexis4-cmd.md b/docs/branch/py4lexis4-cmd.md new file mode 100644 index 0000000..fee98a5 --- /dev/null +++ b/docs/branch/py4lexis4-cmd.md @@ -0,0 +1,228 @@ +# Branch: py4lexis4-cmd + +**Epic**: Modernize CLI Architecture +**Goal**: Rework `owilix.cmd` and `owilix.plugins` packages to use Typer consistently +**Status**: 🟡 Planning Phase + +## Cycle 1: Analysis and Planning + +### Phase 1.1: Current Status Analysis + +#### Architecture Overview + +``` +owilix/ +├── cli.py # Main CLI entry point (Click-based, 505 lines) +├── tcli.py # Experimental Typer CLI (partial, 310 lines) +└── cmd/ # Command implementations + ├── __init__.py # Exports: Local, Remote, Admin, Config, Query, Batch + ├── base.py # Base classes (1783 lines!) + ├── admin.py # AdminCommands (6.6KB) + ├── batch.py # BatchCommands (22KB) + ├── config.py # ConfigCommands (7.2KB) + ├── local.py # LocalCommands (29KB) + ├── query.py # QueryCommands (18.8KB) + ├── remote.py # RemoteCommands (32KB) + ├── workflows.py # WorkflowCommands (37.8KB) + └── subcmds/ # Extended subcommands + ├── query_extended.py (28.5KB) + ├── query_graphs.py (97KB!) + ├── graph_utils.py (14KB) + └── query_warc/ # WARC subcommands + +plugins/ + ├── push/ # Push consumers + │ ├── opensearch.py (16.6KB) + │ └── convert.py + ├── ngram/ # N-gram processing + └── search_consumers.py +``` + +**Total**: ~320KB of command code, 1783 lines in base.py alone + +--- + +### Current Implementation Issues + +#### 1. Dual CLI Framework (Click + Typer) +- **cli.py**: Primary entry point using Click (~505 lines) +- **tcli.py**: Experimental Typer reimplementation (~310 lines, incomplete) +- **Problem**: Maintenance burden, inconsistent behavior + +#### 2. Custom SubCommand System (`base.py`) +```python +class SubCommand: + def __init__(self): + self.commands = {} + self.commands["help"] = self.help + + @classmethod + def register(cls, func): + # Custom decorator for registering commands + +class SubCommandMeta(type): + # Metaclass for automatic command discovery +``` +**Problem**: Non-standard, complex, hard to extend + +#### 3. Parameter Parsing Issues +- Uses custom `_cast_args()` for type conversion via pydantic +- String escaping inconsistent (e.g., `files="['**/*']"`) +- Not adhering to CLI conventions like `--flag` vs positional args + +#### 4. Giant Base Class Antipattern +`base.py` contains 1783 lines with: +- `BaseCommand` (UI, logging, output formatting) +- `SQLBaseCommands` (extends BaseCommand, 696 lines) +- Helper functions (error collection, progress display) +- Many nested inner functions + +#### 5. Command Registration Pattern +```python +@QueryCommands.register +def less(self, local_specifier: str, remote_specifier: str, ...): +``` +- Registers functions as methods via decorator +- Inconsistent with Typer's `@app.command()` pattern + +--- + +### Testing Coverage + +| Test File | Type | Coverage | +|-----------|------|----------| +| `tests/owilix/cmd/functional_test.py` | Functional | CLI commands via subprocess | +| `tests/owilix/cli/test_smoke.py` | Smoke | Basic CLI invocation | +| `tests/owilix/cli/test_ai_verifiable.py` | AI | Machine-parseable results | + +**Missing**: Unit tests for individual command functions + +--- + +### Documentation + +| Location | Content | +|----------|---------| +| `docs/commands.rst` | CLI reference (if exists) | +| `cli.py` docstrings | Main CLI documentation | +| Command class docstrings | Per-command docs | + +--- + +## Proposed Architecture (Typer-based) + +### Goals +1. **Single framework**: Typer only (no Click) +2. **Modular**: Each command group in separate file +3. **Extensible**: Plugin system for new commands +4. **Standard**: Consistent parameter handling +5. **Testable**: Commands as pure functions + +### Target Structure +``` +owilix/ +├── cli/ # New CLI package +│ ├── __init__.py # Main app +│ ├── main.py # Entry point, common options +│ ├── remote.py # remote_app (Typer) +│ ├── local.py # local_app (Typer) +│ ├── query.py # query_app (Typer) +│ ├── admin.py # admin_app (Typer) +│ ├── config.py # config_app (Typer) +│ ├── batch.py # batch_app (Typer) +│ └── utils/ # Shared utilities +│ ├── output.py # Table, JSON, console output +│ ├── context.py # Shared context (OWIlixManager) +│ └── decorators.py # Common decorators + +cmd/ # Keep as command logic (no CLI) + ├── base.py # SLIM: Just shared logic + ├── query_logic.py # Query business logic + └── ... +``` + +### Key Changes +1. **Separate CLI from logic**: `owilix/cli/` for CLI, `owilix/cmd/` for business logic +2. **Per-command Typer apps**: Modular, composable +3. **Typer.Option/Argument**: Standard parameter handling +4. **Context dependency injection**: Via Typer callback + +--- + +## Cycle Tasks + +### Phase 1.1: Analysis ✅ +- [x] Analyze `owilix.cmd` package structure +- [x] Analyze `owilix.plugins` package structure +- [x] Document current implementation issues +- [x] Review test coverage +- [x] Create branch documentation + +### Phase 1.2: Planning (TODO) +- [ ] Design target architecture +- [ ] Define migration strategy (incremental vs. big-bang) +- [ ] Identify shared utilities to extract +- [ ] Plan test migration +- [ ] Create implementation plan + +### Phase 1.3: Prototype (TODO) +- [ ] Create `owilix/cli/` package skeleton +- [ ] Implement one command end-to-end (e.g., `remote ls`) +- [ ] Verify tests pass +- [ ] Document pattern for other commands + +--- + +## Key Decisions + +| Decision | Rationale | +|----------|-----------| +| Keep `owilix.cmd` for logic | Separation of concerns | +| New `owilix.cli` for Typer | Clean slate, no tech debt | +| Incremental migration | Lower risk, testable | +| Keep existing tests | Validate behavior unchanged | + +--- + +## Command Inventory + +### LocalCommands +- `ls` - List local datasets +- `remove` - Remove dataset from repository +- `free` - Free local copy +- `insert` - Insert files from filesystem +- `analyze_jsonl` - Analyze JSONL files +- `export` - Export with CIFF merge + +### RemoteCommands +- `ls` - List remote datasets +- `pull` - Download datasets +- `push` - Upload datasets +- `remove` - Remove remote dataset +- `diff` - Compare local/remote +- `doctor` - Check connection status +- `logout` - Logout from remote +- `catalog` - Generate dataset catalog + +### QueryCommands +- `less` - Interactive browser +- `sites` - URL-based queries +- `analyze` - Transaction log analysis +- Extended: `slice`, `aggregate`, `stream`, `warc` + +### AdminCommands +- (TBD - analyze admin.py) + +### ConfigCommands +- (TBD - analyze config.py) + +### BatchCommands +- (TBD - analyze batch.py) + +--- + +## References + +- [Typer Documentation](https://typer.tiangolo.com/) +- [Click to Typer Migration](https://typer.tiangolo.com/tutorial/commands/one-or-multiple/) +- Current entry point: `owilix/cli.py:main` -- 2.51.2