From 08cd89ff7056cb48a382fc30c6ff74e80a7600b6 Mon Sep 17 00:00:00 2001 From: Michael Granitzer Date: Fri, 2 Jan 2026 15:54:19 +0100 Subject: [PATCH] Add comprehensive testing documentation - Create docs/testing.md with AI agent testing guide - Document test types, timing, markers, and verification patterns - Add testing.md link to docs/epics.md key resources - Include CLI smoke tests and AI-verifiable test patterns --- docs/epics.md | 1 + docs/testing.md | 197 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 198 insertions(+) create mode 100644 docs/testing.md diff --git a/docs/epics.md b/docs/epics.md index 463c095..504ebf7 100644 --- a/docs/epics.md +++ b/docs/epics.md @@ -4,6 +4,7 @@ Epic and backlog tracking for owilix development. > **For AI Agents**: This is your starting point. Check the **In Progress** section for current work, or the **Backlog** for planned work. Key resources: > - `docs/AI_DEVELOPMENT.md` - AI development guide +> - `docs/testing.md` - Testing guide for AI agents > - `.agent/workflows/` - Workflows (invoke with `/development`, `/testing`, etc.) > - `docs/architecture.md` - Project structure > - `docs/CODING_GUIDELINES.md` - Coding standards diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 0000000..332a6a0 --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,197 @@ +# Testing Guide + +Comprehensive testing documentation for OWILIX developers and AI agents. + +> **For AI Agents**: This is your testing reference. Use the Quick Verification section for fast checks, and CLI Smoke Tests for subprocess-based testing. + +## Quick Verification (Recommended for AI) + +Before committing changes, run these quick checks: + +```bash +# turbo +# 1. Check imports work +uv run python -c "from owilix.core.manager import OWIlixManager; print('✅ Imports OK')" + +# turbo +# 2. Run fast CLI smoke tests (~8s) +uv run pytest tests/owilix/cli/test_smoke.py -v -m "not integration" + +# turbo +# 3. Run AI verification suite (structured output) +uv run python tests/owilix/cli/test_ai_verifiable.py +``` + +## Test Organization + +``` +tests/ +├── owilix/ +│ ├── cli/ # CLI smoke tests (fast, subprocess-based) +│ │ ├── test_smoke.py # 7 quick tests (~8s total) +│ │ └── test_ai_verifiable.py # AI-interpretable JSON output +│ └── core/ +│ ├── repository/ # Repository tests +│ │ ├── test_integration.py # CLI integration (slow, network) +│ │ └── test_repository_legacy_ported.py +│ ├── fsspec/ # fsspec tests (19/21 passing) +│ │ ├── test_core_fsspec_unit.py +│ │ └── benchmark_async.py +│ └── test_metadata.py +└── data/ # Test fixtures +``` + +## Test Types & Timing + +| Test Suite | Command | Duration | Network | +|------------|---------|----------|---------| +| CLI Smoke | `pytest tests/owilix/cli/test_smoke.py -m "not integration"` | ~8s | No | +| fsspec Unit | `pytest tests/owilix/core/fsspec/` | ~8s | No | +| Repository Integration | `pytest tests/owilix/core/repository/test_integration.py` | 5-10min | Yes | + +## Pytest Markers + +Registered in `pyproject.toml`: + +```bash +# Skip network-dependent tests +uv run pytest tests/ -m "not integration" + +# Run only integration tests +uv run pytest tests/ -m integration -v -s + +# Skip slow tests +uv run pytest tests/ -m "not slow" +``` + +## CLI Smoke Tests + +Fast, subprocess-based tests that run actual `owi` commands: + +```python +# tests/owilix/cli/test_smoke.py + +def run_owi(*args, timeout=30): + """Run owi command via subprocess.""" + result = subprocess.run( + ["uv", "run", "owi"] + list(args), + capture_output=True, + text=True, + timeout=timeout + ) + return result.returncode, result.stdout, result.stderr + +# Example test +def test_config_version(self): + code, out, err = run_owi("config", "version") + assert code == 0 + assert "Config version:" in out +``` + +## AI-Verifiable Tests + +Tests with structured JSON output for AI parsing: + +```bash +# Run verification suite +uv run python tests/owilix/cli/test_ai_verifiable.py + +# Example output: +# ✅ owi --help (1.33s) +# ✅ owi config version (1.29s) +# --- SUMMARY --- +# {"total": 3, "passed": 3, "failed": 0, "all_passed": true} +``` + +The output includes `--- AI RESULT ---` blocks with structured JSON: + +```json +{ + "command": "owi config version", + "exit_code": 0, + "success": true, + "elapsed_seconds": 1.29, + "stdout_preview": "Config version: 3.0...", + "contains": { + "error": false, + "traceback": false, + "success_icon": true + } +} +``` + +## Integration Tests (Network Required) + +These tests invoke full CLI commands with network access: + +```bash +# Run with output visible (-s flag important for progress) +uv run pytest tests/owilix/core/repository/test_integration.py -v -s + +# Expected duration: 5-10 minutes +# Tests include: remote ls, remote pull, local ls +``` + +Progress indicators show timing during slow operations: + +``` +[0.0s] Starting remote ls test... +[5.2s] CLI completed with exit code 0 +[5.2s] ✅ Remote ls test passed +``` + +## Known Issues + +### Import Errors + +- `test_graph_commands.py` requires `owilix.core.ui` (not yet implemented) +- Skip with: `--ignore=tests/owilix/core/test_graph_commands.py` + +### Mock Failures in fsspec + +2 tests fail due to mock subscripting issues: +- `test_ls_returns_paths` +- `test_ls_with_detail` + +These are test issues, not code issues. + +## Adding New Tests + +### Unit Tests (Fast) + +```python +def test_my_feature(): + """Unit test - fast, no network.""" + result = my_function() + assert result == expected +``` + +### Integration Tests (Network) + +```python +import pytest + +@pytest.mark.integration +def test_remote_operation(capsys): + """Integration test - requires network.""" + print("[0.0s] Starting test...") # Progress with -s flag + # ... test code ... + print("[5.0s] ✅ Test passed") +``` + +## Recommended Workflow + +1. **Before each commit**: Run smoke tests + ```bash + uv run pytest tests/owilix/cli/test_smoke.py -v -m "not integration" + ``` + +2. **Before merging**: Run full test suite + ```bash + uv run pytest tests/ -v --ignore=tests/owilix/core/test_graph_commands.py + ``` + +3. **For AI agents**: Use the verification script + ```bash + uv run python tests/owilix/cli/test_ai_verifiable.py + ``` -- 2.51.2