# Development Guide

This guide covers development workflow, testing, release procedures, and documentation building.

## Architecture

OWILIX distinguishes between:
- **Core packages** (`owilix/core/`): Managers, repositories, models, and the DB query engine
- **CLI packages** (`owilix/cli/`): Typer-based command-line interface
- **Compat shim** (`owilix/compat/`): `BaseCommand`/`SubCommand` kept for plugin backward compatibility. New code should not import from here.

```
owilix/
├── cli/                # Typer CLI
│   ├── remote.py       # Remote repository commands
│   ├── local.py        # Local repository commands
│   ├── query.py        # DuckDB query commands
│   └── config.py       # Configuration commands
├── core/
│   ├── manager/        # OWIlixManager, session, config, UI helpers
│   ├── repository/     # Repository implementations
│   ├── db/             # DuckDB executors
│   ├── fsspec/         # iRODS filesystem (Http2IrodsFileSystem)
│   ├── models/         # Dataset, metadata models
│   └── tasks/          # Query and data tasks
└── compat/             # Legacy BaseCommand/SubCommand for plugins
```

---

## Development Environment

### Prerequisites

- Python 3.11+
- [uv](https://docs.astral.sh/uv/) package manager

### Setup

```bash
# Clone repository
git clone https://opencode.it4i.eu/openwebsearcheu-public/owi-cli.git
cd owi-cli

# Install dependencies
uv sync

# Install with dev/docs groups
uv sync --group dev --group docs

# Run owilix
uv run owilix --help
```

### Running Tests

```bash
# Run all tests
uv run pytest tests/

# Run specific test module
uv run pytest tests/owilix/core/ -v

# Run with coverage
uv run pytest --cov=owilix tests/
```

### Code Quality

```bash
# Format code
uv run ruff format owilix/

# Lint
uv run ruff check owilix/
```

---

## Semantic Versioning

OWILIX uses [Semantic Versioning](https://semver.org/):
- **MAJOR**: Breaking changes (use `feat!` or `BREAKING CHANGE` in commit)
- **MINOR**: New features (backward compatible)
- **PATCH**: Bug fixes

### Commit Message Conventions

Follow [Angular commit conventions](https://github.com/angular/angular.js/blob/master/DEVELOPERS.md#commits):

| Type | Description |
|------|-------------|
| `feat` | New feature |
| `fix` | Bug fix |
| `docs` | Documentation only |
| `style` | Formatting, no code change |
| `refactor` | Code change without feature/fix |
| `perf` | Performance improvement |
| `test` | Adding/fixing tests |
| `chore` | Build, tooling changes |

Example:
```
feat(query): add --prefetch option to query commands
fix(config): handle migration from V3 to V4
docs: update README with v4.0.0 migration guide
```

---

## Release Procedure

### Quick Release (Recommended)

The simplest way to create a release is via git tag. The CI/CD pipeline handles everything automatically.

```bash
# 1. Update version in owilix/_version.py and pyproject.toml
# 2. Commit changes
git add . && git commit -m "chore: bump version to X.Y.Z"
git push origin main

# 3. Create and push version tag (triggers package publish)
git tag vX.Y.Z
git push origin vX.Y.Z
```

The `publish-package` CI job triggers automatically on version tags (`v*.*.*`) and:
- Builds the package with `uv build`
- Publishes to GitLab Package Registry

**Runner requirement**: All CI jobs are tagged `owseu runner` (see the `default:` block in `.gitlab-ci.yml`) and run on the project's docker-executor runner registered under that tag.

The runner is **not** running as a permanent service — it is started on demand on the runner host:

```bash
# Start the runner (foreground; uses ~/.gitlab-runner/config.toml)
gitlab-runner run

# Stop it again with Ctrl+C once the pipeline is done
```

While the runner is offline, pipelines queue as *pending* and are picked up as soon as the runner starts — so the order of pushing a tag and starting the runner does not matter. If a pipeline stays *pending* even with the runner online, check (Project → Settings → CI/CD → Runners) that the runner's tag still matches the jobs' `owseu runner` tag, and that the user running `gitlab-runner` can access the Docker daemon (`docker info`).

To register a runner on a new machine: create a project runner under Project → Settings → CI/CD → Runners (set the tag `owseu runner`), then on the machine run `gitlab-runner register --url https://opencode.it4i.eu --token <runner-token>` choosing the `docker` executor and any default image (e.g. `python:3.12`).

### Pre-Release Checklist
1. **Verify Dependencies**:
   ```bash
   # Ensure lockfile is in sync and uncorrupted
   uv lock --upgrade
   ```
2. **Run Tests**:
   ```bash
   uv run pytest tests/ -v
   ```
   Current default gate on 2026-04-09: `448 passed, 2 skipped, 40 deselected`.
3. **Check Config Migration**:
   ```bash
   uv run owilix config version
   uv run owilix config migrate --dry-run
   ```
4. **Build Docs**:
   ```bash
   uv run sphinx-build -b html docs/source docs/build/html
   ```
5. **Check release metadata**:
   - Update both `owilix/_version.py` and `pyproject.toml`
   - Move release-ready entries from `CHANGELOG.md` `Unreleased` into the new version section
   - Confirm README install snippets match the current pinned dependency versions

### Version Bump

Edit `owilix/_version.py`:
```python
__version__ = "4.0.0"
__version_tuple__ = (4, 0, 0)
```

Also update `pyproject.toml`:
```toml
version = "4.0.0"
```

### Manual Publishing (Alternative)

If CI/CD is not available:

```bash
# Build package
uv build --out-dir dist/

# Publish to GitLab registry
UV_PUBLISH_USERNAME=__token__ \
UV_PUBLISH_PASSWORD=your-gitlab-token \
uv publish --index opencode dist/*

# Or use build script (requires .env-rc with credentials)
./scripts/build.sh
```


### Server Config Update

Update the server config for the new version:
1. Edit `owilix/core/cfg/server-cfg-V{X}.json`
2. Deploy to `https://dashboard.ows.eu/api/owilix/default-cfg-{X}-{Y}`

### Post-Release

- Verify package in GitLab Package Registry
- Update documentation on GitLab Pages (auto-deployed on main push)
- Announce release if needed

---

## Documentation

### Building Docs

```bash
# Install docs dependencies
uv sync --group docs

# Build HTML documentation
uv run sphinx-build -b html docs/source/ docs/build/html

# Serve locally for preview
cd docs/build/html && python -m http.server 8000
```

### Doc Structure

```
docs/
├── source/
│   ├── index.md          # Main index
│   ├── install.md        # Installation guide
│   ├── config.md         # Configuration guide
│   ├── migration.md      # Migration guide
│   ├── commands.md       # CLI commands reference
│   ├── dev.md            # This file
│   └── details/          # Detailed topic guides
└── build/                # Generated output
```

### Deploying Docs

Docs are auto-deployed via GitLab CI when pushing to `main`.

### Recent Internal Changes

- The legacy `owilix.cmd` package has been removed. Backward-compatibility imports now live in `owilix.compat`.
- Shared graph helper code now lives in `owilix/core/tasks/_graph_utils.py`.
- Remote full-text search is implemented in `owilix/core/tasks/search.py` and exposed via `owi remote search` plus `query ... --search`.

---

## Workflows

For AI-assisted development, see workflows in `.agent/workflows/`:
- `/development` - Branch-based development
- `/testing` - Running tests
- `/release` - Release process
- `/documentation` - Updating docs

---

## Roadmap

- [x] Docker installation
- [x] Web-graph support
- [x] Improve metadata management
- [x] DuckDB query integration
- [x] py4lexis 5.x migration
- [ ] Query federation support
- [ ] Tag-based specifiers
- [ ] Dataset cleaning tools