Command line interface for accessing the Open Web Index shards and slices.

Fork 0

owi-cli docs source dev.md

7.9 kB

Markdown

at commit af53accb

mgrani · docs: describe on-demand CI runner workflow · 3mo ago

a953ce33

Development Guide #

This guide covers development workflow, testing, release procedures, and documentation building.

Architecture #

OWILIX distinguishes between:

Core packages (owilix/core/): Managers, repositories, models, and the DB query engine
CLI packages (owilix/cli/): Typer-based command-line interface
Compat shim (owilix/compat/): BaseCommand/SubCommand kept for plugin backward compatibility. New code should not import from here.

owilix/
├── cli/                # Typer CLI
│   ├── remote.py       # Remote repository commands
│   ├── local.py        # Local repository commands
│   ├── query.py        # DuckDB query commands
│   └── config.py       # Configuration commands
├── core/
│   ├── manager/        # OWIlixManager, session, config, UI helpers
│   ├── repository/     # Repository implementations
│   ├── db/             # DuckDB executors
│   ├── fsspec/         # iRODS filesystem (Http2IrodsFileSystem)
│   ├── models/         # Dataset, metadata models
│   └── tasks/          # Query and data tasks
└── compat/             # Legacy BaseCommand/SubCommand for plugins

Development Environment #

Prerequisites #

Python 3.11+
uv package manager

Setup #

# Clone repository
git clone https://opencode.it4i.eu/openwebsearcheu-public/owi-cli.git
cd owi-cli

# Install dependencies
uv sync

# Install with dev/docs groups
uv sync --group dev --group docs

# Run owilix
uv run owilix --help

Running Tests #

# Run all tests
uv run pytest tests/

# Run specific test module
uv run pytest tests/owilix/core/ -v

# Run with coverage
uv run pytest --cov=owilix tests/

Code Quality #

# Format code
uv run ruff format owilix/

# Lint
uv run ruff check owilix/

Semantic Versioning #

OWILIX uses Semantic Versioning:

MAJOR: Breaking changes (use feat! or BREAKING CHANGE in commit)
MINOR: New features (backward compatible)
PATCH: Bug fixes

Commit Message Conventions #

Follow Angular commit conventions:

Type	Description
`feat`	New feature
`fix`	Bug fix
`docs`	Documentation only
`style`	Formatting, no code change
`refactor`	Code change without feature/fix
`perf`	Performance improvement
`test`	Adding/fixing tests
`chore`	Build, tooling changes

Example:

feat(query): add --prefetch option to query commands
fix(config): handle migration from V3 to V4
docs: update README with v4.0.0 migration guide

Release Procedure #

Quick Release (Recommended) #

The simplest way to create a release is via git tag. The CI/CD pipeline handles everything automatically.

# 1. Update version in owilix/_version.py and pyproject.toml
# 2. Commit changes
git add . && git commit -m "chore: bump version to X.Y.Z"
git push origin main

# 3. Create and push version tag (triggers package publish)
git tag vX.Y.Z
git push origin vX.Y.Z

The publish-package CI job triggers automatically on version tags (v*.*.*) and:

Builds the package with uv build
Publishes to GitLab Package Registry

Runner requirement: All CI jobs are tagged owseu runner (see the default: block in .gitlab-ci.yml) and run on the project's docker-executor runner registered under that tag.

The runner is not running as a permanent service — it is started on demand on the runner host:

# Start the runner (foreground; uses ~/.gitlab-runner/config.toml)
gitlab-runner run

# Stop it again with Ctrl+C once the pipeline is done

While the runner is offline, pipelines queue as pending and are picked up as soon as the runner starts — so the order of pushing a tag and starting the runner does not matter. If a pipeline stays pending even with the runner online, check (Project → Settings → CI/CD → Runners) that the runner's tag still matches the jobs' owseu runner tag, and that the user running gitlab-runner can access the Docker daemon (docker info).

To register a runner on a new machine: create a project runner under Project → Settings → CI/CD → Runners (set the tag owseu runner), then on the machine run gitlab-runner register --url https://opencode.it4i.eu --token <runner-token> choosing the docker executor and any default image (e.g. python:3.12).

Pre-Release Checklist #

Verify Dependencies:

# Ensure lockfile is in sync and uncorrupted
uv lock --upgrade

Run Tests:
```
uv run pytest tests/ -v
```
Current default gate on 2026-04-09: 448 passed, 2 skipped, 40 deselected.

Check Config Migration:

uv run owilix config version
uv run owilix config migrate --dry-run

Build Docs:

uv run sphinx-build -b html docs/source docs/build/html

Check release metadata:
- Update both owilix/_version.py and pyproject.toml
- Move release-ready entries from CHANGELOG.md Unreleased into the new version section
- Confirm README install snippets match the current pinned dependency versions

Version Bump #

Edit owilix/_version.py:

__version__ = "4.0.0"
__version_tuple__ = (4, 0, 0)

Also update pyproject.toml:

version = "4.0.0"

Manual Publishing (Alternative) #

If CI/CD is not available:

# Build package
uv build --out-dir dist/

# Publish to GitLab registry
UV_PUBLISH_USERNAME=__token__ \
UV_PUBLISH_PASSWORD=your-gitlab-token \
uv publish --index opencode dist/*

# Or use build script (requires .env-rc with credentials)
./scripts/build.sh

Server Config Update #

Update the server config for the new version:

Edit owilix/core/cfg/server-cfg-V{X}.json
Deploy to https://dashboard.ows.eu/api/owilix/default-cfg-{X}-{Y}

Post-Release #

Verify package in GitLab Package Registry
Update documentation on GitLab Pages (auto-deployed on main push)
Announce release if needed

Documentation #

Building Docs #

# Install docs dependencies
uv sync --group docs

# Build HTML documentation
uv run sphinx-build -b html docs/source/ docs/build/html

# Serve locally for preview
cd docs/build/html && python -m http.server 8000

Doc Structure #

docs/
├── source/
│   ├── index.md          # Main index
│   ├── install.md        # Installation guide
│   ├── config.md         # Configuration guide
│   ├── migration.md      # Migration guide
│   ├── commands.md       # CLI commands reference
│   ├── dev.md            # This file
│   └── details/          # Detailed topic guides
└── build/                # Generated output

Deploying Docs #

Docs are auto-deployed via GitLab CI when pushing to main.

Recent Internal Changes #

The legacy owilix.cmd package has been removed. Backward-compatibility imports now live in owilix.compat.
Shared graph helper code now lives in owilix/core/tasks/_graph_utils.py.
Remote full-text search is implemented in owilix/core/tasks/search.py and exposed via owi remote search plus query ... --search.

Workflows #

For AI-assisted development, see workflows in .agent/workflows/:

/development - Branch-based development
/testing - Running tests
/release - Release process
/documentation - Updating docs