diff --git a/docs/source/advanced.md b/docs/source/advanced.md new file mode 100644 index 0000000..35e3d47 --- /dev/null +++ b/docs/source/advanced.md @@ -0,0 +1,92 @@ +# Advanced Usage and Configuration + +## printing logs + +`owilix logs` allows to print different kinds of logs. + +- `metrics` gives an overview over repository metrics +- `lexis` shows the lexis logs +- `events` shows a history of events +- `errors` shows errors + +## Debugging Problems + +`owilix` usually only reports user information with a log level only on exceptions, errors or warning. +However, to get more information, you can set teh `--loglevel DEBUG` option, which will log everything to the output. + +e.g. +```bash +owilix --loglevel DEBUG remote ls all/access=public;subResourceType*=.*parq.* +``` + +Note that by prefixing "all:" to the log level, you can extend the log level to the root logger (and thus to all modules) + + +## Using the `LexisHTTP` repository (experimental) + +`owilix` is primarily using direct file based repositories, particularly iRODS. +However, this requires non-default ports to be open, which can cause problems in non-standard environments. + +To overcome this, `owilix` provides the so called `lexishttp`repository, which is deactivated by default. +The repository is slower and does not support all commands as of know. But you should be able to list and pull datasets with it. + +The option `--remote lexishttp` activates the repository. +Note that in order to keep the data consistent, no other datacenter which is available via the Lexis Portal should be active. + +```bash +owilix --remotes lexishttp remote ls all/access=public;dataCenter=it4i files=**/language=eng/index* +``` + +Note also, the filtering for data center changes. +Usually, `owilix` is configured with one repository per data center in order to enable maximum parallel data transfer. +However, the lexis http repo aggregates datasets over data centers. +The specifier specifies the repository to be used, here lexishttp, which does not filter appropriately for the dataCenter. +Consequently, filtering for datacenter must be done in the query part, as in the example above. + +## Administrative Commands + +Should be handled with care + +### Setting metadata directly + +Mainly intended to be used after an HPC execution workflow. Metadata are inferred automatically and placeholders in strings (e.g. {resourcetype}) are also replaced + +There is an interactive mode for correcting metadata which can be turned off using `owilix --yes ......` + +```bash +owilix admin set_irods_metadata irods://username:password@server:port/ZONE/path_in_zone metadta1=key1 +``` + +Alternative, metdata can be set using a py4lexis session without specifying username:password above (but might require login via B2ACCESS browser login) + +```bash +admin set_irods_metadata owilix:///[public|project]/ do_infer=True do_count=True metadata1=value1 metadata2=value2 +``` + +## Example 1: Establishing a Session and Querying Data +```python +from py4lexis.session import LexisSession +import fsspec, irods_fsspec, duckdb +from py4lexis.lexis_irods import iRODS + +# Authenticate +session = LexisSession(in_cli=True) + +# Setup filesystem and retrieve data +irods = iRODS(session) +fs = fsspec.filesystem("irods", session=irods._iRODS__get_irods_session()) +files = fs.glob('/IT4ILexisV2/public/proj862c5962623246664c1fda27b7afb108/**/*.parquet') + +# Register filesystem in DuckDB and execute a query +parquet_files = ["irods://" + file for file in files] +duckdb.register_filesystem(fs) +df = duckdb.sql(f"SELECT COUNT(*) FROM read_parquet({parquet_files[:2]})").df() +print(df) +``` + +## Example 2: Running a Filtered Query +```python +query = f"SELECT url FROM read_parquet('{files[0]}') WHERE url LIKE 'https://www.ru.nl/' LIMIT 5" +physical_plan = duckdb.sql(f"EXPLAIN {query}").df().iloc[0, 1] +print(physical_plan) +``` \ No newline at end of file diff --git a/docs/source/architecture.md b/docs/source/architecture.md deleted file mode 100644 index 88973a9..0000000 --- a/docs/source/architecture.md +++ /dev/null @@ -1,5 +0,0 @@ -# Architecture - -`owilix` dinstinguishes between the core packages, consisting of managers and repositories, and -the command packages, realising several CLI commands. - diff --git a/docs/source/commands.md b/docs/source/commands.md index 2e9b186..9e96a22 100644 --- a/docs/source/commands.md +++ b/docs/source/commands.md @@ -1,4 +1,4 @@ -# OWIlix Commands +# OWIlix Commands and Usage The OWIlix CLI includes custom commands that can be executed via the CLI. Commands are utilising core functions and add a python rich ui based console UI on top of it. @@ -7,8 +7,358 @@ The main components are shown in the figure below. ![](_static/owilix-commands.png) +## Command-Overview -## OWILIX Command Module + +## Usage + +### Defaults +- **Path**: `~/.owi` (modifiable via `OWS_OWI_PATH` environment variable or using the `--target` option) +- **File Names**: `{internalid}.tar.gz` for datasets and `{internalid}.json` for metadata +- **Specifier Format**: `{datacenter|all}:{YYYY-MM-DD|latest}#{days}/{key=value;key=value}` + +#### Specifier Components +1. **Datacenter**: Specific data center or "all" (default: "all"). Current datacenters are `lrz`,`it4i`, `csc` +2. **Date**: In `YYYY-MM-DD` or "latest" (default: current day) +3. **Days**: Days in the past to retrieve (optional) +4. **Key=Value**: Additional filters (optional). If key ends with "*", the value is interpreted as regexp + +### Flags +`owilix` has command independent flags, that allow changing the behaviour. Please look for the online documentation for more details. + +- `owilix --loglevel all:INFO` allows to set the log level for all loggers to INFO. You can use other log levels +- `owilix --remotes it4i,lrz` allows to enable only certain configured remotes. + +### Commandline Escapes + +- all parameters up to the command are usually parsed by click and follow click syntax +- Parameters for a specific command are interepreted as args parameter by default, except when they contain an "=", which translates them to kwargs +- kwargs translation can be avoided using (), so while `"where=WHERE A=b"` will fill the kwargs parameter with name `where` `"(where=WHERE A=b)" will be considered as args parameter at the specific position + + +### Commands and Examples + +Note that `owi` and `owilix` commands are installed. + +`owilix` defines different command groups, namely local, remote and admin + +#### Login and Logout + +`owilix` requires you to login using B2ACCESS when working with remote repositories, i.e. you have to follow the link and complete the device login. Afterwards it retains a token with a specific refresh timeout. + +To revoke that token from the machine, you must use `owilix remote logout` + +#### Local Commands + +Local commands `owi local` are used to manage local datasets. + +- **Listing local datasets** + ```bash + owilix local ls all + owilix local ls all:latest + owilix local ls all:latest#14/access=public + ``` + + Lists dataset according to the specifier + +- **Listing Files** + ```bash + # List datasets and enumerate files matching --files filter + owilix local ls all:latest files="**/*ciff*" + ``` + + ```bash + # List datasets and enumerate files matching --files filter and show grouped aggregation in file count with depth k + owilix local ls all files="**/*" groups=4 + ``` + +- **Creating a local dataset and inserting files into local datasets** + + Creating a dataset from a local directory. Note that the dataset is only moved / copied to the configured owilix + directory and not pushed to the server (which needs to be done separately) + - If no InternalID is provided, the dataset is newly created + ```bash + owilix local insert file:///data/owseu/owilix/it4i/231203 access=public collectionName="mgrani" move=False + ``` + - If an internalID is provided, only files will be inserted + ```bash + owilix local insert file:///data/owseu/owilix/it4i/231203 access=public collectionName="mgrani" internalID=33ad44a3-8a85-41bf-8118-1647987c4e52 move=False + ``` + + - Creating datasets on the project level and test the diffs then: + ```bash + owilix local insert /Users/username/mydata//2023-12-03 access=project collectionName="main" move=False + owilix --profile M remote diff all/access=project + ``` + - Inserting using sub path selector (e.g. only parts of a directory): + ```bash + owilix local insert file:///data/migration/it4i/2023-12-5 'sub_path=year=2023/month=12/day=6/**/*' access=public collectionName="main" move=False owner="OpenWebSearch.eu Consortium" creator="OpenWebSearch.eu Consortium" publisher="OpenWebSearch.eu Consortium" + ``` + - Listing all datasets that have been inserted, but not pushed (i.e. data center is unknown) + ```sh + local ls all/dataCenter=unknown + ``` + +#### Remote Commands + +Remote commands operate on the datacenters specified. You can use the `--exclude dc1,dc2` flag to exclude some data centers + +##### Pulling datasets + +Pulling works per file and allows for specifying file-based glob filters. +Locally datasets are stored in the configured directory, by default `~/.owi` +(can be changed via `OWS_OWI_PATH` environment variable or in using `owilix config`). +The download is also synced, i.e. files that exists locally are not downloaded from remote. +Please be as specific as possible to avoid downloading too much data. + +- **Basic Pulls:** + ```bash + owilix remote pull lrz:latest/access=public + owilix remote pull lrz:2024-01-03 + owilix remote pull all/access=public + ``` + +- **advanced pulls using query types:** + ```bash + owilix remote ls lrz:2023-11-29#0/access=public --details --file_select ".*eng.*" + owilix remote pull lrz:2024-01-03/access=public;ResourceType=warc + owilix remote pull all/access=public;subResourceType*=.*parq.* --files "**/*=slv*" --details + ``` + + note that in the query part a '*' suffix allows to specify a regular expression (e.g. `subResourceType*=.*parq.*`) + + - **Pulls and push to another remote** + ```bash + remote pull it4i:latest#7 num_threads=1 push_to_remote=myrepository + ``` + + Note that you can add your own s3 repository to be available in owilix (but be careful, as datasets appera multiple times them). + Todo so, add teh following entry to the `owilix.cfg` file (usually under `.owi`) + ```yaml + repositories: + config: + ..... + myrepository: + options: + protocol: s3a # protocol: file, s3a, irods + key: yourkey # optional access credentials + secret: yoursecret # optional access credentials + endpoint: https://yourendpoint + path: openwebsearch-public/{access} # your path. note that {access} is needed to tell owilix, where to find different access levels dataset (e.g. public/project/privat). + async: False # whether the protocol is async capebale. To with FAlse as safe option + anonymous: True # whether the connetion is anonymous. to be used for publci buckets set to download + repository: s3a + ``` + + +- **Diffs between datasets** + You can create diffs between datasets on both, the dataset and teh file level. Note that we usually use the minimal display profile to get the full ids. + ```bash + owilix --profile M remote diff all/access=project + owilix --exclude lrz remote diff all:latest/access=public;subResourceType*=.*parq.* files=**/* + ``` + +- **Configuration:** + ```bash + owilix config fields "Date,Title,Access,DataCenter" + ``` + +- **DuckDB Queries:** + ```bash + owilix duckdb lrz:2023-11-29#0/Access=public --files "*.parquet" + ``` + +- **Logout:** + `owilix` logs in via LEXIS and keeps a refresh / offline token. If you want to remove that token after usage or for security reasons, do: + ```bash + owilix remote logout + ``` + +### Working with the Web-Graph + +Owilix supports the creation of a web-graph statistics and reports stored per dataset and can potentially aggregate over datasets. + +Graph data is stored in `.stats/graph/` containining a duckdb database with links and nodes (for hosts and domains) and a `graph_report.html` file if creatd. + + +- **Creating the Web Graph** The following command creates the web-graph. With update_mode you control if the web-graph is created or updated at the remote site or (if avoided) created locally +```bash +owilix query create_graphs --remote it4i/id=76381d62-561b-11f0-ab83-528c047b29ff update_mode=create +``` +Note that storing data along side the dataset requires project rights! + +- **Creating the Web Graph report** Create a HTML report with a table on node measures and a heatmap visualisation of teh topk nodes according to page rank. + +```bash +owilix query graph_reports --remote it4i/id=76381d62-561b-11f0-ab83-528c047b29ff mode=create +``` + +```bash +owilix query graph_reports --remote it4i/id=76381d62-561b-11f0-ab83-528c047b29ff mode=view +``` + +Note that report creation requires that `.stats/graph` exists. + + + +### Workflows for creating datasets + +Datasets can be created in two steps: + +1. Inserting dataset into the local repository: + + ```bash + owilix local insert /Users/username/mydata//2023-12-03 access=project collectionName="main" move=False + ``` + + After inserting, you can still add some more metadata to the local json file, or you can add additional files in the repository. + + Metadata can be overwritten during insert. + ```bash + owilix local insert /Users/mgrani/owseudata/migration/it4i/2023-12-03 access=project collectionName="main" move=False owner="OpenWebSearch.eu Consortium" creator="OpenWebSearch.eu Consortium" publisher="OpenWebSearch.eu Consortium" + ``` + +2. Pushing the data to the server. Note that this changes the internal id. + ```bash + owilix remote push it4i:latest/access=project;internalId=6ebaf89d-adf2-4680-ab67-82b8eaf999fa + ``` + Pushing includes a selection of local datasets and the specification of the remote properties, particularly the data center. + Note that in this case the datacenter in the dataset specifier is interpreted as being the remote datacenter. + This can be also set explicitly by using `dataCenter=`. Further remarks: + - you can specify a file glob using `files=glob` and only those files will be considered for syncing + - sync does not push files already at the server. This behaviour can be changed using `overwrite=True` + - Be careful when pushing data to not pollute datasets server side. + + +### Querying Commands + +`owilix` allows to query datasets using parquet files plus duckdb. +This works on both, local and remote data in parallel. +However, on remote data it can take some time to issue the query, especially when larger chunks of data need to be transferred. +So depending on the query (e.g. not very specific select or where statements), it might be better to first pull the dataset and then issue the query. + +#### Listing dataset content: + +`owilix` supports a less command to browse content in datasets: + +- list all ".at" websites in local datasets from all data-centers with date 2023-12-4 + ```sh + owilix query less --local all:2023-12-4 limit=500 "where=url_suffix='at'" + ``` +- list all ".at" websites in remote datasets from all data-centers with date 2023-12-4, but only use repository 'it4i' + ```sh + owilix --remotes it4i query less --remote all:2023-12-4 limit=500 "where=url_suffix='at'" + ``` +- list all ".at" websites but fetch 10 batches as buffer and select only url and title (should be faster) + ```sh + owilix --remotes it4i query less --remote it4i:2023-12-3 limit=500 "where=url_suffix='at'" select=url,title page_size=30 prefetch=10 + ``` + +- list all ".at" websites but only consider files under partition "language=deu". + This should speed up the setup time, as fewer files need to be considered (note that file selector need to select parquet files) + ```sh + owilix --remotes it4i query less --remote it4i:2023-12-3 limit=500 "where=url_suffix='at' and domain_label='Regional'" files="**/language=deu/*parquet" + ``` + Note that querying goes linear over the data, so the more specific a query is, the longer it could take. + Querying over multiple data center is parallelized and thus should be faster. + +#### Calculating Aggregates + + +- calculate resource type distribution +```sh +owilix +query +aggregate +--remote +all:latest/collectionName=main +select="ows_resource_type,count(url) as c " +where="WHERE microdata is not null or 'json-ld' is not null" +groupby="ows_resource_type" +limit=10 +pq_batch_size=1 +aggregate_query="SELECT ows_resource_type,sum(c) from aggregates group by ows_resource_type" +``` + +- calculate tld distribution +```sh +owilix +query +aggregate +--remote +all:latest/collectionName=main +select="url_suffix,count(url) as c " +groupby="url_suffix" +limit=10 +pq_batch_size=1 +aggregate_query="SELECT url_suffix,sum(c) from aggregates group by url_suffix" +``` + +### Filtering for a list of websites + +`owilix` can be used to filter for list of websites given as text file. + +```bash +echo "uni-passau.de\nopensearchfoundation.org\n"> ~/tmp/list-of-urls.txt +query sites --remote all "files=**/language=eng/*.parquet" "select=url,title,curlielabels_en" urls_file=~/tmp/list-of-urls.txt +``` + +Note that since this is a query, it can take some time. + +Another example: + +```bash +owilix query sites --remote up:2025-01-15#15/collectionName=main "select=url,title,curlielabels_en,warc_date,warc_file" urls_file=list.csv as_json=True json_file=list-result.json +``` + +You can get some statistics of the resulting json using: + +```bash +owilix local analyze_jsonl list-result.json topk=100 +``` + +### Query specific configurations + +Querying has often longer timeouts. to account for that, we are setting the irods connection timeout to None. You can use the "OWILIX_IRODS_CONNECTION_TIMEOUT" to a different value. See code + +```python +fs.session.connection_timeout = os.getenv("OWILIX_IRODS_CONNECTION_TIMEOUT", None) +``` + + +## Configuration + +`owilix` relies on a yaml based configuration file. +The default directory for the yaml configuration file is `~/.owi/owilix.cfg`. +However, the `owi` directory and consequently the used config file can be changed using the `--target` option. +For changing the configuration, you can edit the file directly or also use the owilix config command: + +1. List the config and its location: `owilix config list` +2. Set a (list of) config entry: `owilix config set repositories.config.local.options.path=/home/joe/.owi/{access}` +3. get a list of entries: `owilix config get repositories.config server-config` + +The local configuration is patched upon every start with a server call to `openwebindex.eu` containing the newest configuration for remote repositories. + +You can avoid this patching by using the flag `--no-remote-config`. + +**Examples** + +Setting your identity: + +```bash +owilix config set "user.name=Michael Granitzer" "user.email=michael.granitzer@uni-passau.de" "user.organistion=University of Passau" +owilix config get user.name user.email user.organistion +``` + +Setting the default path for the local repository: + +```bash +owilix config set repositories.config.local.options.path=/home/joe/.owi/{access} +``` + + + +## OWILIX Command Modules ```{automodule} owilix.cmd :members: @@ -29,7 +379,7 @@ The main components are shown in the figure below. :show-inheritance: ``` -```{autofunction} owilix.cmd.local.stats +```{autofunction} owilix.cmd.local.details ``` ```{autofunction} owilix.cmd.local.ls diff --git a/docs/source/dev.md b/docs/source/dev.md new file mode 100644 index 0000000..e005f74 --- /dev/null +++ b/docs/source/dev.md @@ -0,0 +1,57 @@ +# Development + +# Architecture + +`owilix` dinstinguishes between the core packages, consisting of managers and repositories, and +the command packages, realising several CLI commands. + +TODO: + +## Semantic Versioning + +`owilix` uses semantic versioning for releases, concretely [python semantic versioning](https://python-semantic-release.readthedocs.io/en/latest/). +While it is not perfectly setup yet, there is a build script `./scripts/build.sh`. +The script expects a `.env-rc` file containing credentials for pushing releases with env variables `GITLAB_TOKEN_NAME` and `GITLAB_TOKEN_SECRET`. + +Commit messages are used according to the [Angular git Log message conventions](https://github.com/angular/angular.js/blob/master/DEVELOPERS.md#commits): +- feat: A new feature +- fix: A bug fix +- docs: Documentation only changes +- style: Changes that do not affect the meaning of the code (white-space, formatting, missing semicolons, etc.) +- refactor: A code change that neither fixes a bug nor adds a feature +- perf: A code change that improves performance +- test: Adding missing or correcting existing tests +- chore: Changes to the build process or auxiliary tools and libraries such as documentation generation + + +### Misc + +- `poetry add --group dev sphinx myst-parser ` +- `poetry run sphinx-build -b html docs/source/ docs/build/html` + + +## Roadmap + +- [ ] Add more tests +- [X] Update repository statistics on download / bandwith +- [ ] add query federation support +- [ ] allow tag based specifiers for query federation +- [X] Add web-graph support. +- [ ] overall clean-up and restructuring +- [ ] Dataset cleaning +- [X] Improve metadata management when slicing +- [ ] Change metdata of dataset +- [X] Docker install + +## Using poetry with conda + + + + +## Releases +To publish via Poetry: +```bash +poetry config repositories.opencode https://opencode.it4i.eu/api/v4/projects/92/packages/pypi +poetry build +poetry publish --repository opencode -u -p +``` diff --git a/docs/source/docker.md b/docs/source/docker.md new file mode 100644 index 0000000..b4534f0 --- /dev/null +++ b/docs/source/docker.md @@ -0,0 +1,96 @@ + + +# Docker + +## Building the Docker Image + +To build the Docker image locally: + +``` bash +# Build the image +docker build -t owilix . + +# Build with a specific tag +docker build -t owilix:latest . +``` + +## Using Docker + +Once the image is built, you can use OWILIX commands through Docker +instead of installing it locally. + +### Basic Usage + +Replace `owilix` commands with `docker run --rm -it owilix`: + +``` bash +# Instead of: owilix --help +# Test if +docker run --rm -it owilix --help + +# Instead of: owilix remote ls +docker run --rm -it owilix remote ls + +# Instead of: owilix remote doctor +docker run --rm -it owilix remote doctor +``` +**Note:** + +### With Persistant Configuration and Data Persistence + +`owilix` stores its configuration files and the indices under `~/.owi`. +since docker executions do not persist the configuration file, you need to expose the volume for data persistency. +Otherwise, every execution would require you to login and accept the license (except for `owilix --help`) + +In general, run ` docker run -v ~/.owi:/home/owi/.owi --rm -it owilix ` to have the same behaviour as if you would run it without docker. + +Some examples on using a directory local configuration directory: +` +``` bash +# Create directories for config and data +mkdir -p ./config ./data + +# Run with mounted volumes +docker run --rm -it \ + -v $(pwd)/config:/home/owi/.owi \ + owilix remote ls all + +# Pull data with persistence +docker run --rm -it \ + -v $(pwd)/config:/home/owi/.owi \ + owilix remote pull lrz:latest/access=public +``` + +### Using a separate data dictionary. + +You can also use a separate folder where the datasets are stored (default under `~/.owi/`) +The docker images exposes a `/data` path therefore. +However, you also need to change the configuration of the local repository that the `/data` is used. + +so change `owilix.cfg` in your configuration directory as follows: +```yaml + local: + options: + path: /data/{access} + async: True + repository: file +``` + +Afterwards, you need to mount also the data directory when running docker: + +``` bash +# Create directories for config and data +mkdir -p ./config ./data + +# Run with mounted volumes +docker run --rm -it \ + -v $(pwd)/config:/home/owi/.owi \ + -v $(pwd)/data:/data \ + owilix remote ls all + +# Pull data with persistence +docker run --rm -it \ + -v $(pwd)/config:/home/owi/.owi \ + -v $(pwd)/data:/data \ + owilix remote pull lrz:latest/access=public +``` diff --git a/docs/source/index.rst b/docs/source/index.rst index 40448b2..2fa354e 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -16,7 +16,10 @@ Welcome to Owilix's documentation! install.md commands.md - architecture.md + advanced.md + docker.md + troubleshooting.md + dev.md details/warc.md .. automodule:: owilix diff --git a/docs/source/install.md b/docs/source/install.md index 991b0ff..ff8deee 100644 --- a/docs/source/install.md +++ b/docs/source/install.md @@ -1,8 +1,13 @@ + # Installation -owilix is available via pip install. +## Requirements +- **Python Version**: 3.10 or 3.11 (Python 3.12 may have issues with `py4lexis`) +- **Package Manager**: Pip +- **Optional Manager**: Poetry (recommended for managing Python dependencies) +- **Operating Systems**: Linux, MacOSX (not tested under Windows) -## Using Mamba / Conda +## Method 1: Install from Package URL ```bash # Create a new environment @@ -15,10 +20,83 @@ pip install owilix --index-url https://opencode.it4i.eu/api/v4/projects/92/packa owilix --help ``` +**upgrade** + +```shell +pip install --upgrade py4lexis --index-url https://opencode.it4i.eu/api/v4/projects/107/packages/pypi/simple +pip install --upgrade owilix --index-url https://opencode.it4i.eu/api/v4/projects/92/packages/pypi/simple +``` + +## Method 2: Install from Repository + +Installing from the repository directly requires having [poetry installed (in your base environment)](https://python-poetry.org/docs/) + +```bash +# Clone the repository +git clone https://opencode.it4i.eu/openwebsearcheu-public/owi-cli.git +cd owi-cli +# Install directly from the repository +poetry install +``` + + +## Method 3: Using Docker -## Install from source / git +Pull the image: +```bash +docker pull opencode.it4i.eu:5050/openwebsearcheu-public/owi-cli:latest +# see if everyting is working +docker run -it owilix --help +``` + +Run commands: + +```bash +# use an owilix command using current configuration +docker run --rm -it -v ~/.owi/:/home/owi/.owi owilix remote ls +``` + +Set an alias: ```bash -pip install git+https://opencode.it4i.eu/openwebsearcheu-public/owi-cli.git +# Add the alias at the end of the file +alias owilix='docker run --rm -it -v ~/.owi/:/home/owi/.owi owilix' ``` +Note that you have to copy the url in the browser to login. + +Please see the [Docker](#docker) section below. + + +### Install From Source + +```bash +# Add py4lexis as a source and install +poetry source add --priority=supplemental py4lexis https://opencode.it4i.eu/api/v4/projects/107/packages/pypi/simple +poetry add --source py4lexis py4lexis + +# Clone the repository and install with Poetry +git clone https://opencode.it4i.eu/openwebsearcheu-public/owi-cli.git +cd owi-cli +poetry install +``` + + + + +### Quick Checks after installation: + +After installation, run quick checks that everything is working with commands + +```bash +owilix remote doctor +``` + to list the connection status and + +```bash +owilix remote ls all +``` + +To list all available datasets. + +For more usage please see the [commands section](./commands.md) \ No newline at end of file diff --git a/docs/source/troubleshooting.md b/docs/source/troubleshooting.md new file mode 100644 index 0000000..a0971b0 --- /dev/null +++ b/docs/source/troubleshooting.md @@ -0,0 +1,5 @@ +# Troubleshooting + +- **Authentication Issues**: Delete the `.env` and/ or `~/.tokens_lxs` files. you can also run `owilix clean` which does it for you. +- **Python Errors**: Ensure Python 3.11 is used. Reinstall dependencies if needed. +- **DuckDB Issues**: Reinstall DuckDB via pip or as an OWI plugin. diff --git a/tests/lexis_session_test.py b/tests/lexis_session_test.py new file mode 100644 index 0000000..e69de29