diff --git a/docs/source/commands.md b/docs/source/commands.md index b25cb8f..339df6e 100644 --- a/docs/source/commands.md +++ b/docs/source/commands.md @@ -15,7 +15,7 @@ The OWIlix CLI provides a suite of commands for managing datasets, interacting w ## Detailed Documentation - **[Local Commands](details/local.md)**: `ls`, `rm`, `analyze`, `init` -- **[Remote Commands](details/remote.md)**: `ls`, `pull`, `push`, `diff`, `doctor`, `logout` +- **[Remote Commands](details/remote.md)**: `ls`, `pull`, `push`, `diff`, `summarize`, `summarize-local`, `doctor`, `logout` - **[Query Commands](details/query.md)**: - **[Slice](details/query_slice.md)**: Create new datasets from queries. - **[WARC](details/warc.md)**: WARC file extraction. @@ -61,4 +61,46 @@ Examples: Admin commands are used for troubleshooting and system monitoring. - `owi admin logs [level]`: View logs. -- `owi admin stats`: View system statistics. \ No newline at end of file +- `owi admin stats`: View system statistics. + +## Publishing to OpenSearch + +OWILIX can stream query results into OpenSearch using the built-in plugin and helper script. + +### Environment Variables + +Set these in `.env-rc` (the script sources it) or your shell: + +```bash +export OWILIX_OPENSEARCH_HOST="https://opensearch.example.org:9200" +export OWILIX_OPENSEARCH_USERNAME="your-user" +export OWILIX_OPENSEARCH_PASSWORD="your-password" +export OWILIX_OPENSEARCH_VERIFY_CERTS="true" # optional +export OWILIX_OPENSEARCH_CA_CERTS="/path/to/ca.pem" # optional +``` + +### Using `scripts/ours_index.sh` + +This script runs a `query less` against a dataset specifier and pipes JSONL into the +OpenSearch indexer using `OURSConverter`. + +```bash +./scripts/ours_index.sh "all/id=" "ows-title-index" +``` + +Arguments: +- `$1`: dataset specifier (e.g. `all/id=`) +- `$2`: OpenSearch index name + +### Direct CLI (no script) + +```bash +owilix --format jsonl query less --remote "all/id=" \ + --select "url,id,title,warc_date,day,month,year,curlielabels_en as ows_curlielabel" \ + | owilix plugin owilix.plugins.push.opensearch.OpenSearchIndexer index "ows-title-index" \ + convert_fn=owilix.plugins.push.convert.OURSConverter +``` + +Notes: +- The OpenSearch index is created automatically if missing. +- Use `delete_first=true` to recreate the index on each run. diff --git a/docs/source/dataset-structure-and-metadata.md b/docs/source/dataset-structure-and-metadata.md index ee28570..4047863 100644 --- a/docs/source/dataset-structure-and-metadata.md +++ b/docs/source/dataset-structure-and-metadata.md @@ -16,7 +16,7 @@ Datasets follow a standardized hierarchical structure in the repository: /{zone}/{access}/{project_id}/{dataset_id}/ ├── .json # Dataset metadata (DataCite format) ├── README.md # Auto-generated human-readable documentation - ├── stats.json # Statistics breakdown (by language, file counts, sizes) + ├── stats.json # Optional dataset statistics snapshot (legacy + compatibility) ├── changelog.json # Version history and changes └── data/ # Partitioned data files └── year=YYYY/month=MM/day=DD/language=LLL/ @@ -25,6 +25,14 @@ Datasets follow a standardized hierarchical structure in the repository: └── *.warc.gz # WARC archive files (raw crawl data) ``` +Local summary and host aggregations are persisted in DuckDB: + +``` +$OWS_OWI_PATH/summaries/ + ├── stats.duckdb # Local stats/hosts cache used by summarize commands + └── stats-summary-*.json # Generated summary outputs +``` + ### File Partitioning Scheme Data files are organized using a standardized partition pattern that enables efficient data discovery and selective access: @@ -325,7 +333,7 @@ owilix uses a **DataCite-compatible metadata schema** for dataset description an #### Statistics -Stored in `stats.json` and aggregated in metadata: +Available from dataset statistics artifacts (`stats.json` where present, plus local DuckDB cache in summary workflows) and aggregated in metadata: - **totalSize**: Total dataset size in bytes ```json @@ -488,10 +496,12 @@ repositories: - Object storage backends - Distributed access -#### LEXIS Repository (`lexis+http`) +#### LEXIS Repository (`lexis` / `lexis+http`) **Purpose:** LEXIS DDI (Data Discovery Interface) with iRODS HTTP API +**Note:** `lexis` and `lexis+http` are aliases for the same implementation. + **Configuration:** ```yaml repositories: @@ -499,10 +509,35 @@ repositories: it4i: repository: lexis+http options: - zone: "IT4ILexisV2" + default_zone: "IT4ILexisV2" + default_location_name: "it4i" + zones: + IT4ILexisV2: + location_name: "it4i" + OWSLRZ: + location_name: "lrz" + api_url: "https://lrz.example.org/irods_api" # Authentication via environment variables or token manager ``` +**Configuration (alias using `repository: lexis`):** +```yaml +repositories: + config: + lexis: + repository: lexis + options: + project_id: proj862c5962623246664c1fda27b7afb108 + default_zone: IT4ILexisV2 + default_location_name: it4i + zones: + IT4ILexisV2: + location_name: it4i + OWSLRZ: + location_name: lrz + api_url: https://lrz.example.org/irods_api +``` + **Path Structure:** ``` /{zone}/{access}/{project_id}/{dataset_id}/ @@ -511,7 +546,7 @@ repositories: **Features:** - Automatic token refresh (IRODSOWI wrapper) - DDI API for metadata operations -- Http2IrodsFileSystem for efficient file operations +- Zone-aware routing over Http2IrodsFileSystem for file operations - Integration with LEXIS AAI (Authentication and Authorization) **Use Cases:** @@ -694,7 +729,11 @@ repositories: it4i: repository: lexis+http options: - zone: "IT4ILexisV2" + default_zone: "IT4ILexisV2" + default_location_name: "it4i" + zones: + IT4ILexisV2: + location_name: "it4i" # Token managed automatically via py4lexis ``` @@ -752,6 +791,7 @@ owi remote summarize all --force # Regenerate existing owi remote summarize all --dry-run # Preview without writing owi remote summarize all --no-create-readme # Skip README creation owi remote summarize all --details # Show per-dataset comparisons +owi remote summarize all --hosts-only # Only process datasets missing local host cache owi remote summarize all --summary --group-by collectionName # Grouped summary ``` @@ -760,7 +800,12 @@ owi remote summarize all --summary --group-by collectionName # Grouped summary - **--details**: Print per-dataset comparison tables (metadata vs stats) and a top-5 language summary - **--group-by**: Group summary results by a metadata field (used with `--summary`) - **--hosts-topk**: Top-K hosts to include in stats.json (0 to disable; default: 1000) -- **--hosts-summary-topk**: Aggregate cached host counters in summary output (0 to disable) +- **--hosts-only**: Skip datasets that already have local host statistics in the summary cache + - When set, `--force` does not override the cache check; only datasets missing cached host stats are processed. +- **--hosts-summary-topk**: Aggregate host counters in summary output from local DuckDB cache (0 to disable) +- **--domains-topk**: Top-K domains to include in summary host aggregation (0 to disable) +- **--markdown-report**: Print a Markdown report at the end (useful for CI logs) +- **--languages-topk**: Top-K languages to include in summary/Markdown report (0 to disable; default: 30) **README Format Example:** @@ -823,9 +868,9 @@ owi remote pull all/id=b313af04-f101-11f0-89ba-02a47ca5d9fd 2. Extracts language from partition path (`language=XXX`) 3. Counts files and sizes per language 4. Optionally counts rows in parquet files (`--count-rows`) -5. Computes top hosts via parquet queries and caches counters under `$OWS_OWI_PATH/summaries/hosts` +5. Computes top hosts via parquet queries and stores counters in local DuckDB cache (`$OWS_OWI_PATH/summaries/stats.duckdb`) -Host counters are cached per dataset at `$OWS_OWI_PATH/summaries/hosts/.jsonl` and reused unless `--force` is set. +When present, legacy host cache files (`$OWS_OWI_PATH/summaries/hosts/.jsonl`) are migrated into DuckDB automatically. **Generate statistics:** ```bash @@ -839,7 +884,9 @@ The `--summary` flag aggregates statistics from multiple datasets into a single Use `--hosts-summary-topk` to aggregate cached host counters across the matched datasets and include: - total number of unique hosts - top-K hosts by count -The aggregation reads local host caches (new and legacy locations) and falls back to `stats.json` host data when caches are missing. +The aggregation uses local DuckDB cache (`$OWS_OWI_PATH/summaries/stats.duckdb`) and migrates legacy local host caches (`summaries/hosts/*.jsonl`) when found. + +Use `--topk-collection` to restrict top-K calculations (languages/hosts/domains) to a single collection name (default: `main`). **Generate summary:** ```bash @@ -847,21 +894,24 @@ owi remote summarize all --summary # Generate summary for all da owi remote summarize all --summary -o summary.json # Custom output file owi remote summarize it4i:latest --summary # Summary for specific datacenter owi remote summarize all/collectionName=main --summary # Summary for collection +owi remote summarize all --summary --topk-collection main # Top-K stats for main only ``` **How it works:** 1. Lists all datasets matching the specifier 2. For each dataset: - - Reads existing `stats.json` if available - - Generates statistics from parquet files if `stats.json` is missing + - Reads existing local DuckDB stats cache if available + - Migrates existing `stats.json` and local legacy host caches into DuckDB when found + - Generates statistics from parquet files if cache data is missing - Can be forced to regenerate with `--force` flag 3. Aggregates all statistics by language across datasets 4. Outputs a comprehensive JSON summary file -**Summary File Format (`$OWS_OWI_PATH/summaries/stats-summary.json` by default):** +**Summary File Format (`$OWS_OWI_PATH/summaries/stats-summary--.json` by default):** ```json { "generated": "2024-01-16T12:00:00.000000", + "command": "owi remote summarize all --summary", "specifier": "all", "datasetsCount": 10, "totals": { @@ -896,6 +946,14 @@ owi remote summarize all/collectionName=main --summary # Summary for collection "objects": 100000 } }, + "topkCollectionName": "main", + "byLanguageTopkCollection": { + "eng": { + "files": 400, + "size": 15000000, + "objects": 200000 + } + }, "groupBy": "collectionName", "groups": { "main": { @@ -931,21 +989,25 @@ owi remote summarize all/collectionName=main --summary # Summary for collection **Summary Output Includes:** - **generated**: Timestamp of summary generation +- **command**: Command used to generate the summary - **specifier**: Dataset specifier used for filtering - **datasetsCount**: Number of datasets included in summary - **totals**: Grand totals across all datasets (files, size, objects) -- **totalsStats**: Totals derived from stats.json across datasets +- **totalsStats**: Totals derived from statistics artifacts across datasets - **totalsMetadata**: Totals derived from dataset metadata across datasets - **byLanguage**: Aggregated statistics by language code - Sorted alphabetically by language code - Contains combined counts from all datasets +- **topkCollectionName**: Collection name used for top-K calculations (if restricted) +- **byLanguageTopkCollection**: Aggregated language stats for the top-K collection - **datasets**: Per-dataset breakdown with: - Basic metadata (id, title, datacenter, collection, date range) - Language-specific statistics - Dataset-level totals - **hostsSummary**: Optional host aggregation with coverage counts - - Includes `datasetsWithCache`, `datasetsWithStats`, and `datasetsMissingHosts` + - Includes cache coverage based on local DuckDB-backed host statistics - **groupBy/groups**: Optional grouped totals when `--group-by` is provided +- **topkByGroup**: Optional top-K matrices by group (domains/hosts) when `--group-by` is provided **Console Output:** @@ -963,11 +1025,11 @@ When running with `--summary`, the command displays: Fetching datasets for specifier all Found 10 datasets Processing datasets... ━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% - ✓ Read stats.json for Dataset 1... + ✓ Read DuckDB stats for Dataset 1... ○ Generating stats for Dataset 2... - ✓ Read stats.json for Dataset 3... + ✓ Read DuckDB stats for Dataset 3... -Summary written to ./stats-summary.json +Summary written to ~/.owi/summaries/stats-summary-20240116-120000-owi-remote-summarize-all-summary.json Summary Statistics: Datasets processed: 10 @@ -991,6 +1053,22 @@ By Language: - **Reporting**: Generate reports for stakeholders about dataset collection - **Monitoring**: Track growth of dataset collections over time +#### Local Hosts Summary + +You can also generate a local-only summary from cached host counters in +`$OWS_OWI_PATH/summaries/stats.duckdb`. This avoids remote reads and inspects +local cached host statistics (migrating legacy `summaries/hosts/*.jsonl` files automatically when present). + +**Generate local summary:** +```bash +owi remote summarize-local +owi remote summarize-local --hosts-topk 200 --domains-topk 200 +``` + +The local summary includes total unique hosts and domains, plus optional top-k +lists. The output is written to `$OWS_OWI_PATH/summaries/stats-summary--.json` +by default. + ### Python API Access For programmatic access, use the owilix Python API: diff --git a/docs/source/details/remote.md b/docs/source/details/remote.md index c91d44d..f158495 100644 --- a/docs/source/details/remote.md +++ b/docs/source/details/remote.md @@ -134,6 +134,36 @@ owi remote doctor --- +## `remote summarize` / `remote summarize-local` + +Generate dataset statistics summaries and optional README files, and build aggregated reports. + +### Usage + +```bash +owi remote summarize [SPECIFIER] [OPTIONS] +owi remote summarize-local [OPTIONS] +``` + +### Notes + +- `--summary` writes an aggregated JSON report (default under `$OWS_OWI_PATH/summaries/`). +- Summary host/domain aggregation is DuckDB-backed and stored in: + - `$OWS_OWI_PATH/summaries/stats.duckdb` +- Legacy local host cache files under `$OWS_OWI_PATH/summaries/hosts/*.jsonl` are migrated automatically when found. +- Typical high-volume usage: + +```bash +owi remote summarize all:2026-02-12#10 \ + --summary \ + --group-by collectionName \ + --hosts-summary-topk 300 \ + --domains-topk 300 \ + --languages-topk 40 +``` + +--- + ## `remote logout` Revoke the current access token, effectively logging out from remote services. diff --git a/owilix/core/tasks/remote.py b/owilix/core/tasks/remote.py index cdba618..ac00623 100644 --- a/owilix/core/tasks/remote.py +++ b/owilix/core/tasks/remote.py @@ -6,11 +6,13 @@ import asyncio import inspect import os import json +import re import tempfile from collections import Counter from concurrent.futures import ThreadPoolExecutor, as_completed from typing import Optional, Dict, List, Any +import duckdb from rich.console import Console from owilix.core.types import CommandResult @@ -488,6 +490,18 @@ def _format_size(size_bytes: int) -> str: return f"{size_bytes / 1024 ** 3:.2f} GiB" +def _markdown_table(headers: List[str], rows: List[List[Any]]) -> str: + if not headers: + return "" + lines = [] + lines.append("| " + " | ".join(headers) + " |") + lines.append("| " + " | ".join(["---"] * len(headers)) + " |") + for row in rows: + values = ["" if v is None else str(v) for v in row] + lines.append("| " + " | ".join(values) + " |") + return "\n".join(lines) + + def _extract_language_from_path(path: str) -> Optional[str]: """Extract language code from path like 'language=deu' or 'language=eng'.""" import re @@ -506,9 +520,29 @@ def _build_host(subdomain: Optional[str], domain: Optional[str], suffix: Optiona return f"{subdomain}.{base}" if subdomain else base -def _summary_output_path(owi_path: Optional[str]) -> str: +def _slugify_command(command: Optional[str], max_length: int = 80) -> str: + if not command: + return "" + slug = re.sub(r"[^A-Za-z0-9._-]+", "-", command.strip()) + slug = slug.strip("-").lower() + if len(slug) > max_length: + slug = slug[:max_length].rstrip("-") + return slug + + +def _summary_output_path( + owi_path: Optional[str], + command: Optional[str] = None, + prefix: str = "stats-summary", +) -> str: owi_root = os.path.expanduser(owi_path or "~/.owi") - return os.path.join(owi_root, "summaries", "stats-summary.json") + timestamp = __import__("datetime").datetime.now().strftime("%Y%m%d-%H%M%S") + slug = _slugify_command(command) + if slug: + filename = f"{prefix}-{timestamp}-{slug}.json" + else: + filename = f"{prefix}-{timestamp}.json" + return os.path.join(owi_root, "summaries", filename) def _hosts_cache_root(owi_path: Optional[str]) -> str: @@ -522,6 +556,531 @@ def _hosts_cache_path(dataset, cache_dir: Optional[str] = None) -> str: return os.path.join(cache_root, f"{ds_id}.jsonl") +def _host_domain_from_host(host: str) -> Optional[str]: + if not host: + return None + normalized = host.strip().lower().strip(".") + if not normalized: + return None + parts = [part for part in normalized.split(".") if part] + if len(parts) <= 1: + return normalized + return ".".join(parts[-2:]) + + +def _dataset_id(dataset: Any) -> str: + return str(dataset.metadata.get("internalID") or dataset.metadata.get("id", "unknown")) + + +def _stats_db_path(owi_path: Optional[str]) -> str: + owi_root = os.path.expanduser(owi_path or "~/.owi") + return os.path.join(owi_root, "summaries", "stats.duckdb") + + +def _ensure_stats_db_schema(conn: duckdb.DuckDBPyConnection) -> None: + conn.execute( + """ + CREATE TABLE IF NOT EXISTS dataset_stats ( + dataset_id VARCHAR PRIMARY KEY, + title VARCHAR, + data_center VARCHAR, + collection_name VARCHAR, + start_date VARCHAR, + end_date VARCHAR, + source VARCHAR, + updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP + ) + """ + ) + conn.execute( + """ + CREATE TABLE IF NOT EXISTS language_stats ( + dataset_id VARCHAR, + language VARCHAR, + files BIGINT, + size BIGINT, + objects BIGINT, + PRIMARY KEY(dataset_id, language) + ) + """ + ) + conn.execute( + """ + CREATE TABLE IF NOT EXISTS host_stats ( + dataset_id VARCHAR, + host VARCHAR, + domain VARCHAR, + count BIGINT, + PRIMARY KEY(dataset_id, host) + ) + """ + ) + + +def _open_stats_db(db_path: str) -> duckdb.DuckDBPyConnection: + os.makedirs(os.path.dirname(db_path), exist_ok=True) + conn = duckdb.connect(db_path) + _ensure_stats_db_schema(conn) + return conn + + +def _dataset_hosts_count_in_db(conn: duckdb.DuckDBPyConnection, dataset_id: str) -> int: + row = conn.execute( + "SELECT COUNT(*) FROM host_stats WHERE dataset_id = ?", + [dataset_id], + ).fetchone() + return int(row[0]) if row else 0 + + +def _upsert_hosts_counter_db( + conn: duckdb.DuckDBPyConnection, + dataset_id: str, + counter: Counter, +) -> None: + conn.execute("DELETE FROM host_stats WHERE dataset_id = ?", [dataset_id]) + if not counter: + return + sql = "INSERT INTO host_stats(dataset_id, host, domain, count) VALUES (?, ?, ?, ?)" + rows: List[tuple] = [] + for host, count in counter.items(): + host_norm = str(host) + rows.append((dataset_id, host_norm, _host_domain_from_host(host_norm), int(count))) + if len(rows) >= 50000: + conn.executemany(sql, rows) + rows = [] + if rows: + conn.executemany(sql, rows) + + +def _read_hosts_counter_db( + conn: duckdb.DuckDBPyConnection, + dataset_id: str, +) -> Optional[Counter]: + rows = conn.execute( + "SELECT host, count FROM host_stats WHERE dataset_id = ?", + [dataset_id], + ).fetchall() + if not rows: + return None + return Counter({str(host): int(count) for host, count in rows}) + + +def _read_stats_db( + conn: duckdb.DuckDBPyConnection, + dataset_id: str, + hosts_topk: int = 0, +) -> Optional[Dict[str, Any]]: + ds_row = conn.execute( + "SELECT dataset_id FROM dataset_stats WHERE dataset_id = ?", + [dataset_id], + ).fetchone() + lang_rows = conn.execute( + """ + SELECT language, files, size, objects + FROM language_stats + WHERE dataset_id = ? + ORDER BY language + """, + [dataset_id], + ).fetchall() + + if not ds_row and not lang_rows and _dataset_hosts_count_in_db(conn, dataset_id) == 0: + return None + + stats: Dict[str, Any] = { + "statistics": [{ + "kind": "language", + "values": { + str(language): { + "files": int(files or 0), + "size": int(size or 0), + "objects": int(objects or 0), + } + for language, files, size, objects in lang_rows + }, + }] + } + + if hosts_topk and hosts_topk > 0: + host_rows = conn.execute( + """ + SELECT host, count + FROM host_stats + WHERE dataset_id = ? + ORDER BY count DESC, host ASC + LIMIT ? + """, + [dataset_id, int(hosts_topk)], + ).fetchall() + if host_rows: + stats["hosts"] = { + "topk": int(hosts_topk), + "values": [{"host": str(host), "count": int(count)} for host, count in host_rows], + } + return stats + + +def _upsert_dataset_stats_db( + conn: duckdb.DuckDBPyConnection, + dataset: Any, + stats: Dict[str, Any], + source: str, + replace_hosts: bool = False, +) -> None: + dataset_id = _dataset_id(dataset) + title = str(dataset.metadata.get("title", "Untitled")) + data_center = str(getattr(dataset, "dataCenter", None) or dataset.metadata.get("dataCenter", "unknown")) + collection_name = str(dataset.metadata.get("collectionName", "unknown")) + start_date = str(dataset.metadata.get("startDate", "?"))[:10] + end_date = str(dataset.metadata.get("endDate", "?"))[:10] + + conn.execute("DELETE FROM dataset_stats WHERE dataset_id = ?", [dataset_id]) + conn.execute( + """ + INSERT INTO dataset_stats( + dataset_id, title, data_center, collection_name, start_date, end_date, source, updated_at + ) + VALUES (?, ?, ?, ?, ?, ?, ?, CURRENT_TIMESTAMP) + """, + [dataset_id, title, data_center, collection_name, start_date, end_date, source], + ) + + lang_values = _stats_totals(stats).get("languages", {}) or {} + conn.execute("DELETE FROM language_stats WHERE dataset_id = ?", [dataset_id]) + if lang_values: + rows = [] + for language, data in lang_values.items(): + rows.append( + ( + dataset_id, + str(language), + int(data.get("files", 0)), + int(data.get("size", 0)), + int(data.get("objects", 0)), + ) + ) + conn.executemany( + "INSERT INTO language_stats(dataset_id, language, files, size, objects) VALUES (?, ?, ?, ?, ?)", + rows, + ) + + hosts_counter = _hosts_counter_from_stats(stats) + if replace_hosts: + _upsert_hosts_counter_db(conn, dataset_id, hosts_counter or Counter()) + elif hosts_counter and _dataset_hosts_count_in_db(conn, dataset_id) == 0: + _upsert_hosts_counter_db(conn, dataset_id, hosts_counter) + + +def _migrate_hosts_cache_json_to_db( + cache_dir: str, + conn: duckdb.DuckDBPyConnection, + dataset_ids: Optional[set[str]] = None, +) -> Dict[str, int]: + if not os.path.isdir(cache_dir): + return {"total": 0, "migrated": 0, "skipped": 0} + + total = 0 + migrated = 0 + skipped = 0 + for name in os.listdir(cache_dir): + if not name.endswith(".jsonl"): + continue + total += 1 + dataset_id = os.path.splitext(name)[0] + if dataset_ids is not None and dataset_id not in dataset_ids: + skipped += 1 + continue + if _dataset_hosts_count_in_db(conn, dataset_id) > 0: + skipped += 1 + continue + counter = _load_hosts_counter(os.path.join(cache_dir, name)) + if not counter: + skipped += 1 + continue + _upsert_hosts_counter_db(conn, dataset_id, counter) + migrated += 1 + + return {"total": total, "migrated": migrated, "skipped": skipped} + + +def _aggregate_hosts_db( + conn: duckdb.DuckDBPyConnection, + dataset_ids: List[str], + topk: int, + domains_topk: int = 0, +) -> Optional[Dict[str, Any]]: + if (not topk or topk <= 0) and (not domains_topk or domains_topk <= 0): + return None + ids = sorted({str(v) for v in dataset_ids if v}) + if not ids: + return None + + conn.execute("CREATE TEMP TABLE IF NOT EXISTS tmp_host_target(dataset_id VARCHAR)") + conn.execute("DELETE FROM tmp_host_target") + conn.executemany("INSERT INTO tmp_host_target(dataset_id) VALUES (?)", [(v,) for v in ids]) + + unique_hosts_row = conn.execute( + """ + SELECT COUNT(*) FROM ( + SELECT h.host + FROM host_stats h + JOIN tmp_host_target t ON t.dataset_id = h.dataset_id + GROUP BY h.host + ) + """ + ).fetchone() + unique_hosts = int(unique_hosts_row[0]) if unique_hosts_row else 0 + if unique_hosts == 0: + return None + + unique_domains_row = conn.execute( + """ + SELECT COUNT(*) FROM ( + SELECT h.domain + FROM host_stats h + JOIN tmp_host_target t ON t.dataset_id = h.dataset_id + WHERE h.domain IS NOT NULL AND h.domain <> '' + GROUP BY h.domain + ) + """ + ).fetchone() + unique_domains = int(unique_domains_row[0]) if unique_domains_row else 0 + + hosts_values: List[Dict[str, Any]] = [] + if topk and topk > 0: + host_rows = conn.execute( + """ + SELECT h.host, SUM(h.count) AS total_count + FROM host_stats h + JOIN tmp_host_target t ON t.dataset_id = h.dataset_id + GROUP BY h.host + ORDER BY total_count DESC, h.host ASC + LIMIT ? + """, + [int(topk)], + ).fetchall() + hosts_values = [{"host": str(host), "count": int(count)} for host, count in host_rows] + + domains_values: List[Dict[str, Any]] = [] + if domains_topk and domains_topk > 0: + domain_rows = conn.execute( + """ + SELECT h.domain, SUM(h.count) AS total_count + FROM host_stats h + JOIN tmp_host_target t ON t.dataset_id = h.dataset_id + WHERE h.domain IS NOT NULL AND h.domain <> '' + GROUP BY h.domain + ORDER BY total_count DESC, h.domain ASC + LIMIT ? + """, + [int(domains_topk)], + ).fetchall() + domains_values = [{"domain": str(domain), "count": int(count)} for domain, count in domain_rows] + + datasets_with_hosts_row = conn.execute( + """ + SELECT COUNT(*) FROM ( + SELECT h.dataset_id + FROM host_stats h + JOIN tmp_host_target t ON t.dataset_id = h.dataset_id + GROUP BY h.dataset_id + ) + """ + ).fetchone() + datasets_with_hosts = int(datasets_with_hosts_row[0]) if datasets_with_hosts_row else 0 + datasets_missing_hosts = max(0, len(ids) - datasets_with_hosts) + + return { + "uniqueHosts": unique_hosts, + "topk": topk, + "values": hosts_values, + "uniqueDomains": unique_domains, + "domainsTopk": domains_topk, + "domains": domains_values, + "datasetsWithDb": datasets_with_hosts, + "datasetsWithCache": datasets_with_hosts, + "datasetsWithStats": 0, + "datasetsMissingHosts": datasets_missing_hosts, + } + + +def _aggregate_group_topk_db( + conn: duckdb.DuckDBPyConnection, + group_dataset_ids: Dict[str, List[str]], + hosts_topk: int, + domains_topk: int, +) -> Dict[str, Any]: + result: Dict[str, Any] = {} + if not group_dataset_ids: + return result + + group_rows = [] + for group_key, ids in group_dataset_ids.items(): + for dataset_id in sorted({str(v) for v in ids if v}): + group_rows.append((dataset_id, str(group_key))) + if not group_rows: + return result + + conn.execute("CREATE TEMP TABLE IF NOT EXISTS tmp_group_target(dataset_id VARCHAR, group_key VARCHAR)") + conn.execute("DELETE FROM tmp_group_target") + conn.executemany("INSERT INTO tmp_group_target(dataset_id, group_key) VALUES (?, ?)", group_rows) + groups = sorted(group_dataset_ids.keys()) + + if domains_topk and domains_topk > 0: + top_domains = conn.execute( + """ + SELECT h.domain, SUM(h.count) AS total_count + FROM host_stats h + JOIN tmp_group_target g ON g.dataset_id = h.dataset_id + WHERE h.domain IS NOT NULL AND h.domain <> '' + GROUP BY h.domain + ORDER BY total_count DESC, h.domain ASC + LIMIT ? + """, + [int(domains_topk)], + ).fetchall() + top_domain_keys = [str(domain) for domain, _ in top_domains] + if top_domain_keys: + conn.execute("CREATE TEMP TABLE IF NOT EXISTS tmp_top_domains(domain VARCHAR)") + conn.execute("DELETE FROM tmp_top_domains") + conn.executemany("INSERT INTO tmp_top_domains(domain) VALUES (?)", [(v,) for v in top_domain_keys]) + rows = conn.execute( + """ + SELECT h.domain, g.group_key, SUM(h.count) AS total_count + FROM host_stats h + JOIN tmp_group_target g ON g.dataset_id = h.dataset_id + JOIN tmp_top_domains t ON t.domain = h.domain + GROUP BY h.domain, g.group_key + """ + ).fetchall() + counts: Dict[str, Dict[str, int]] = {} + for domain, group_key, count in rows: + counts.setdefault(str(domain), {})[str(group_key)] = int(count) + domain_rows = [] + for domain in top_domain_keys: + domain_rows.append( + { + "domain": domain, + "counts": {group: counts.get(domain, {}).get(group, 0) for group in groups}, + } + ) + result["domains"] = {"topk": domains_topk, "groups": groups, "rows": domain_rows} + + if hosts_topk and hosts_topk > 0: + top_hosts = conn.execute( + """ + SELECT h.host, SUM(h.count) AS total_count + FROM host_stats h + JOIN tmp_group_target g ON g.dataset_id = h.dataset_id + GROUP BY h.host + ORDER BY total_count DESC, h.host ASC + LIMIT ? + """, + [int(hosts_topk)], + ).fetchall() + top_host_keys = [str(host) for host, _ in top_hosts] + if top_host_keys: + conn.execute("CREATE TEMP TABLE IF NOT EXISTS tmp_top_hosts(host VARCHAR)") + conn.execute("DELETE FROM tmp_top_hosts") + conn.executemany("INSERT INTO tmp_top_hosts(host) VALUES (?)", [(v,) for v in top_host_keys]) + rows = conn.execute( + """ + SELECT h.host, g.group_key, SUM(h.count) AS total_count + FROM host_stats h + JOIN tmp_group_target g ON g.dataset_id = h.dataset_id + JOIN tmp_top_hosts t ON t.host = h.host + GROUP BY h.host, g.group_key + """ + ).fetchall() + counts: Dict[str, Dict[str, int]] = {} + for host, group_key, count in rows: + counts.setdefault(str(host), {})[str(group_key)] = int(count) + host_rows = [] + for host in top_host_keys: + host_rows.append( + { + "host": host, + "counts": {group: counts.get(host, {}).get(group, 0) for group in groups}, + } + ) + result["hosts"] = {"topk": hosts_topk, "groups": groups, "rows": host_rows} + + return result + + +def _aggregate_local_hosts_cache( + cache_dir: str, + hosts_topk: int, + domains_topk: int, + console: Optional[Console] = None, +) -> Optional[Dict[str, Any]]: + if not os.path.exists(cache_dir): + if console: + console.print(f"[yellow]Host cache directory not found: {cache_dir}[/yellow]") + return None + + counter = Counter() + total_files = 0 + loaded_files = 0 + skipped_files = 0 + + for name in os.listdir(cache_dir): + if not name.endswith(".jsonl"): + continue + total_files += 1 + path = os.path.join(cache_dir, name) + cached = _load_hosts_counter(path) + if cached: + counter.update(cached) + loaded_files += 1 + else: + skipped_files += 1 + + if not counter: + if console: + console.print("[yellow]No host data found in local cache.[/yellow]") + return None + + domain_counter = Counter() + for host, count in counter.items(): + domain = _host_domain_from_host(host) + if domain: + domain_counter[domain] += count + + hosts_values = [] + if hosts_topk and hosts_topk > 0: + hosts_values = [ + {"host": host, "count": count} + for host, count in counter.most_common(hosts_topk) + ] + + domains_values = [] + if domains_topk and domains_topk > 0: + domains_values = [ + {"domain": domain, "count": count} + for domain, count in domain_counter.most_common(domains_topk) + ] + + return { + "cacheDir": cache_dir, + "cacheFiles": { + "total": total_files, + "loaded": loaded_files, + "skipped": skipped_files, + }, + "hosts": { + "unique": len(counter), + "topk": hosts_topk, + "values": hosts_values, + }, + "domains": { + "unique": len(domain_counter), + "topk": domains_topk, + "values": domains_values, + }, + } + + def _load_hosts_counter(path: str) -> Optional[Counter]: try: with open(path, "r", encoding="utf-8") as handle: @@ -579,15 +1138,11 @@ def _hosts_counter_from_stats(stats: Optional[Dict[str, Any]]) -> Optional[Count return None -def _aggregate_hosts_caches( +def _hosts_counter_for_datasets( datasets: List[Any], - topk: int, cache_dir: Optional[str] = None, console: Optional[Console] = None, -) -> Optional[Dict[str, Any]]: - if not topk or topk <= 0: - return None - +) -> tuple[Counter, Dict[str, int]]: counter = Counter() datasets_with_cache = 0 datasets_with_stats = 0 @@ -615,25 +1170,56 @@ def _aggregate_hosts_caches( else: datasets_missing_hosts += 1 - if not counter: - if console: - console.print("[yellow]No host data found for host aggregation.[/yellow]") - return None + if not counter and console: + console.print("[yellow]No host data found for host aggregation.[/yellow]") - unique_hosts = len(counter) - return { - "uniqueHosts": unique_hosts, - "topk": topk, - "values": [ - {"host": host, "count": count} - for host, count in counter.most_common(topk) - ], + return counter, { "datasetsWithCache": datasets_with_cache, "datasetsWithStats": datasets_with_stats, "datasetsMissingHosts": datasets_missing_hosts, } +def _aggregate_hosts_caches( + datasets: List[Any], + topk: int, + domains_topk: int = 0, + cache_dir: Optional[str] = None, + db_conn: Optional[duckdb.DuckDBPyConnection] = None, + console: Optional[Console] = None, +) -> Optional[Dict[str, Any]]: + if (not topk or topk <= 0) and (not domains_topk or domains_topk <= 0): + return None + + dataset_ids = [_dataset_id(d) for d in datasets] + own_conn = False + conn = db_conn + if conn is None: + if cache_dir: + conn = _open_stats_db(os.path.join(os.path.dirname(cache_dir), "stats.duckdb")) + else: + conn = _open_stats_db(_stats_db_path(None)) + own_conn = True + try: + _migrate_hosts_cache_json_to_db( + cache_dir=cache_dir or _hosts_cache_root(None), + conn=conn, + dataset_ids=set(dataset_ids), + ) + summary = _aggregate_hosts_db( + conn=conn, + dataset_ids=dataset_ids, + topk=topk, + domains_topk=domains_topk, + ) + if not summary and console: + console.print("[yellow]No host data found for host aggregation.[/yellow]") + return summary + finally: + if own_conn: + conn.close() + + def _close_executor(db: OWIDuckDBSelectExecutor) -> None: result = db.close() if inspect.iscoroutine(result): @@ -687,87 +1273,104 @@ def _generate_hosts_counter( console: Optional[Console] = None, force: bool = False, cache_dir: Optional[str] = None, + db_conn: Optional[duckdb.DuckDBPyConnection] = None, ) -> Counter: - cache_path = _hosts_cache_path(dataset, cache_dir=cache_dir) - if not force and os.path.exists(cache_path): - cached = _load_hosts_counter(cache_path) - if cached is not None: - return cached - - pq_files = _dataset_parquet_files(dataset) - if not pq_files: - return Counter() - - def run_query(select_clause: str) -> Counter: - sql = (OWIlixSQLQuery.from_templates("pq_select") - .select(select_clause) - .where("") - .groupby("") - .partitioned_by("") - .postfix("") - .limit(None)) - counter = Counter() - db = OWIDuckDBSelectExecutor( - pq_files, - sql, - pq_batch_size=10, - batch_size=1000, - prefetch=1, - ) - try: - for results in db.query_aggregator(): - if not results.success: - continue - for row in results.rows: - if select_clause == "url": - url = row.get("url") if isinstance(row, dict) else getattr(row, "url", None) - if not url: - continue - parts = extract_domain_components(str(url)) - if not parts: - continue - host = _build_host( - parts.get("url_subdomain"), - parts.get("url_domain"), - parts.get("url_suffix"), - ) - else: - if isinstance(row, dict): - subdomain = row.get("url_subdomain") - domain = row.get("url_domain") - suffix = row.get("url_suffix") + dataset_id = _dataset_id(dataset) + own_conn = False + conn = db_conn + if conn is None: + conn = _open_stats_db(_stats_db_path(None)) + own_conn = True + try: + if not force: + cached_db = _read_hosts_counter_db(conn, dataset_id) + if cached_db is not None: + return cached_db + + cache_path = _hosts_cache_path(dataset, cache_dir=cache_dir) + if not force and os.path.exists(cache_path): + cached = _load_hosts_counter(cache_path) + if cached is not None: + _upsert_hosts_counter_db(conn, dataset_id, cached) + return cached + + pq_files = _dataset_parquet_files(dataset) + if not pq_files: + return Counter() + + def run_query(select_clause: str) -> Counter: + sql = (OWIlixSQLQuery.from_templates("pq_select") + .select(select_clause) + .where("") + .groupby("") + .partitioned_by("") + .postfix("") + .limit(None)) + counter = Counter() + db = OWIDuckDBSelectExecutor( + pq_files, + sql, + pq_batch_size=10, + batch_size=1000, + prefetch=1, + ) + try: + for results in db.query_aggregator(): + if not results.success: + continue + for row in results.rows: + if select_clause == "url": + url = row.get("url") if isinstance(row, dict) else getattr(row, "url", None) + if not url: + continue + parts = extract_domain_components(str(url)) + if not parts: + continue + host = _build_host( + parts.get("url_subdomain"), + parts.get("url_domain"), + parts.get("url_suffix"), + ) else: - subdomain = getattr(row, "url_subdomain", None) - domain = getattr(row, "url_domain", None) - suffix = getattr(row, "url_suffix", None) - host = _build_host(subdomain, domain, suffix) - if host: - counter[host] += 1 - finally: - _close_executor(db) - return counter + if isinstance(row, dict): + subdomain = row.get("url_subdomain") + domain = row.get("url_domain") + suffix = row.get("url_suffix") + else: + subdomain = getattr(row, "url_subdomain", None) + domain = getattr(row, "url_domain", None) + suffix = getattr(row, "url_suffix", None) + host = _build_host(subdomain, domain, suffix) + if host: + counter[host] += 1 + finally: + _close_executor(db) + return counter - try: - has_parts = _parquet_has_columns( - pq_files, - ["url_subdomain", "url_domain", "url_suffix"], - ) - if has_parts is False: - counter = run_query("url") - else: - counter = run_query("url_subdomain, url_domain, url_suffix") - except Exception as e: - if console: - console.print(f"[yellow]Host query failed ({e}); falling back to URL parsing.[/yellow]") try: - counter = run_query("url") - except Exception as inner: + has_parts = _parquet_has_columns( + pq_files, + ["url_subdomain", "url_domain", "url_suffix"], + ) + if has_parts is False: + counter = run_query("url") + else: + counter = run_query("url_subdomain, url_domain, url_suffix") + except Exception as e: if console: - console.print(f"[red]Host query failed ({inner}); no host data generated.[/red]") - counter = Counter() + console.print(f"[yellow]Host query failed ({e}); falling back to URL parsing.[/yellow]") + try: + counter = run_query("url") + except Exception as inner: + if console: + console.print(f"[red]Host query failed ({inner}); no host data generated.[/red]") + counter = Counter() - _write_hosts_counter(cache_path, counter) - return counter + _upsert_hosts_counter_db(conn, dataset_id, counter) + return counter + finally: + if own_conn: + conn.close() def _generate_stats_json( @@ -776,6 +1379,7 @@ def _generate_stats_json( hosts_topk: int = 1000, cache_dir: Optional[str] = None, force: bool = False, + db_conn: Optional[duckdb.DuckDBPyConnection] = None, ) -> Dict: """ Generate statistics JSON by analyzing parquet files in the dataset. @@ -824,6 +1428,7 @@ def _generate_stats_json( console=console, force=force, cache_dir=cache_dir, + db_conn=db_conn, ) if hosts_counter: stats["hosts"] = { @@ -1160,6 +1765,7 @@ def remote_readme( create_readme: bool = True, details: bool = False, hosts_topk: int = 1000, + hosts_only: bool = False, console: Optional[Console] = None, auto_yes: bool = False ) -> CommandResult: @@ -1199,6 +1805,7 @@ def remote_readme( # Check if files already exist has_readme = False has_stats = False + has_hosts_cache = False try: readme_content = d.repository.readlines(d, "README.md") @@ -1212,10 +1819,16 @@ def remote_readme( except: pass - if create_readme: - needs_processing = force or not has_readme or not has_stats + cache_path = _hosts_cache_path(d, cache_dir=hosts_cache_dir) + has_hosts_cache = os.path.exists(cache_path) + + if hosts_only: + needs_processing = not has_hosts_cache else: - needs_processing = force or not has_stats + if create_readme: + needs_processing = force or not has_readme or not has_stats + else: + needs_processing = force or not has_stats status = [] if has_readme: @@ -1226,10 +1839,17 @@ def remote_readme( status.append("[green]stats.json[/green]") else: status.append("[red]stats.json[/red]") + if hosts_only: + if has_hosts_cache: + status.append("[green]hosts cache[/green]") + else: + status.append("[red]hosts cache[/red]") action = "[yellow]SKIP[/yellow]" if not needs_processing else "[cyan]GENERATE[/cyan]" - if force and (has_readme or has_stats): + if force and (has_readme or has_stats) and not hosts_only: action = "[cyan]OVERWRITE[/cyan]" + if hosts_only and not has_hosts_cache: + action = "[cyan]GENERATE (hosts cache missing)[/cyan]" console.print(f" {ds_id}\t{title}\t{' '.join(status)}\t{action}") @@ -1248,13 +1868,18 @@ def remote_readme( if not force: stats = _read_stats_json(d, console) if stats is None: - stats = _generate_stats_json( - d, - console, - hosts_topk=hosts_topk, - cache_dir=hosts_cache_dir, - force=force, - ) + cache_path = _hosts_cache_path(d, cache_dir=hosts_cache_dir) + has_hosts_cache = os.path.exists(cache_path) + if hosts_only and has_hosts_cache: + stats = {} + else: + stats = _generate_stats_json( + d, + console, + hosts_topk=hosts_topk, + cache_dir=hosts_cache_dir, + force=force, + ) stats_cache[ds_key] = stats meta_totals = _metadata_totals(d.metadata) @@ -1372,14 +1997,19 @@ def remote_readme( if ds_key in stats_cache: stats = stats_cache[ds_key] else: - console.print(f" Analyzing parquet files for {title}...") - stats = _generate_stats_json( - d, - console, - hosts_topk=hosts_topk, - cache_dir=hosts_cache_dir, - force=force, - ) + cache_path = _hosts_cache_path(d, cache_dir=hosts_cache_dir) + has_hosts_cache = os.path.exists(cache_path) + if hosts_only and has_hosts_cache: + stats = {} + else: + console.print(f" Analyzing parquet files for {title}...") + stats = _generate_stats_json( + d, + console, + hosts_topk=hosts_topk, + cache_dir=hosts_cache_dir, + force=force, + ) # Generate README.md (optional) readme_content = None @@ -1447,14 +2077,19 @@ def remote_readme( if ds_key in stats_cache: stats = stats_cache[ds_key] else: - console.print(f" Analyzing parquet files for {title}...") - stats = _generate_stats_json( - d, - console, - hosts_topk=hosts_topk, - cache_dir=hosts_cache_dir, - force=force, - ) + cache_path = _hosts_cache_path(d, cache_dir=hosts_cache_dir) + has_hosts_cache = os.path.exists(cache_path) + if hosts_only and has_hosts_cache: + stats = {} + else: + console.print(f" Analyzing parquet files for {title}...") + stats = _generate_stats_json( + d, + console, + hosts_topk=hosts_topk, + cache_dir=hosts_cache_dir, + force=force, + ) # Generate README.md (optional) readme_content = None @@ -1497,342 +2132,534 @@ def remote_readme_summary( group_by: Optional[str] = None, hosts_topk: int = 1000, hosts_summary_topk: int = 0, + domains_topk: int = 0, + languages_topk: int = 30, + topk_collection: Optional[str] = "main", + command: Optional[str] = None, + markdown_report: bool = False, console: Optional[Console] = None, ) -> CommandResult: - """ - Generate an aggregated summary of statistics from all datasets matching the specifier. - - For each dataset: - 1. Read stats.json if it exists - 2. If not found, generate stats by analyzing parquet files - 3. Aggregate all statistics into a combined summary - - The summary includes: - - Per-dataset statistics - - Aggregated totals by language across all datasets - - Overall totals (files, size, objects) - """ if console is None: console = Console() console.print(f"Fetching datasets for specifier {specifier}") - if output_file is None: - output_file = _summary_output_path(getattr(manager, "owi_path", None)) + output_file = _summary_output_path(getattr(manager, "owi_path", None), command=command) hosts_cache_dir = _hosts_cache_root(getattr(manager, "owi_path", None)) + stats_db = _stats_db_path(getattr(manager, "owi_path", None)) + db_conn = _open_stats_db(stats_db) - spec = manager.parse_specifier(specifier) - datasets = manager.remote_data.list( - spec.pop("data_center", None), - spec.get("query", {}).pop("access", "public"), - day=spec.pop("day", None), - duration=spec.pop("duration", 0), - query=spec.pop("query", {}) - ) - - console.print(f"Found {len(datasets)} datasets") - - if not datasets: - console.print("[yellow]No datasets found matching specifier[/yellow]") - return CommandResult(success=True, object=None, msg="No datasets to summarize") - - # Aggregated statistics - aggregated_by_language: Dict[str, Dict] = {} - dataset_summaries: List[Dict] = [] - grouped: Dict[str, Dict] = {} - detail_rows: List[Dict[str, Any]] = [] - totals_metadata = {"size": 0, "files": 0, "objects": 0} - totals_stats = {"size": 0, "files": 0, "objects": 0} + try: + spec = manager.parse_specifier(specifier) + datasets = manager.remote_data.list( + spec.pop("data_center", None), + spec.get("query", {}).pop("access", "public"), + day=spec.pop("day", None), + duration=spec.pop("duration", 0), + query=spec.pop("query", {}), + ) + console.print(f"Found {len(datasets)} datasets") + if not datasets: + console.print("[yellow]No datasets found matching specifier[/yellow]") + return CommandResult(success=True, object=None, msg="No datasets to summarize") - processed = 0 - errors = 0 + aggregated_by_language: Dict[str, Dict] = {} + aggregated_by_language_topk: Dict[str, Dict] = {} + dataset_summaries: List[Dict] = [] + grouped: Dict[str, Dict] = {} + group_dataset_ids: Dict[str, List[str]] = {} + detail_rows: List[Dict[str, Any]] = [] + totals_metadata = {"size": 0, "files": 0, "objects": 0} + totals_stats = {"size": 0, "files": 0, "objects": 0} + processed = 0 + errors = 0 - with currentItemProgress() as progress: - task = progress.add_task("Collecting statistics", total=len(datasets), current_item="Setup") + topk_collection_name = None + if topk_collection is not None: + candidate = str(topk_collection).strip() + if candidate and candidate.lower() not in ("all", "*", "any", "none"): + topk_collection_name = candidate - for d in datasets: - ds_id = d.metadata.get("internalID") or d.metadata.get("id", "N/A") - title = d.metadata.get("title", "Untitled") - data_center = getattr(d, 'dataCenter', None) or d.metadata.get("dataCenter", "unknown") - collection_name = d.metadata.get("collectionName", "unknown") - start_date = str(d.metadata.get("startDate", "?"))[:10] - end_date = str(d.metadata.get("endDate", "?"))[:10] + migration = _migrate_hosts_cache_json_to_db( + cache_dir=hosts_cache_dir, + conn=db_conn, + dataset_ids={_dataset_id(d) for d in datasets}, + ) + if migration["migrated"] > 0: + console.print( + f"[green]Migrated {migration['migrated']} host cache files into {stats_db}[/green]" + ) - progress.update(task, current_item=f"Processing {title[:30]}...") + with currentItemProgress() as progress: + task = progress.add_task("Collecting statistics", total=len(datasets), current_item="Setup") + for d in datasets: + ds_id = _dataset_id(d) + title = d.metadata.get("title", "Untitled") + data_center = getattr(d, "dataCenter", None) or d.metadata.get("dataCenter", "unknown") + collection_name = d.metadata.get("collectionName", "unknown") + collection_name_norm = str(collection_name) + start_date = str(d.metadata.get("startDate", "?"))[:10] + end_date = str(d.metadata.get("endDate", "?"))[:10] + progress.update(task, current_item=f"Processing {title[:30]}...") - stats = None - stats_generated = False + stats = None + stats_generated = False + if not force: + stats = _read_stats_db(db_conn, ds_id, hosts_topk=hosts_topk) + if stats: + console.print(f" [green]✓[/green] Read DuckDB stats for {title[:40]}...") - # Try to read existing stats.json - if not force: - stats = _read_stats_json(d, console) - if stats: - console.print(f" [green]✓[/green] Read stats.json for {title[:40]}...") + if not force and stats is None: + stats = _read_stats_json(d, console) + if stats: + _upsert_dataset_stats_db(db_conn, d, stats, source="stats.json", replace_hosts=False) + console.print(f" [green]✓[/green] Migrated stats.json for {title[:40]}...") - # Generate stats if not found or force is True - if stats is None: try: - console.print(f" [cyan]○[/cyan] Generating stats for {title[:40]}...") - stats = _generate_stats_json( - d, - console, - hosts_topk=hosts_topk, - cache_dir=hosts_cache_dir, - force=force, - ) - stats_generated = True + if stats is None: + console.print(f" [cyan]○[/cyan] Generating stats for {title[:40]}...") + stats = _generate_stats_json( + d, + console=console, + hosts_topk=hosts_topk, + cache_dir=hosts_cache_dir, + force=force, + db_conn=db_conn, + ) + stats_generated = True + elif hosts_topk and hosts_topk > 0 and not stats.get("hosts"): + stats = _generate_stats_json( + d, + console=console, + hosts_topk=hosts_topk, + cache_dir=hosts_cache_dir, + force=force, + db_conn=db_conn, + ) + stats_generated = True except Exception as e: console.print(f" [red]✗[/red] Error generating stats for {title}: {e}") errors += 1 progress.update(task, advance=1) continue - elif hosts_topk and hosts_topk > 0 and not stats.get("hosts"): - try: - stats = _generate_stats_json( + + if stats: + _upsert_dataset_stats_db( + db_conn, d, - console, - hosts_topk=hosts_topk, - cache_dir=hosts_cache_dir, - force=force, + stats, + source="generated" if stats_generated else "cache", + replace_hosts=bool(stats_generated), ) - stats_generated = True - except Exception as e: - console.print(f" [red]✗[/red] Error updating host stats for {title}: {e}") - - if stats_generated: - try: - d.repository.writelines(d, "stats.json", json.dumps(stats, indent=2)) - except Exception as e: - console.print(f"[red]Error writing stats.json for {title}: {e}[/red]") - - stats_info = _stats_totals(stats) - lang_values = stats_info["languages"] - - # Calculate dataset totals - ds_total_files = stats_info["totals"]["files"] - ds_total_size = stats_info["totals"]["size"] - ds_total_objects = stats_info["totals"]["objects"] - - for lang, data in lang_values.items(): - files_count = data.get("files", 0) - size = data.get("size", 0) - objects = data.get("objects", 0) - - # Aggregate by language - if lang not in aggregated_by_language: - aggregated_by_language[lang] = {"files": 0, "size": 0, "objects": 0} - - aggregated_by_language[lang]["files"] += files_count - aggregated_by_language[lang]["size"] += size - aggregated_by_language[lang]["objects"] += objects - - # Store per-dataset summary - dataset_summaries.append({ - "id": ds_id, - "title": title, - "dataCenter": data_center, - "collectionName": collection_name, - "startDate": start_date, - "endDate": end_date, - "statistics": lang_values, - "totals": { - "files": ds_total_files, - "size": ds_total_size, - "objects": ds_total_objects - } - }) - - meta_totals = _metadata_totals(d.metadata) - if meta_totals.get("size") is not None: - totals_metadata["size"] += meta_totals["size"] - if meta_totals.get("objects") is not None: - totals_metadata["objects"] += meta_totals["objects"] - if meta_totals.get("files") is not None: - totals_metadata["files"] += meta_totals["files"] - - totals_stats["size"] += ds_total_size - totals_stats["objects"] += ds_total_objects - totals_stats["files"] += ds_total_files - - if group_by: - group_value = d.metadata.get(group_by) - if group_value is None: - group_value = "unknown" - group_key = str(group_value) - if group_key not in grouped: - grouped[group_key] = { - "datasetsCount": 0, - "totals": {"files": 0, "size": 0, "objects": 0}, - "byLanguage": {}, - "datasets": [], - } - grouped[group_key]["datasetsCount"] += 1 - grouped[group_key]["datasets"].append(ds_id) - grouped[group_key]["totals"]["files"] += ds_total_files - grouped[group_key]["totals"]["size"] += ds_total_size - grouped[group_key]["totals"]["objects"] += ds_total_objects + stats_info = _stats_totals(stats) + lang_values = stats_info["languages"] + ds_total_files = stats_info["totals"]["files"] + ds_total_size = stats_info["totals"]["size"] + ds_total_objects = stats_info["totals"]["objects"] - group_langs = grouped[group_key]["byLanguage"] for lang, data in lang_values.items(): - if lang not in group_langs: - group_langs[lang] = {"files": 0, "size": 0, "objects": 0} - group_langs[lang]["files"] += data.get("files", 0) - group_langs[lang]["size"] += data.get("size", 0) - group_langs[lang]["objects"] += data.get("objects", 0) - - if details: - dataset_label = f"{title} ({ds_id})" - if not create_readme: - dataset_label = f"[red]{dataset_label}[/red]" - - meta_size = meta_totals.get("size") - meta_objects = meta_totals.get("objects") - meta_files = meta_totals.get("files") - - detail_rows.append( + files_count = data.get("files", 0) + size = data.get("size", 0) + objects = data.get("objects", 0) + if lang not in aggregated_by_language: + aggregated_by_language[lang] = {"files": 0, "size": 0, "objects": 0} + aggregated_by_language[lang]["files"] += files_count + aggregated_by_language[lang]["size"] += size + aggregated_by_language[lang]["objects"] += objects + if topk_collection_name is None or collection_name_norm.lower() == topk_collection_name.lower(): + if lang not in aggregated_by_language_topk: + aggregated_by_language_topk[lang] = {"files": 0, "size": 0, "objects": 0} + aggregated_by_language_topk[lang]["files"] += files_count + aggregated_by_language_topk[lang]["size"] += size + aggregated_by_language_topk[lang]["objects"] += objects + + dataset_summaries.append( { - "dataset": dataset_label, - "source": "metadata", - "size": _format_size(meta_size) if meta_size is not None else "n/a", - "objects": f"{meta_objects:,}" if meta_objects is not None else "n/a", - "files": f"{meta_files:,}" if meta_files is not None else "n/a", - "languages": "", + "id": ds_id, + "title": title, + "dataCenter": data_center, + "collectionName": collection_name, + "startDate": start_date, + "endDate": end_date, + "statistics": lang_values, + "totals": {"files": ds_total_files, "size": ds_total_size, "objects": ds_total_objects}, } ) - size_style = _diff_style(meta_size, ds_total_size) - objects_style = _diff_style(meta_objects, ds_total_objects) - files_style = _diff_style(meta_files, ds_total_files) - - size_value = _format_size(ds_total_size) - objects_value = f"{ds_total_objects:,}" - files_value = f"{ds_total_files:,}" + meta_totals = _metadata_totals(d.metadata) + if meta_totals.get("size") is not None: + totals_metadata["size"] += meta_totals["size"] + if meta_totals.get("objects") is not None: + totals_metadata["objects"] += meta_totals["objects"] + if meta_totals.get("files") is not None: + totals_metadata["files"] += meta_totals["files"] + + totals_stats["size"] += ds_total_size + totals_stats["objects"] += ds_total_objects + totals_stats["files"] += ds_total_files + + if group_by: + group_value = d.metadata.get(group_by) + if group_value is None: + group_value = "unknown" + group_key = str(group_value) + if group_key not in grouped: + grouped[group_key] = { + "datasetsCount": 0, + "totals": {"files": 0, "size": 0, "objects": 0}, + "byLanguage": {}, + "datasets": [], + } + grouped[group_key]["datasetsCount"] += 1 + grouped[group_key]["datasets"].append(ds_id) + group_dataset_ids.setdefault(group_key, []).append(ds_id) + grouped[group_key]["totals"]["files"] += ds_total_files + grouped[group_key]["totals"]["size"] += ds_total_size + grouped[group_key]["totals"]["objects"] += ds_total_objects + for lang, data in lang_values.items(): + if lang not in grouped[group_key]["byLanguage"]: + grouped[group_key]["byLanguage"][lang] = {"files": 0, "size": 0, "objects": 0} + grouped[group_key]["byLanguage"][lang]["files"] += data.get("files", 0) + grouped[group_key]["byLanguage"][lang]["size"] += data.get("size", 0) + grouped[group_key]["byLanguage"][lang]["objects"] += data.get("objects", 0) + + if details: + dataset_label = f"{title} ({ds_id})" + if not create_readme: + dataset_label = f"[red]{dataset_label}[/red]" + meta_size = meta_totals.get("size") + meta_objects = meta_totals.get("objects") + meta_files = meta_totals.get("files") + detail_rows.append( + { + "dataset": dataset_label, + "source": "metadata", + "size": _format_size(meta_size) if meta_size is not None else "n/a", + "objects": f"{meta_objects:,}" if meta_objects is not None else "n/a", + "files": f"{meta_files:,}" if meta_files is not None else "n/a", + "languages": "", + } + ) + size_style = _diff_style(meta_size, ds_total_size) + objects_style = _diff_style(meta_objects, ds_total_objects) + files_style = _diff_style(meta_files, ds_total_files) + size_value = _format_size(ds_total_size) + objects_value = f"{ds_total_objects:,}" + files_value = f"{ds_total_files:,}" + if size_style: + size_value = f"[{size_style}]{size_value}[/{size_style}]" + if objects_style: + objects_value = f"[{objects_style}]{objects_value}[/{objects_style}]" + if files_style: + files_value = f"[{files_style}]{files_value}[/{files_style}]" + detail_rows.append( + { + "dataset": dataset_label, + "source": "stats.json", + "size": size_value, + "objects": objects_value, + "files": files_value, + "languages": _format_top_languages(lang_values), + } + ) - if size_style: - size_value = f"[{size_style}]{size_value}[/{size_style}]" - if objects_style: - objects_value = f"[{objects_style}]{objects_value}[/{objects_style}]" - if files_style: - files_value = f"[{files_style}]{files_value}[/{files_style}]" + processed += 1 + progress.update(task, advance=1) - detail_rows.append( - { - "dataset": dataset_label, - "source": "stats.json", - "size": size_value, - "objects": objects_value, - "files": files_value, - "languages": _format_top_languages(lang_values), - } + topk_dataset_ids = [_dataset_id(ds) for ds in datasets] + if topk_collection_name is not None: + topk_dataset_ids = [ + _dataset_id(ds) + for ds in datasets + if str(ds.metadata.get("collectionName", "unknown")).lower() == topk_collection_name.lower() + ] + if not topk_dataset_ids: + console.print( + f"[yellow]No datasets found for top-K collection '{topk_collection_name}'. " + "Top-K tables will be empty.[/yellow]" ) - processed += 1 - progress.update(task, advance=1) - - # Calculate grand totals - grand_total_files = sum(v["files"] for v in aggregated_by_language.values()) - grand_total_size = sum(v["size"] for v in aggregated_by_language.values()) - grand_total_objects = sum(v["objects"] for v in aggregated_by_language.values()) - - # Build summary structure - summary = { - "generated": __import__("datetime").datetime.now().isoformat(), - "specifier": specifier, - "datasetsCount": len(dataset_summaries), - "totals": { - "files": grand_total_files, - "size": grand_total_size, - "objects": grand_total_objects - }, - "totalsStats": { - "files": totals_stats["files"], - "size": totals_stats["size"], - "objects": totals_stats["objects"] - }, - "totalsMetadata": { - "files": totals_metadata["files"], - "size": totals_metadata["size"], - "objects": totals_metadata["objects"] - }, - "byLanguage": dict(sorted(aggregated_by_language.items())), - "datasets": dataset_summaries - } - - hosts_summary = _aggregate_hosts_caches( - datasets, - hosts_summary_topk, - cache_dir=hosts_cache_dir, - console=console, - ) - if hosts_summary: - summary["hostsSummary"] = hosts_summary - - if group_by: - summary["groupBy"] = group_by - summary["groups"] = dict(sorted(grouped.items())) + grand_total_files = sum(v["files"] for v in aggregated_by_language.values()) + grand_total_size = sum(v["size"] for v in aggregated_by_language.values()) + grand_total_objects = sum(v["objects"] for v in aggregated_by_language.values()) + summary = { + "generated": __import__("datetime").datetime.now().isoformat(), + "command": command, + "specifier": specifier, + "datasetsCount": len(dataset_summaries), + "totals": {"files": grand_total_files, "size": grand_total_size, "objects": grand_total_objects}, + "totalsStats": { + "files": totals_stats["files"], + "size": totals_stats["size"], + "objects": totals_stats["objects"], + }, + "totalsMetadata": { + "files": totals_metadata["files"], + "size": totals_metadata["size"], + "objects": totals_metadata["objects"], + }, + "byLanguage": dict(sorted(aggregated_by_language.items())), + "datasets": dataset_summaries, + } + if topk_collection_name is not None: + summary["topkCollectionName"] = topk_collection_name + summary["topkCollectionDatasetsCount"] = len(topk_dataset_ids) + summary["byLanguageTopkCollection"] = dict(sorted(aggregated_by_language_topk.items())) + + hosts_summary = _aggregate_hosts_db( + conn=db_conn, + dataset_ids=topk_dataset_ids, + topk=hosts_summary_topk, + domains_topk=domains_topk, + ) + if hosts_summary: + if topk_collection_name is not None: + hosts_summary["collectionName"] = topk_collection_name + summary["hostsSummary"] = hosts_summary + + if group_by: + summary["groupBy"] = group_by + summary["groups"] = dict(sorted(grouped.items())) + group_topk = _aggregate_group_topk_db( + conn=db_conn, + group_dataset_ids=group_dataset_ids, + hosts_topk=hosts_summary_topk, + domains_topk=domains_topk, + ) + if group_topk: + summary["topkByGroup"] = group_topk - # Write output file - try: output_dir = os.path.dirname(output_file) if output_dir: os.makedirs(output_dir, exist_ok=True) - with open(output_file, 'w') as f: + with open(output_file, "w") as f: json.dump(summary, f, indent=2) console.print(f"\n[green]Summary written to {output_file}[/green]") - except Exception as e: - console.print(f"[red]Error writing summary file: {e}[/red]") - return CommandResult(success=False, msg=f"Failed to write output file: {e}") - # Print summary table - console.print("\n[bold]Summary Statistics:[/bold]") - console.print(f" Datasets processed: {processed}") - if errors > 0: - console.print(f" [red]Errors: {errors}[/red]") + console.print("\n[bold]Summary Statistics:[/bold]") + console.print(f" Datasets processed: {processed}") + if errors > 0: + console.print(f" [red]Errors: {errors}[/red]") + console.print(f"\n[bold]Totals:[/bold]") + console.print(f" Files: {grand_total_files:,}") + console.print(f" Size: {_format_size(grand_total_size)}") + console.print(f" Objects: {grand_total_objects:,}") + + if hosts_summary: + console.print("\n[bold]Hosts Summary:[/bold]") + console.print(f" Unique hosts: {hosts_summary['uniqueHosts']:,}") + console.print( + " Host data coverage: " + f"{hosts_summary.get('datasetsWithDb', 0):,} DuckDB, " + f"{hosts_summary['datasetsMissingHosts']:,} missing" + ) - console.print(f"\n[bold]Totals:[/bold]") - console.print(f" Files: {grand_total_files:,}") - console.print(f" Size: {_format_size(grand_total_size)}") - console.print(f" Objects: {grand_total_objects:,}") + console.print("\n[bold]By Language:[/bold]") + languages_source = aggregated_by_language_topk if topk_collection_name is not None else aggregated_by_language + sorted_langs = sorted(languages_source.items(), key=lambda x: x[1]["size"], reverse=True) + show_langs = 15 if languages_topk is None else languages_topk + if not show_langs or show_langs <= 0: + show_langs = 0 + for lang, data in sorted_langs[:show_langs]: + console.print(f" {lang}: {data['files']:,} files, {_format_size(data['size'])}, {data['objects']:,} objects") + if show_langs and len(sorted_langs) > show_langs: + console.print(f" ... and {len(sorted_langs) - show_langs} more languages") + + if group_by: + console.print(_build_group_summary_table(group_by, dict(sorted(grouped.items())))) + if details: + console.print( + _build_details_table( + detail_rows, + totals_metadata=totals_metadata, + totals_stats=totals_stats, + totals_languages=aggregated_by_language, + title="Dataset Statistics (summary)", + ) + ) - if hosts_summary: - console.print(f"\n[bold]Hosts Summary:[/bold]") - console.print(f" Unique hosts: {hosts_summary['uniqueHosts']:,}") - console.print( - " Host data coverage: " - f"{hosts_summary['datasetsWithCache']:,} cache, " - f"{hosts_summary['datasetsWithStats']:,} stats.json, " - f"{hosts_summary['datasetsMissingHosts']:,} missing" - ) + msg = f"Summary generated for {processed} datasets" + if errors > 0: + msg += f" ({errors} errors)" + + if markdown_report: + md_lines = ["## OWI Summary Report", ""] + md_lines.append(f"- Generated: {summary.get('generated')}") + if command: + md_lines.append(f"- Command: `{command}`") + md_lines.append(f"- Specifier: `{specifier}`") + md_lines.append(f"- Datasets: {summary.get('datasetsCount', 0)}") + if topk_collection_name is not None: + md_lines.append(f"- Top-K collection: `{topk_collection_name}`") + md_lines.append("") + md_lines.append("### Totals") + md_lines.append( + _markdown_table( + ["Files", "Size", "Objects"], + [[f"{grand_total_files:,}", _format_size(grand_total_size), f"{grand_total_objects:,}"]], + ) + ) + md_lines.append("") + if hosts_summary: + md_lines.append("### Hosts Summary") + md_lines.append( + _markdown_table( + ["Unique Hosts", "Unique Domains", "Datasets in DuckDB", "Datasets Missing Hosts"], + [[ + f"{hosts_summary.get('uniqueHosts', 0):,}", + f"{hosts_summary.get('uniqueDomains', 0):,}", + f"{hosts_summary.get('datasetsWithDb', 0):,}", + f"{hosts_summary.get('datasetsMissingHosts', 0):,}", + ]], + ) + ) + md_lines.append("") + console.print("\n" + "\n".join(md_lines), markup=False) + + return CommandResult(success=True, object=summary, msg=msg) + finally: + db_conn.close() + + +def local_hosts_summary( + manager: Any, + output_file: Optional[str] = None, + hosts_topk: int = 1000, + domains_topk: int = 1000, + command: Optional[str] = None, + markdown_report: bool = False, + console: Optional[Console] = None, +) -> CommandResult: + """ + Analyze local host cache summaries and write a stats-summary.json file. + """ + if console is None: + console = Console() - console.print(f"\n[bold]By Language:[/bold]") - # Sort languages by size (descending) - sorted_langs = sorted(aggregated_by_language.items(), key=lambda x: x[1]["size"], reverse=True) - for lang, data in sorted_langs[:15]: # Show top 15 - console.print(f" {lang}: {data['files']:,} files, {_format_size(data['size'])}, {data['objects']:,} objects") + cache_dir = _hosts_cache_root(getattr(manager, "owi_path", None)) + stats_db = _stats_db_path(getattr(manager, "owi_path", None)) + db_conn = _open_stats_db(stats_db) + try: + migration = _migrate_hosts_cache_json_to_db(cache_dir=cache_dir, conn=db_conn, dataset_ids=None) + if migration["migrated"] > 0: + console.print(f"[green]Migrated {migration['migrated']} cache files into {stats_db}[/green]") + + dataset_ids = [row[0] for row in db_conn.execute("SELECT DISTINCT dataset_id FROM host_stats").fetchall()] + db_summary = _aggregate_hosts_db( + conn=db_conn, + dataset_ids=dataset_ids, + topk=hosts_topk, + domains_topk=domains_topk, + ) + finally: + db_conn.close() + + summary = None + if db_summary: + loaded_datasets = len(dataset_ids) + summary = { + "cacheDir": cache_dir, + "cacheFiles": { + "total": migration.get("total", 0), + "loaded": loaded_datasets, + "skipped": migration.get("skipped", 0), + }, + "hosts": { + "unique": db_summary.get("uniqueHosts", 0), + "topk": hosts_topk, + "values": db_summary.get("values", []), + }, + "domains": { + "unique": db_summary.get("uniqueDomains", 0), + "topk": domains_topk, + "values": db_summary.get("domains", []), + }, + } + else: + summary = _aggregate_local_hosts_cache( + cache_dir=cache_dir, + hosts_topk=hosts_topk, + domains_topk=domains_topk, + console=console, + ) - if len(sorted_langs) > 15: - console.print(f" ... and {len(sorted_langs) - 15} more languages") + if not summary: + return CommandResult(success=True, object=None, msg="No local host summary data found") - if group_by: - console.print(_build_group_summary_table(group_by, dict(sorted(grouped.items())))) + summary["generated"] = __import__("datetime").datetime.now().isoformat() + summary["command"] = command - if details: - console.print( - _build_details_table( - detail_rows, - totals_metadata=totals_metadata, - totals_stats=totals_stats, - totals_languages=aggregated_by_language, - title="Dataset Statistics (summary)", - ) + if output_file is None: + output_file = _summary_output_path( + getattr(manager, "owi_path", None), + command=command, + prefix="stats-summary", ) - msg = f"Summary generated for {processed} datasets" - if errors > 0: - msg += f" ({errors} errors)" + try: + output_dir = os.path.dirname(output_file) + if output_dir: + os.makedirs(output_dir, exist_ok=True) + with open(output_file, "w") as handle: + json.dump(summary, handle, indent=2) + console.print(f"\n[green]Local summary written to {output_file}[/green]") + except Exception as e: + console.print(f"[red]Error writing local summary file: {e}[/red]") + return CommandResult(success=False, msg=f"Failed to write output file: {e}") - return CommandResult(success=True, object=summary, msg=msg) + console.print("\n[bold]Local Host Summary:[/bold]") + console.print(f" Cache files: {summary['cacheFiles']['loaded']:,}/{summary['cacheFiles']['total']:,} loaded") + console.print(f" Unique hosts: {summary['hosts']['unique']:,}") + console.print(f" Unique domains: {summary['domains']['unique']:,}") + + if markdown_report: + md_lines = [] + md_lines.append("## OWI Local Hosts Summary") + md_lines.append("") + md_lines.append(f"- Generated: {summary.get('generated')}") + if command: + md_lines.append(f"- Command: `{command}`") + md_lines.append("") + md_lines.append("### Cache Coverage") + md_lines.append(_markdown_table( + ["Total Cache Files", "Loaded", "Skipped"], + [[ + f"{summary['cacheFiles']['total']:,}", + f"{summary['cacheFiles']['loaded']:,}", + f"{summary['cacheFiles']['skipped']:,}", + ]], + )) + md_lines.append("") + md_lines.append("### Unique Counts") + md_lines.append(_markdown_table( + ["Unique Hosts", "Unique Domains"], + [[ + f"{summary['hosts']['unique']:,}", + f"{summary['domains']['unique']:,}", + ]], + )) + md_lines.append("") + + domain_rows = [] + for item in summary.get("domains", {}).get("values", [])[:domains_topk]: + domain_rows.append([item.get("domain"), f"{item.get('count', 0):,}"]) + if domain_rows: + md_lines.append("### Top Domains") + md_lines.append(_markdown_table(["Domain", "Count"], domain_rows)) + md_lines.append("") + + host_rows = [] + for item in summary.get("hosts", {}).get("values", [])[:hosts_topk]: + host_rows.append([item.get("host"), f"{item.get('count', 0):,}"]) + if host_rows: + md_lines.append("### Top Hosts") + md_lines.append(_markdown_table(["Host", "Count"], host_rows)) + md_lines.append("") + + console.print("\n" + "\n".join(md_lines), markup=False) + + return CommandResult(success=True, object=summary, msg="Local host summary generated") def remote_diff( diff --git a/tests/owilix/core/tasks/__init__.py b/tests/owilix/core/tasks/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/owilix/core/tasks/test_remote.py b/tests/owilix/core/tasks/test_remote.py new file mode 100644 index 0000000..4ef721f --- /dev/null +++ b/tests/owilix/core/tasks/test_remote.py @@ -0,0 +1,674 @@ +""" +Unit tests for owilix.core.tasks.remote module. + +Tests pure/helper functions without external dependencies, +and command-level functions with mocked manager/repository objects. +""" +import json +import os +import tempfile +from collections import Counter +from types import SimpleNamespace +from unittest.mock import MagicMock, patch, PropertyMock + +import pytest + +from owilix.core.tasks.remote import ( + _format_size, + _markdown_table, + _extract_language_from_path, + _build_host, + _slugify_command, + _summary_output_path, + _hosts_cache_root, + _hosts_cache_path, + _stats_db_path, + _host_domain_from_host, + _load_hosts_counter, + _write_hosts_counter, + _hosts_counter_from_stats, + _open_stats_db, + _migrate_hosts_cache_json_to_db, + _aggregate_hosts_db, + _metadata_totals, + _stats_totals, + _diff_style, + _format_top_languages, + _aggregate_local_hosts_cache, + _generate_readme_markdown, + remote_doctor, + remote_pull, + remote_push, + remote_remove, +) +from owilix.core.types import CommandResult + + +# --------------------------------------------------------------------------- +# _format_size +# --------------------------------------------------------------------------- +class TestFormatSize: + def test_bytes(self): + assert _format_size(0) == "0 B" + assert _format_size(512) == "512 B" + assert _format_size(1023) == "1023 B" + + def test_kib(self): + assert _format_size(1024) == "1.0 KiB" + assert _format_size(1536) == "1.5 KiB" + + def test_mib(self): + assert _format_size(1024**2) == "1.0 MiB" + assert _format_size(int(1.5 * 1024**2)) == "1.5 MiB" + + def test_gib(self): + assert _format_size(1024**3) == "1.00 GiB" + assert _format_size(int(2.5 * 1024**3)) == "2.50 GiB" + + +# --------------------------------------------------------------------------- +# _markdown_table +# --------------------------------------------------------------------------- +class TestMarkdownTable: + def test_empty_headers(self): + assert _markdown_table([], []) == "" + + def test_basic_table(self): + result = _markdown_table(["A", "B"], [["1", "2"], ["3", "4"]]) + lines = result.split("\n") + assert len(lines) == 4 # header, separator, 2 data rows + assert "| A | B |" in lines[0] + assert "| --- | --- |" in lines[1] + assert "| 1 | 2 |" in lines[2] + + def test_none_values_rendered_as_empty(self): + result = _markdown_table(["X"], [[None]]) + assert "| |" in result + + +# --------------------------------------------------------------------------- +# _extract_language_from_path +# --------------------------------------------------------------------------- +class TestExtractLanguageFromPath: + def test_three_letter_code(self): + assert _extract_language_from_path("/data/language=deu/part-0.parquet") == "deu" + + def test_two_letter_code(self): + assert _extract_language_from_path("/data/language=en/file.parquet") == "en" + + def test_no_language(self): + assert _extract_language_from_path("/data/part-0.parquet") is None + + def test_empty_string(self): + assert _extract_language_from_path("") is None + + +# --------------------------------------------------------------------------- +# _build_host +# --------------------------------------------------------------------------- +class TestBuildHost: + def test_full_host(self): + assert _build_host("www", "example", "com") == "www.example.com" + + def test_no_subdomain(self): + assert _build_host(None, "example", "com") == "example.com" + assert _build_host("", "example", "com") == "example.com" + + def test_no_suffix(self): + assert _build_host("www", "example", None) == "www.example" + assert _build_host("www", "example", "") == "www.example" + + def test_no_domain(self): + assert _build_host("www", None, "com") is None + assert _build_host("www", "", "com") is None + + def test_strips_whitespace(self): + assert _build_host(" www ", " example ", " com ") == "www.example.com" + + +# --------------------------------------------------------------------------- +# _slugify_command +# --------------------------------------------------------------------------- +class TestSlugifyCommand: + def test_basic(self): + assert _slugify_command("remote pull all") == "remote-pull-all" + + def test_special_chars(self): + result = _slugify_command("query --format=json") + assert all(c.isalnum() or c in ".-_" for c in result) + + def test_max_length(self): + result = _slugify_command("a" * 200, max_length=10) + assert len(result) <= 10 + + def test_empty(self): + assert _slugify_command(None) == "" + assert _slugify_command("") == "" + + +# --------------------------------------------------------------------------- +# _summary_output_path +# --------------------------------------------------------------------------- +class TestSummaryOutputPath: + def test_returns_json_file_in_summaries_dir(self): + path = _summary_output_path("/tmp/owi", command="remote-ls") + assert path.startswith("/tmp/owi/summaries/") + assert path.endswith(".json") + assert "remote-ls" in path + + def test_no_command(self): + path = _summary_output_path("/tmp/owi") + assert "stats-summary-" in os.path.basename(path) + + def test_default_owi_path(self): + path = _summary_output_path(None) + assert os.path.expanduser("~/.owi/summaries") in path + + +# --------------------------------------------------------------------------- +# _hosts_cache_root / _hosts_cache_path +# --------------------------------------------------------------------------- +class TestHostsCachePaths: + def test_cache_root(self): + assert _hosts_cache_root("/tmp/owi") == "/tmp/owi/summaries/hosts" + + def test_cache_root_default(self): + result = _hosts_cache_root(None) + assert result.endswith("summaries/hosts") + + def test_cache_path_uses_dataset_id(self): + ds = SimpleNamespace(metadata={"internalID": "abc-123"}) + path = _hosts_cache_path(ds, cache_dir="/tmp/hosts") + assert path == "/tmp/hosts/abc-123.jsonl" + + def test_cache_path_fallback_id(self): + ds = SimpleNamespace(metadata={"id": "fallback-id"}) + path = _hosts_cache_path(ds, cache_dir="/tmp/hosts") + assert path == "/tmp/hosts/fallback-id.jsonl" + + def test_stats_db_path(self): + assert _stats_db_path("/tmp/owi") == "/tmp/owi/summaries/stats.duckdb" + + +# --------------------------------------------------------------------------- +# _host_domain_from_host +# --------------------------------------------------------------------------- +class TestHostDomainFromHost: + def test_standard(self): + assert _host_domain_from_host("www.example.com") == "example.com" + + def test_subdomain(self): + assert _host_domain_from_host("a.b.example.co.uk") == "co.uk" + + def test_single_part(self): + assert _host_domain_from_host("localhost") == "localhost" + + def test_empty(self): + assert _host_domain_from_host("") is None + assert _host_domain_from_host(None) is None + + def test_trailing_dot(self): + assert _host_domain_from_host("example.com.") == "example.com" + + +# --------------------------------------------------------------------------- +# _load_hosts_counter / _write_hosts_counter +# --------------------------------------------------------------------------- +class TestHostsCounterIO: + def test_roundtrip(self, tmp_path): + path = str(tmp_path / "hosts.jsonl") + original = Counter({"example.com": 10, "test.org": 5}) + _write_hosts_counter(path, original) + loaded = _load_hosts_counter(path) + assert loaded == original + + def test_load_nonexistent(self, tmp_path): + assert _load_hosts_counter(str(tmp_path / "nope.jsonl")) is None + + def test_load_empty_file(self, tmp_path): + path = str(tmp_path / "empty.jsonl") + with open(path, "w") as f: + f.write("") + assert _load_hosts_counter(path) is None + + def test_load_malformed_json(self, tmp_path): + path = str(tmp_path / "bad.jsonl") + with open(path, "w") as f: + f.write("not json\n") + assert _load_hosts_counter(path) is None + + def test_write_creates_dirs(self, tmp_path): + path = str(tmp_path / "a" / "b" / "hosts.jsonl") + _write_hosts_counter(path, Counter({"x.com": 1})) + assert os.path.exists(path) + + +# --------------------------------------------------------------------------- +# _hosts_counter_from_stats +# --------------------------------------------------------------------------- +class TestHostsCounterFromStats: + def test_with_values_list(self): + stats = { + "hosts": { + "topk": 10, + "values": [ + {"host": "a.com", "count": 5}, + {"host": "b.org", "count": 3}, + ], + } + } + result = _hosts_counter_from_stats(stats) + assert result["a.com"] == 5 + assert result["b.org"] == 3 + + def test_with_flat_dict(self): + stats = {"hosts": {"a.com": 10, "b.org": 20}} + result = _hosts_counter_from_stats(stats) + assert result["a.com"] == 10 + + def test_none_input(self): + assert _hosts_counter_from_stats(None) is None + + def test_empty_dict(self): + assert _hosts_counter_from_stats({}) is None + + def test_no_hosts_key(self): + assert _hosts_counter_from_stats({"other": 1}) is None + + +# --------------------------------------------------------------------------- +# _metadata_totals / _stats_totals +# --------------------------------------------------------------------------- +class TestMetadataAndStatsTotals: + def test_metadata_totals(self): + md = {"totalSize": 100, "fileCount": 10, "objectCount": 50} + result = _metadata_totals(md) + assert result == {"size": 100, "files": 10, "objects": 50} + + def test_metadata_totals_missing_keys(self): + result = _metadata_totals({}) + assert result == {"size": None, "files": None, "objects": None} + + def test_stats_totals_with_data(self): + stats = { + "statistics": [ + { + "values": { + "eng": {"files": 5, "size": 100, "objects": 50}, + "deu": {"files": 3, "size": 80, "objects": 30}, + } + } + ] + } + result = _stats_totals(stats) + assert result["totals"]["files"] == 8 + assert result["totals"]["size"] == 180 + assert result["totals"]["objects"] == 80 + assert "eng" in result["languages"] + assert "deu" in result["languages"] + + def test_stats_totals_none(self): + result = _stats_totals(None) + assert result["totals"] == {"size": 0, "files": 0, "objects": 0} + assert result["languages"] == {} + + +# --------------------------------------------------------------------------- +# _diff_style +# --------------------------------------------------------------------------- +class TestDiffStyle: + def test_none_meta(self): + assert _diff_style(None, 100) is None + + def test_both_zero(self): + assert _diff_style(0, 0) == "green" + + def test_zero_meta_nonzero_stats(self): + assert _diff_style(0, 100) == "red" + + def test_exact_match(self): + assert _diff_style(100, 100) == "green" + + def test_small_diff(self): + # 4% difference + assert _diff_style(100, 104) == "orange3" + + def test_large_diff(self): + # 50% difference + assert _diff_style(100, 150) == "red" + + +# --------------------------------------------------------------------------- +# _format_top_languages +# --------------------------------------------------------------------------- +class TestFormatTopLanguages: + def test_empty(self): + assert _format_top_languages({}) == "n/a" + + def test_single_language(self): + result = _format_top_languages({"eng": {"size": 1024, "files": 2, "objects": 10}}) + assert "eng" in result + assert "2" in result + + def test_sorted_by_size(self): + langs = { + "small": {"size": 10, "files": 1, "objects": 1}, + "big": {"size": 10000, "files": 100, "objects": 1000}, + } + result = _format_top_languages(langs) + # "big" should appear first since sorted by size desc + assert result.index("big") < result.index("small") + + +# --------------------------------------------------------------------------- +# _aggregate_local_hosts_cache +# --------------------------------------------------------------------------- +class TestAggregateLocalHostsCache: + def test_nonexistent_dir(self, tmp_path): + result = _aggregate_local_hosts_cache(str(tmp_path / "nope"), 10, 10) + assert result is None + + def test_empty_dir(self, tmp_path): + result = _aggregate_local_hosts_cache(str(tmp_path), 10, 10) + assert result is None + + def test_with_cache_files(self, tmp_path): + # Write two cache files + _write_hosts_counter(str(tmp_path / "ds1.jsonl"), Counter({"a.com": 5, "b.org": 3})) + _write_hosts_counter(str(tmp_path / "ds2.jsonl"), Counter({"a.com": 2, "c.net": 1})) + + result = _aggregate_local_hosts_cache(str(tmp_path), hosts_topk=10, domains_topk=10) + assert result is not None + assert result["cacheFiles"]["total"] == 2 + assert result["cacheFiles"]["loaded"] == 2 + assert result["hosts"]["unique"] == 3 # a.com, b.org, c.net + + # Check host values sum correctly + host_map = {v["host"]: v["count"] for v in result["hosts"]["values"]} + assert host_map["a.com"] == 7 # 5 + 2 + + +class TestDuckDBHostMigration: + def test_migrate_jsonl_to_duckdb_and_aggregate(self, tmp_path): + cache_dir = tmp_path / "hosts" + cache_dir.mkdir() + _write_hosts_counter(str(cache_dir / "ds1.jsonl"), Counter({"a.com": 5, "b.org": 3})) + _write_hosts_counter(str(cache_dir / "ds2.jsonl"), Counter({"a.com": 2, "c.net": 1})) + + db_path = tmp_path / "stats.duckdb" + conn = _open_stats_db(str(db_path)) + try: + migration = _migrate_hosts_cache_json_to_db(str(cache_dir), conn) + assert migration["migrated"] == 2 + + summary = _aggregate_hosts_db( + conn=conn, + dataset_ids=["ds1", "ds2"], + topk=10, + domains_topk=10, + ) + assert summary is not None + assert summary["uniqueHosts"] == 3 + values = {item["host"]: item["count"] for item in summary["values"]} + assert values["a.com"] == 7 + assert summary["datasetsWithDb"] == 2 + finally: + conn.close() + + +# --------------------------------------------------------------------------- +# _generate_readme_markdown +# --------------------------------------------------------------------------- +class TestGenerateReadmeMarkdown: + def _make_dataset(self, **overrides): + defaults = { + "title": "Test Dataset", + "internalID": "test-123", + "id": "test-123", + "collectionName": "main", + "startDate": "2024-01-01", + "endDate": "2024-06-30", + "dataCenter": "dc1", + "resourceType": "Dataset", + "subResourceType": "WebCorpus", + "lastModified": "2024-07-01", + "descriptions": [{"description": "A test dataset", "descriptionType": "Abstract"}], + "creators": [{"name": "Alice", "affiliation": "Uni"}], + "rightsList": [{"rights": "CC-BY-4.0", "rightsURI": "https://creativecommons.org/licenses/by/4.0/"}], + "fundingReferences": [{"funderName": "EU", "awardNumber": "12345"}], + } + defaults.update(overrides) + md = MagicMock() + md.get.side_effect = lambda k, d=None: defaults.get(k, d) + md.__getitem__ = lambda self, k: defaults[k] + md.items.return_value = defaults.items() + ds = SimpleNamespace(metadata=md, dataCenter="dc1") + return ds + + def test_contains_title(self): + ds = self._make_dataset() + stats = {"statistics": [{"values": {"eng": {"files": 1, "size": 1024, "objects": 10}}}]} + result = _generate_readme_markdown(ds, stats) + assert "# OpenWebIndex: Test Dataset" in result + + def test_contains_statistics_table(self): + ds = self._make_dataset() + stats = {"statistics": [{"values": {"eng": {"files": 1, "size": 1024, "objects": 10}}}]} + result = _generate_readme_markdown(ds, stats) + assert "| eng |" in result + assert "## Statistics" in result + + def test_contains_pull_command(self): + ds = self._make_dataset() + stats = {"statistics": [{"values": {}}]} + result = _generate_readme_markdown(ds, stats) + assert "owilix remote pull all/id=test-123" in result + + def test_empty_stats(self): + ds = self._make_dataset() + stats = {"statistics": [{"values": {}}]} + result = _generate_readme_markdown(ds, stats) + assert "No statistics available." in result + + +# --------------------------------------------------------------------------- +# Helper: mock manager & dataset factories +# --------------------------------------------------------------------------- +def _make_mock_dataset(ds_id="ds-001", title="Dataset One", path="/data/ds1", + total_size=1024, file_count=5, object_count=100, + access="public", data_center="dc1", zone="zone1", + collection_name="main"): + """Create a mock dataset object mimicking the real Dataset interface.""" + metadata = { + "internalID": ds_id, + "id": ds_id, + "title": title, + "collectionName": collection_name, + "startDate": "2024-01-01", + "endDate": "2024-06-30", + "totalSize": total_size, + "fileCount": file_count, + "objectCount": object_count, + "zone": zone, + "access": access, + } + md = MagicMock() + md.get.side_effect = lambda k, d=None: metadata.get(k, d) + md.__getitem__ = lambda self, k: metadata[k] + md.items.return_value = metadata.items() + md.id = ds_id + md.collectionName = collection_name + + repo = MagicMock() + ds = SimpleNamespace( + metadata=md, + path=path, + access=access, + dataCenter=data_center, + zone=zone, + internalID=ds_id, + repository=repo, + ) + return ds + + +def _make_mock_manager(datasets=None, local_datasets=None, repos=None): + """Create a mock manager with remote_data and local attributes.""" + manager = MagicMock() + + if datasets is None: + datasets = [_make_mock_dataset()] + + manager.remote_data.list.return_value = datasets + manager.remote_data.get_repos.return_value = repos or {} + manager.remote_data.files.return_value = [] + manager.remote_data.exists_repo.return_value = True + + if local_datasets is not None: + manager.local.list.return_value = local_datasets + else: + manager.local.list.return_value = [] + + manager.local.files.return_value = [] + manager.parse_specifier.return_value = { + "data_center": "dc1", + "query": {"access": "public"}, + "day": None, + "duration": 0, + } + manager.owi_path = "/tmp/owi-test" + manager.name = "test-project" + manager.logpath = "/tmp/owi-test/logs" + manager.logfile = "/tmp/owi-test/logs/test.log" + + return manager + + +# --------------------------------------------------------------------------- +# remote_doctor +# --------------------------------------------------------------------------- +class TestRemoteDoctor: + def test_basic_no_repos(self): + manager = _make_mock_manager(repos={}) + result = remote_doctor(manager, console=MagicMock()) + assert result.success is True + + def test_as_json_output(self, tmp_path): + manager = _make_mock_manager(repos={}) + json_file = str(tmp_path / "doctor.json") + result = remote_doctor(manager, as_json=True, json_file=json_file, console=MagicMock()) + assert result.success is True + assert os.path.exists(json_file) + with open(json_file) as f: + data = json.load(f) + assert "timestamp" in data + + def test_with_repos(self): + repo_mock = MagicMock() + repo_mock.status.return_value = { + "status": True, + "backend": "irods", + "description": "test", + "message": "OK", + "user": "me", + "project": "proj", + "public": True, + } + manager = _make_mock_manager(repos={"test-dc": repo_mock}) + result = remote_doctor(manager, console=MagicMock()) + assert result.success is True + + def test_repo_error(self): + repo_mock = MagicMock() + repo_mock.status.side_effect = ConnectionError("refused") + manager = _make_mock_manager(repos={"broken": repo_mock}) + result = remote_doctor(manager, console=MagicMock()) + assert result.success is True # doctor still succeeds, reports errors + + def test_show_tokens(self): + manager = _make_mock_manager(repos={}) + manager.session = MagicMock() + manager.session.get_access_token.return_value = "a" * 50 + manager.session.get_refresh_token.return_value = "r" * 50 + console = MagicMock() + result = remote_doctor(manager, show_tokens=True, console=console) + assert result.success is True + + +# --------------------------------------------------------------------------- +# remote_pull +# --------------------------------------------------------------------------- +class TestRemotePull: + def test_no_datasets_found(self): + manager = _make_mock_manager(datasets=[]) + result = remote_pull(manager, specifier="dc1/public", auto_yes=True, console=MagicMock()) + assert result.success is True + assert "0 datasets" in result.msg + + @patch("owilix.core.tasks.remote.ask_yes_no", return_value=False) + def test_user_declines(self, mock_ask): + ds = _make_mock_dataset() + manager = _make_mock_manager(datasets=[ds]) + result = remote_pull(manager, specifier="dc1/public", console=MagicMock()) + assert result.success is True + # No downloads should happen since user declined + + @patch("owilix.core.tasks.remote.currentItemProgress") + @patch("owilix.core.tasks.remote.ask_yes_no", return_value=True) + def test_pull_up_to_date(self, mock_ask, mock_progress): + ds = _make_mock_dataset() + manager = _make_mock_manager(datasets=[ds]) + # remote files = local files → nothing to download + manager.remote_data.files.return_value = ["/data/ds1/file1.parquet"] + dest_ds = _make_mock_dataset(ds_id="ds-001", path="/local/ds1") + manager.local.list.return_value = [dest_ds] + manager.local.files.return_value = ["/local/ds1/file1.parquet"] + result = remote_pull( + manager, specifier="dc1/public", + files="['**/*']", console=MagicMock() + ) + assert result.success is True + + +# --------------------------------------------------------------------------- +# remote_push +# --------------------------------------------------------------------------- +class TestRemotePush: + def test_no_local_datasets(self): + manager = _make_mock_manager(local_datasets=[]) + result = remote_push(manager, specifier="dc1/public", auto_yes=True, console=MagicMock()) + assert result.success is True + + @patch("owilix.core.tasks.remote.ask_yes_no", return_value=False) + def test_user_declines(self, mock_ask): + ds = _make_mock_dataset() + manager = _make_mock_manager(local_datasets=[ds]) + result = remote_push(manager, specifier="dc1/public", console=MagicMock()) + assert result.success is True + + +# --------------------------------------------------------------------------- +# remote_remove +# --------------------------------------------------------------------------- +class TestRemoteRemove: + def test_no_datasets(self): + manager = _make_mock_manager(datasets=[]) + result = remote_remove(manager, specifier="dc1/public", auto_yes=True, console=MagicMock()) + assert result.success is True + assert "0 datasets" in result.msg + + @patch("owilix.core.tasks.remote.ask_yes_no", return_value=False) + def test_user_declines(self, mock_ask): + ds = _make_mock_dataset() + manager = _make_mock_manager(datasets=[ds]) + result = remote_remove(manager, specifier="dc1/public", console=MagicMock()) + assert result.success is True + + @patch("owilix.core.tasks.remote.currentItemProgress") + @patch("owilix.core.tasks.remote.ask_yes_no", return_value=True) + def test_remove_with_auto_yes(self, mock_ask, mock_progress): + ds = _make_mock_dataset() + manager = _make_mock_manager(datasets=[ds]) + result = remote_remove(manager, specifier="dc1/public", auto_yes=True, console=MagicMock()) + assert result.success is True + ds.repository.delete.assert_called_once_with(ds) -- 2.51.2 From db65288c0e5d493eed1f103d3ac6153b717b2128 Mon Sep 17 00:00:00 2001 From: mgrani Date: Mon, 16 Feb 2026 15:14:04 +0100 Subject: [PATCH 2/6] feat(remote): add summarize-hosts jsonl workflow and align summarize docs --- docs/source/commands.md | 2 +- docs/source/dataset-structure-and-metadata.md | 60 +- docs/source/details/remote.md | 45 +- owilix/cli/remote.py | 215 ++- owilix/core/tasks/remote.py | 1597 ++++++++++++----- tests/owilix/core/tasks/test_remote.py | 189 ++ 6 files changed, 1583 insertions(+), 525 deletions(-) diff --git a/docs/source/commands.md b/docs/source/commands.md index 339df6e..cc30f9c 100644 --- a/docs/source/commands.md +++ b/docs/source/commands.md @@ -15,7 +15,7 @@ The OWIlix CLI provides a suite of commands for managing datasets, interacting w ## Detailed Documentation - **[Local Commands](details/local.md)**: `ls`, `rm`, `analyze`, `init` -- **[Remote Commands](details/remote.md)**: `ls`, `pull`, `push`, `diff`, `summarize`, `summarize-local`, `doctor`, `logout` +- **[Remote Commands](details/remote.md)**: `ls`, `pull`, `push`, `diff`, `summarize`, `summarize-hosts`, `doctor`, `logout` - **[Query Commands](details/query.md)**: - **[Slice](details/query_slice.md)**: Create new datasets from queries. - **[WARC](details/warc.md)**: WARC file extraction. diff --git a/docs/source/dataset-structure-and-metadata.md b/docs/source/dataset-structure-and-metadata.md index 4047863..12697bf 100644 --- a/docs/source/dataset-structure-and-metadata.md +++ b/docs/source/dataset-structure-and-metadata.md @@ -25,11 +25,12 @@ Datasets follow a standardized hierarchical structure in the repository: └── *.warc.gz # WARC archive files (raw crawl data) ``` -Local summary and host aggregations are persisted in DuckDB: +Local summary outputs and host caches are persisted as JSON/JSONL: ``` $OWS_OWI_PATH/summaries/ - ├── stats.duckdb # Local stats/hosts cache used by summarize commands + ├── hosts/ # Per-dataset host counters (.jsonl) + ├── collections/ # Persistent raw/sorted files for summarize-hosts └── stats-summary-*.json # Generated summary outputs ``` @@ -333,7 +334,7 @@ owilix uses a **DataCite-compatible metadata schema** for dataset description an #### Statistics -Available from dataset statistics artifacts (`stats.json` where present, plus local DuckDB cache in summary workflows) and aggregated in metadata: +Available from dataset statistics artifacts (`stats.json` where present, plus local host-cache JSONL files in summary workflows) and aggregated in metadata: - **totalSize**: Total dataset size in bytes ```json @@ -802,8 +803,6 @@ owi remote summarize all --summary --group-by collectionName # Grouped summary - **--hosts-topk**: Top-K hosts to include in stats.json (0 to disable; default: 1000) - **--hosts-only**: Skip datasets that already have local host statistics in the summary cache - When set, `--force` does not override the cache check; only datasets missing cached host stats are processed. -- **--hosts-summary-topk**: Aggregate host counters in summary output from local DuckDB cache (0 to disable) -- **--domains-topk**: Top-K domains to include in summary host aggregation (0 to disable) - **--markdown-report**: Print a Markdown report at the end (useful for CI logs) - **--languages-topk**: Top-K languages to include in summary/Markdown report (0 to disable; default: 30) @@ -868,9 +867,7 @@ owi remote pull all/id=b313af04-f101-11f0-89ba-02a47ca5d9fd 2. Extracts language from partition path (`language=XXX`) 3. Counts files and sizes per language 4. Optionally counts rows in parquet files (`--count-rows`) -5. Computes top hosts via parquet queries and stores counters in local DuckDB cache (`$OWS_OWI_PATH/summaries/stats.duckdb`) - -When present, legacy host cache files (`$OWS_OWI_PATH/summaries/hosts/.jsonl`) are migrated into DuckDB automatically. +5. Computes top hosts via parquet queries and stores counters in local host cache files (`$OWS_OWI_PATH/summaries/hosts/.jsonl`) **Generate statistics:** ```bash @@ -881,12 +878,7 @@ owi remote summarize all --count-rows # Include row counts (slower) The `--summary` flag aggregates statistics from multiple datasets into a single local summary file. This is useful for getting an overview of all datasets in a collection or datacenter. -Use `--hosts-summary-topk` to aggregate cached host counters across the matched datasets and include: -- total number of unique hosts -- top-K hosts by count -The aggregation uses local DuckDB cache (`$OWS_OWI_PATH/summaries/stats.duckdb`) and migrates legacy local host caches (`summaries/hosts/*.jsonl`) when found. - -Use `--topk-collection` to restrict top-K calculations (languages/hosts/domains) to a single collection name (default: `main`). +Host/domain top-K aggregation is handled by `owi remote summarize-hosts`. **Generate summary:** ```bash @@ -894,14 +886,13 @@ owi remote summarize all --summary # Generate summary for all da owi remote summarize all --summary -o summary.json # Custom output file owi remote summarize it4i:latest --summary # Summary for specific datacenter owi remote summarize all/collectionName=main --summary # Summary for collection -owi remote summarize all --summary --topk-collection main # Top-K stats for main only ``` **How it works:** 1. Lists all datasets matching the specifier 2. For each dataset: - - Reads existing local DuckDB stats cache if available - - Migrates existing `stats.json` and local legacy host caches into DuckDB when found + - Reads existing local `stats.json` if available + - Reads existing local host cache JSONL (`summaries/hosts/*.jsonl`) when needed - Generates statistics from parquet files if cache data is missing - Can be forced to regenerate with `--force` flag 3. Aggregates all statistics by language across datasets @@ -946,14 +937,6 @@ owi remote summarize all --summary --topk-collection main # Top-K stats for mai "objects": 100000 } }, - "topkCollectionName": "main", - "byLanguageTopkCollection": { - "eng": { - "files": 400, - "size": 15000000, - "objects": 200000 - } - }, "groupBy": "collectionName", "groups": { "main": { @@ -998,14 +981,12 @@ owi remote summarize all --summary --topk-collection main # Top-K stats for mai - **byLanguage**: Aggregated statistics by language code - Sorted alphabetically by language code - Contains combined counts from all datasets -- **topkCollectionName**: Collection name used for top-K calculations (if restricted) -- **byLanguageTopkCollection**: Aggregated language stats for the top-K collection - **datasets**: Per-dataset breakdown with: - Basic metadata (id, title, datacenter, collection, date range) - Language-specific statistics - Dataset-level totals - **hostsSummary**: Optional host aggregation with coverage counts - - Includes cache coverage based on local DuckDB-backed host statistics + - Includes coverage from local host cache JSONL + `stats.json` - **groupBy/groups**: Optional grouped totals when `--group-by` is provided - **topkByGroup**: Optional top-K matrices by group (domains/hosts) when `--group-by` is provided @@ -1025,9 +1006,9 @@ When running with `--summary`, the command displays: Fetching datasets for specifier all Found 10 datasets Processing datasets... ━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% - ✓ Read DuckDB stats for Dataset 1... + ✓ Read stats.json for Dataset 1... ○ Generating stats for Dataset 2... - ✓ Read DuckDB stats for Dataset 3... + ✓ Read stats.json for Dataset 3... Summary written to ~/.owi/summaries/stats-summary-20240116-120000-owi-remote-summarize-all-summary.json @@ -1055,19 +1036,20 @@ By Language: #### Local Hosts Summary -You can also generate a local-only summary from cached host counters in -`$OWS_OWI_PATH/summaries/stats.duckdb`. This avoids remote reads and inspects -local cached host statistics (migrating legacy `summaries/hosts/*.jsonl` files automatically when present). +Use remote metadata to define the dataset set, then aggregate local `stats.json` +host counters for matching local datasets. -**Generate local summary:** +**Generate local host summary:** ```bash -owi remote summarize-local -owi remote summarize-local --hosts-topk 200 --domains-topk 200 +owi remote summarize-hosts all +owi remote summarize-hosts all --collection main +owi remote summarize-hosts all --hosts-topk 200 --domains-topk 200 +owi remote summarize-hosts all --collection legal --create-missing-stats ``` -The local summary includes total unique hosts and domains, plus optional top-k -lists. The output is written to `$OWS_OWI_PATH/summaries/stats-summary--.json` -by default. +The command writes: +- one per-collection host stats JSON file with full sorted host/domain counts +- one consolidated summary JSON with top-k aggregates and a list of datasets missing local stats ### Python API Access diff --git a/docs/source/details/remote.md b/docs/source/details/remote.md index f158495..2b0d1e8 100644 --- a/docs/source/details/remote.md +++ b/docs/source/details/remote.md @@ -134,7 +134,7 @@ owi remote doctor --- -## `remote summarize` / `remote summarize-local` +## `remote summarize` / `remote summarize-hosts` Generate dataset statistics summaries and optional README files, and build aggregated reports. @@ -142,28 +142,59 @@ Generate dataset statistics summaries and optional README files, and build aggre ```bash owi remote summarize [SPECIFIER] [OPTIONS] -owi remote summarize-local [OPTIONS] +owi remote summarize-hosts [SPECIFIER] [OPTIONS] ``` ### Notes - `--summary` writes an aggregated JSON report (default under `$OWS_OWI_PATH/summaries/`). -- Summary host/domain aggregation is DuckDB-backed and stored in: - - `$OWS_OWI_PATH/summaries/stats.duckdb` -- Legacy local host cache files under `$OWS_OWI_PATH/summaries/hosts/*.jsonl` are migrated automatically when found. +- `summarize-hosts` uses remote dataset metadata as source-of-truth, then aggregates hosts from local `stats.json`/host cache files. +- It writes one per-collection host stats file plus a consolidated summary JSON. +- It reports datasets for which local host stats are missing. - Typical high-volume usage: ```bash owi remote summarize all:2026-02-12#10 \ --summary \ --group-by collectionName \ - --hosts-summary-topk 300 \ - --domains-topk 300 \ --languages-topk 40 ``` --- +### `remote summarize-hosts` options + +- `-o, --output FILE` - Output file for consolidated summary JSON. +- `--hosts-topk N` - Top-K hosts to keep in consolidated output per collection. +- `--domains-topk N` - Top-K domains to keep in consolidated output per collection. +- `--collection NAME` - Filter datasets by `collectionName`. +- `--create-missing-stats` - For local datasets missing `stats.json`/host stats, generate `stats.json` locally and include them. +- `--markdown-report` - Print markdown report at the end. + +### `remote summarize-hosts` examples + +```bash +# Process all datasets discovered via remote metadata +owi remote summarize-hosts all + +# Restrict to one collection +owi remote summarize-hosts all --collection main + +# Keep smaller top-k in the consolidated summary +owi remote summarize-hosts all --hosts-topk 200 --domains-topk 200 + +# Try to generate missing local stats.json before marking datasets as missing +owi remote summarize-hosts all --collection legal --create-missing-stats +``` + +### `remote summarize-hosts` output files + +- Consolidated summary JSON: `--output` path (or auto-generated in `$OWS_OWI_PATH/summaries/`). +- Per-collection aggregated JSONL: `stats-hosts-.jsonl` in the same directory as summary output. +- Persistent work files (raw/sorted TSV): `$OWS_OWI_PATH/summaries/collections/`. + +--- + ## `remote logout` Revoke the current access token, effectively logging out from remote services. diff --git a/owilix/cli/remote.py b/owilix/cli/remote.py index e3c4ca4..bd41158 100644 --- a/owilix/cli/remote.py +++ b/owilix/cli/remote.py @@ -10,6 +10,7 @@ Commands: from typing import Optional import typer import os +import sys from ._common.context import CLIContext, get_context from ._common.output import OutputWriter @@ -142,6 +143,7 @@ def ls( "endDate": str(ds.metadata.endDate) if ds.metadata.endDate else None, "size": ds.metadata.get('totalSize'), "fileCount": ds.metadata.get('fileCount'), + "objectCount": ds.metadata.get('objectCount'), "access": getattr(ds, 'access', None), } for ds in datasets_list @@ -381,10 +383,20 @@ def summarize( "--hosts-topk", help="Top-K hosts to include in stats.json (0 to disable)", ), - hosts_summary_topk: int = typer.Option( - 0, - "--hosts-summary-topk", - help="Aggregate cached host counters in summary output (0 to disable)", + hosts_only: bool = typer.Option( + False, + "--hosts-only", + help="Only regenerate stats/README if the local hosts cache is missing", + ), + languages_topk: int = typer.Option( + 30, + "--languages-topk", + help="Top-K languages to include in summary/Markdown report (0 to disable)", + ), + markdown_report: bool = typer.Option( + False, + "--markdown-report", + help="Print a Markdown report at the end (useful for CI logs)", ), ): """ @@ -412,18 +424,18 @@ def summarize( owi remote summarize all --dry-run # Preview without changes owi remote summarize all --no-create-readme # Skip README creation owi remote summarize all --details # Show dataset comparisons + owi remote summarize all --hosts-only # Skip datasets with local host cache owi remote summarize all --summary # Generate aggregated stats owi remote summarize all --summary -o summary.json # Custom output file owi remote summarize all --summary --group-by collectionName # Grouped summary """ cli_ctx: CLIContext = ctx.obj + command = "owi " + " ".join(sys.argv[1:]) + if summary: from owilix.core.tasks.remote import remote_readme_summary - if output is None: - output = os.path.join(cli_ctx.owi.owi_path, "summaries", "stats-summary.json") - result = remote_readme_summary( manager=cli_ctx.owi, specifier=specifier, @@ -433,7 +445,12 @@ def summarize( details=details, group_by=group_by, hosts_topk=hosts_topk, - hosts_summary_topk=hosts_summary_topk, + languages_topk=languages_topk, + hosts_summary_topk=0, + domains_topk=0, + topk_collection="all", + command=command, + markdown_report=markdown_report, console=cli_ctx.console, ) @@ -452,6 +469,7 @@ def summarize( create_readme=create_readme, details=details, hosts_topk=hosts_topk, + hosts_only=hosts_only, console=cli_ctx.console, auto_yes=yes or cli_ctx.auto_yes ) @@ -461,6 +479,183 @@ def summarize( raise typer.Exit(code=1) +def _summarize_hosts_impl( + ctx: typer.Context, + specifier: str = typer.Argument( + "all", + help="Dataset specifier for remote metadata lookup (e.g., 'all', 'lrz:latest', 'all/id=abc123')", + ), + output: Optional[str] = typer.Option( + None, + "--output", + "-o", + help="Output file for consolidated summary JSON", + ), + hosts_topk: int = typer.Option( + 1000, + "--hosts-topk", + help="Top-K hosts to include per collection in consolidated summary", + ), + domains_topk: int = typer.Option( + 1000, + "--domains-topk", + help="Top-K domains to include per collection in consolidated summary", + ), + collection: Optional[str] = typer.Option( + None, + "--collection", + help="Filter by collectionName (e.g., 'main')", + ), + create_missing_stats: bool = typer.Option( + False, + "--create-missing-stats", + help="If local dataset exists but stats.json/host stats are missing, generate stats.json locally", + ), + markdown_report: bool = typer.Option( + False, + "--markdown-report", + help="Print a Markdown report at the end (useful for CI logs)", + ), +): + """ + Summarize local host statistics by matching remote metadata to local stats.json files. + + The command: + 1. Lists datasets from remote metadata (same basis as `remote ls`) + 2. Looks up corresponding local datasets + 3. Reads local stats.json host counters + 4. Writes one joint sorted host file per collection + 5. Writes one consolidated summary file including datasets missing local stats + + Examples: + owi remote summarize-hosts + owi remote summarize-hosts all/collectionName=main --collection main + """ + cli_ctx: CLIContext = ctx.obj + from owilix.core.tasks.remote import summarize_local_host_stats + + command = "owi " + " ".join(sys.argv[1:]) + + result = summarize_local_host_stats( + manager=cli_ctx.owi, + specifier=specifier, + output_file=output, + hosts_topk=hosts_topk, + domains_topk=domains_topk, + collection_filter=collection, + create_missing_stats=create_missing_stats, + command=command, + markdown_report=markdown_report, + console=cli_ctx.console, + ) + + if not result.success: + cli_ctx.console.print(f"[red]Local host stats summary failed: {result.msg}[/red]") + raise typer.Exit(code=1) + + +@app.command("summarize-hosts") +def summarize_hosts( + ctx: typer.Context, + specifier: str = typer.Argument( + "all", + help="Dataset specifier for remote metadata lookup (e.g., 'all', 'lrz:latest', 'all/id=abc123')", + ), + output: Optional[str] = typer.Option( + None, + "--output", + "-o", + help="Output file for consolidated summary JSON", + ), + hosts_topk: int = typer.Option( + 1000, + "--hosts-topk", + help="Top-K hosts to include per collection in consolidated summary", + ), + domains_topk: int = typer.Option( + 1000, + "--domains-topk", + help="Top-K domains to include per collection in consolidated summary", + ), + collection: Optional[str] = typer.Option( + None, + "--collection", + help="Filter by collectionName (e.g., 'main')", + ), + create_missing_stats: bool = typer.Option( + False, + "--create-missing-stats", + help="If local dataset exists but stats.json/host stats are missing, generate stats.json locally", + ), + markdown_report: bool = typer.Option( + False, + "--markdown-report", + help="Print a Markdown report at the end (useful for CI logs)", + ), +): + _summarize_hosts_impl( + ctx=ctx, + specifier=specifier, + output=output, + hosts_topk=hosts_topk, + domains_topk=domains_topk, + collection=collection, + create_missing_stats=create_missing_stats, + markdown_report=markdown_report, + ) + + +@app.command("summarize-local-host-stats", hidden=True) +def summarize_local_host_stats( + ctx: typer.Context, + specifier: str = typer.Argument( + "all", + help="Dataset specifier for remote metadata lookup (e.g., 'all', 'lrz:latest', 'all/id=abc123')", + ), + output: Optional[str] = typer.Option( + None, + "--output", + "-o", + help="Output file for consolidated summary JSON", + ), + hosts_topk: int = typer.Option( + 1000, + "--hosts-topk", + help="Top-K hosts to include per collection in consolidated summary", + ), + domains_topk: int = typer.Option( + 1000, + "--domains-topk", + help="Top-K domains to include per collection in consolidated summary", + ), + collection: Optional[str] = typer.Option( + None, + "--collection", + help="Filter by collectionName (e.g., 'main')", + ), + create_missing_stats: bool = typer.Option( + False, + "--create-missing-stats", + help="If local dataset exists but stats.json/host stats are missing, generate stats.json locally", + ), + markdown_report: bool = typer.Option( + False, + "--markdown-report", + help="Print a Markdown report at the end (useful for CI logs)", + ), +): + _summarize_hosts_impl( + ctx=ctx, + specifier=specifier, + output=output, + hosts_topk=hosts_topk, + domains_topk=domains_topk, + collection=collection, + create_missing_stats=create_missing_stats, + markdown_report=markdown_report, + ) + + @app.command() def catalog( ctx: typer.Context, @@ -469,10 +664,10 @@ def catalog( ): """ Generate a catalog of remote datasets. - + Creates a CSV catalog of datasets matching the specifier, along with individual JSON files for each dataset in a directory named .json. - + Examples: owi remote catalog all owi remote catalog lrz:latest --file my_catalog.csv diff --git a/owilix/core/tasks/remote.py b/owilix/core/tasks/remote.py index ac00623..335a884 100644 --- a/owilix/core/tasks/remote.py +++ b/owilix/core/tasks/remote.py @@ -7,6 +7,7 @@ import inspect import os import json import re +import subprocess import tempfile from collections import Counter from concurrent.futures import ThreadPoolExecutor, as_completed @@ -22,6 +23,18 @@ from owilix.core.tasks.query_utils import extract_domain_components from owilix.core.utils import group_and_count_files, rel_can_path from owilix.cli._common.ui import currentItemProgress, ask_yes_no +_TLD_EXTRACTOR = None + + +def _get_tld_extractor(): + global _TLD_EXTRACTOR + if _TLD_EXTRACTOR is None: + import tldextract + # Keep default PSL behavior; instantiate once and reuse. + _TLD_EXTRACTOR = tldextract.TLDExtract(include_psl_private_domains=True) + return _TLD_EXTRACTOR + + def remote_pull( manager: Any, specifier: str, @@ -636,20 +649,89 @@ def _upsert_hosts_counter_db( conn: duckdb.DuckDBPyConnection, dataset_id: str, counter: Counter, + on_chunk_written: Optional[Any] = None, ) -> None: - conn.execute("DELETE FROM host_stats WHERE dataset_id = ?", [dataset_id]) - if not counter: - return - sql = "INSERT INTO host_stats(dataset_id, host, domain, count) VALUES (?, ?, ?, ?)" - rows: List[tuple] = [] - for host, count in counter.items(): - host_norm = str(host) - rows.append((dataset_id, host_norm, _host_domain_from_host(host_norm), int(count))) - if len(rows) >= 50000: + conn.execute("BEGIN TRANSACTION") + try: + conn.execute("DELETE FROM host_stats WHERE dataset_id = ?", [dataset_id]) + if not counter: + conn.execute("COMMIT") + return + + sql = "INSERT INTO host_stats(dataset_id, host, domain, count) VALUES (?, ?, ?, ?)" + rows: List[tuple] = [] + total_rows = len(counter) + written_rows = 0 + for host, count in counter.items(): + host_norm = str(host) + rows.append((dataset_id, host_norm, _host_domain_from_host(host_norm), int(count))) + if len(rows) >= 50000: + chunk_size = len(rows) + conn.executemany(sql, rows) + written_rows += chunk_size + if on_chunk_written: + on_chunk_written(written_rows, total_rows) + rows = [] + if rows: + chunk_size = len(rows) conn.executemany(sql, rows) - rows = [] - if rows: - conn.executemany(sql, rows) + written_rows += chunk_size + if on_chunk_written: + on_chunk_written(written_rows, total_rows) + conn.execute("COMMIT") + except Exception: + conn.execute("ROLLBACK") + raise + + +def _ingest_hosts_jsonl_to_db_sql( + conn: duckdb.DuckDBPyConnection, + dataset_id: str, + jsonl_path: str, +) -> int: + conn.execute("BEGIN TRANSACTION") + try: + conn.execute("DELETE FROM host_stats WHERE dataset_id = ?", [dataset_id]) + conn.execute( + """ + INSERT INTO host_stats(dataset_id, host, domain, count) + WITH src AS ( + SELECT json + FROM read_json(?, format='newline_delimited', columns={'json': 'JSON'}) + ), + obj AS ( + SELECT COALESCE(json_extract(json, '$.hosts'), json) AS hosts_json + FROM src + ), + pairs AS ( + SELECT + lower(regexp_replace(CAST(key AS VARCHAR), '^\\.+|\\.+$', '')) AS host, + try_cast(CAST(value AS VARCHAR) AS BIGINT) AS cnt + FROM obj, json_each(obj.hosts_json) + ) + SELECT + ? AS dataset_id, + host, + CASE + WHEN host IS NULL OR host = '' THEN NULL + WHEN strpos(host, '.') = 0 THEN host + ELSE regexp_extract(host, '([^.]+\\.[^.]+)$') + END AS domain, + cnt AS count + FROM pairs + WHERE host IS NOT NULL AND host <> '' AND cnt IS NOT NULL + """, + [jsonl_path, dataset_id], + ) + inserted_row = conn.execute( + "SELECT COUNT(*) FROM host_stats WHERE dataset_id = ?", + [dataset_id], + ).fetchone() + conn.execute("COMMIT") + return int(inserted_row[0]) if inserted_row else 0 + except Exception: + conn.execute("ROLLBACK") + raise def _read_hosts_counter_db( @@ -775,32 +857,94 @@ def _migrate_hosts_cache_json_to_db( cache_dir: str, conn: duckdb.DuckDBPyConnection, dataset_ids: Optional[set[str]] = None, + cache_files: Optional[List[str]] = None, + progress: Optional[Any] = None, + progress_task: Optional[Any] = None, ) -> Dict[str, int]: if not os.path.isdir(cache_dir): - return {"total": 0, "migrated": 0, "skipped": 0} - + return {"total": 0, "migrated": 0, "skipped": 0, "migratedFiles": []} + + names = cache_files if cache_files is not None else os.listdir(cache_dir) + jsonl_names = [name for name in names if name.endswith(".jsonl")] + existing_dataset_ids = { + str(row[0]) + for row in conn.execute("SELECT DISTINCT dataset_id FROM host_stats").fetchall() + if row and row[0] is not None + } total = 0 migrated = 0 skipped = 0 - for name in os.listdir(cache_dir): - if not name.endswith(".jsonl"): - continue + migrated_files: List[str] = [] + per_file_units = 100.0 + for name in jsonl_names: total += 1 dataset_id = os.path.splitext(name)[0] + file_progress = 0.0 + if progress is not None and progress_task is not None: + progress.update(progress_task, current_item=f"Checking {name[:40]}") if dataset_ids is not None and dataset_id not in dataset_ids: skipped += 1 + if progress is not None and progress_task is not None: + progress.update(progress_task, advance=per_file_units, current_item=f"Skip {name[:40]}") continue - if _dataset_hosts_count_in_db(conn, dataset_id) > 0: + if dataset_id in existing_dataset_ids: skipped += 1 + if progress is not None and progress_task is not None: + progress.update(progress_task, advance=per_file_units, current_item=f"Already in DB {name[:30]}") + continue + cache_path = os.path.join(cache_dir, name) + sql_ingested = False + try: + if progress is not None and progress_task is not None: + progress.update(progress_task, current_item=f"DuckDB ingest {name[:36]}") + inserted = _ingest_hosts_jsonl_to_db_sql(conn, dataset_id, cache_path) + if inserted > 0: + existing_dataset_ids.add(dataset_id) + migrated += 1 + migrated_files.append(name) + sql_ingested = True + if progress is not None and progress_task is not None: + progress.update(progress_task, advance=per_file_units, current_item=f"Migrated {name[:40]} ({inserted:,})") + continue + except Exception: + sql_ingested = False + + if sql_ingested: continue - counter = _load_hosts_counter(os.path.join(cache_dir, name)) + + if progress is not None and progress_task is not None: + step = per_file_units * 0.1 + progress.update(progress_task, advance=step, current_item=f"Loading {name[:40]}") + file_progress += step + counter = _load_hosts_counter(cache_path) if not counter: skipped += 1 + if progress is not None and progress_task is not None: + progress.update(progress_task, advance=max(0.0, per_file_units - file_progress), current_item=f"Invalid {name[:40]}") continue - _upsert_hosts_counter_db(conn, dataset_id, counter) + if progress is not None and progress_task is not None: + progress.update(progress_task, current_item=f"Inserting {name[:40]} ({len(counter):,} hosts)") + + def _on_chunk_written(written_rows: int, total_rows: int) -> None: + nonlocal file_progress + if progress is None or progress_task is None: + return + # Reserve 10% for loading and 90% for DB writes of this file. + target = per_file_units * (0.1 + 0.9 * (written_rows / max(1, total_rows))) + delta = target - file_progress + if delta > 0: + progress.update(progress_task, advance=delta, current_item=f"Inserting {name[:40]} ({written_rows:,}/{total_rows:,})") + file_progress = target + + _upsert_hosts_counter_db(conn, dataset_id, counter, on_chunk_written=_on_chunk_written) + existing_dataset_ids.add(dataset_id) migrated += 1 + migrated_files.append(name) + if progress is not None and progress_task is not None: + remaining = max(0.0, per_file_units - file_progress) + progress.update(progress_task, advance=remaining, current_item=f"Migrated {name[:40]}") - return {"total": total, "migrated": migrated, "skipped": skipped} + return {"total": total, "migrated": migrated, "skipped": skipped, "migratedFiles": migrated_files} def _aggregate_hosts_db( @@ -1083,11 +1227,15 @@ def _aggregate_local_hosts_cache( def _load_hosts_counter(path: str) -> Optional[Counter]: try: + last_line = None with open(path, "r", encoding="utf-8") as handle: - lines = [line.strip() for line in handle if line.strip()] - if not lines: + for line in handle: + stripped = line.strip() + if stripped: + last_line = stripped + if not last_line: return None - payload = json.loads(lines[-1]) + payload = json.loads(last_line) except Exception: return None @@ -1190,34 +1338,114 @@ def _aggregate_hosts_caches( ) -> Optional[Dict[str, Any]]: if (not topk or topk <= 0) and (not domains_topk or domains_topk <= 0): return None + counter, coverage = _hosts_counter_for_datasets( + datasets, + cache_dir=cache_dir or _hosts_cache_root(None), + console=console, + ) + if not counter: + return None - dataset_ids = [_dataset_id(d) for d in datasets] - own_conn = False - conn = db_conn - if conn is None: - if cache_dir: - conn = _open_stats_db(os.path.join(os.path.dirname(cache_dir), "stats.duckdb")) - else: - conn = _open_stats_db(_stats_db_path(None)) - own_conn = True - try: - _migrate_hosts_cache_json_to_db( + domain_counter = Counter() + for host, count in counter.items(): + domain = _host_domain_from_host(host) + if domain: + domain_counter[domain] += count + + hosts_values = [] + if topk and topk > 0: + hosts_values = [ + {"host": host, "count": count} + for host, count in counter.most_common(topk) + ] + + domains_values = [] + if domains_topk and domains_topk > 0: + domains_values = [ + {"domain": domain, "count": count} + for domain, count in domain_counter.most_common(domains_topk) + ] + + datasets_with_hosts = int(coverage.get("datasetsWithCache", 0)) + int(coverage.get("datasetsWithStats", 0)) + return { + "uniqueHosts": len(counter), + "topk": topk, + "values": hosts_values, + "uniqueDomains": len(domain_counter), + "domainsTopk": domains_topk, + "domains": domains_values, + "datasetsWithDb": 0, + "datasetsWithCache": int(coverage.get("datasetsWithCache", 0)), + "datasetsWithStats": int(coverage.get("datasetsWithStats", 0)), + "datasetsMissingHosts": int(coverage.get("datasetsMissingHosts", 0)), + "datasetsWithHosts": datasets_with_hosts, + } + + +def _aggregate_group_topk_caches( + datasets: List[Any], + group_by: str, + hosts_topk: int, + domains_topk: int, + cache_dir: Optional[str] = None, +) -> Dict[str, Any]: + result: Dict[str, Any] = {} + if not datasets: + return result + + group_datasets: Dict[str, List[Any]] = {} + for dataset in datasets: + key = dataset.metadata.get(group_by) + if key is None: + key = "unknown" + group_datasets.setdefault(str(key), []).append(dataset) + + groups = sorted(group_datasets.keys()) + host_counts_by_group: Dict[str, Counter] = {} + domain_counts_by_group: Dict[str, Counter] = {} + total_hosts = Counter() + total_domains = Counter() + + for group in groups: + group_counter, _ = _hosts_counter_for_datasets( + group_datasets[group], cache_dir=cache_dir or _hosts_cache_root(None), - conn=conn, - dataset_ids=set(dataset_ids), + console=None, ) - summary = _aggregate_hosts_db( - conn=conn, - dataset_ids=dataset_ids, - topk=topk, - domains_topk=domains_topk, - ) - if not summary and console: - console.print("[yellow]No host data found for host aggregation.[/yellow]") - return summary - finally: - if own_conn: - conn.close() + host_counts_by_group[group] = group_counter + total_hosts.update(group_counter) + + domain_counter = Counter() + for host, count in group_counter.items(): + domain = _host_domain_from_host(host) + if domain: + domain_counter[domain] += count + domain_counts_by_group[group] = domain_counter + total_domains.update(domain_counter) + + if domains_topk and domains_topk > 0: + domain_rows = [] + for domain, _ in total_domains.most_common(domains_topk): + domain_rows.append( + { + "domain": domain, + "counts": {group: int(domain_counts_by_group[group].get(domain, 0)) for group in groups}, + } + ) + result["domains"] = {"topk": domains_topk, "groups": groups, "rows": domain_rows} + + if hosts_topk and hosts_topk > 0: + host_rows = [] + for host, _ in total_hosts.most_common(hosts_topk): + host_rows.append( + { + "host": host, + "counts": {group: int(host_counts_by_group[group].get(host, 0)) for group in groups}, + } + ) + result["hosts"] = {"topk": hosts_topk, "groups": groups, "rows": host_rows} + + return result def _close_executor(db: OWIDuckDBSelectExecutor) -> None: @@ -1276,101 +1504,86 @@ def _generate_hosts_counter( db_conn: Optional[duckdb.DuckDBPyConnection] = None, ) -> Counter: dataset_id = _dataset_id(dataset) - own_conn = False - conn = db_conn - if conn is None: - conn = _open_stats_db(_stats_db_path(None)) - own_conn = True - try: - if not force: - cached_db = _read_hosts_counter_db(conn, dataset_id) - if cached_db is not None: - return cached_db - - cache_path = _hosts_cache_path(dataset, cache_dir=cache_dir) - if not force and os.path.exists(cache_path): - cached = _load_hosts_counter(cache_path) - if cached is not None: - _upsert_hosts_counter_db(conn, dataset_id, cached) - return cached - - pq_files = _dataset_parquet_files(dataset) - if not pq_files: - return Counter() - - def run_query(select_clause: str) -> Counter: - sql = (OWIlixSQLQuery.from_templates("pq_select") - .select(select_clause) - .where("") - .groupby("") - .partitioned_by("") - .postfix("") - .limit(None)) - counter = Counter() - db = OWIDuckDBSelectExecutor( - pq_files, - sql, - pq_batch_size=10, - batch_size=1000, - prefetch=1, - ) - try: - for results in db.query_aggregator(): - if not results.success: - continue - for row in results.rows: - if select_clause == "url": - url = row.get("url") if isinstance(row, dict) else getattr(row, "url", None) - if not url: - continue - parts = extract_domain_components(str(url)) - if not parts: - continue - host = _build_host( - parts.get("url_subdomain"), - parts.get("url_domain"), - parts.get("url_suffix"), - ) + cache_path = _hosts_cache_path(dataset, cache_dir=cache_dir) + if not force and os.path.exists(cache_path): + cached = _load_hosts_counter(cache_path) + if cached is not None: + return cached + + pq_files = _dataset_parquet_files(dataset) + if not pq_files: + return Counter() + + def run_query(select_clause: str) -> Counter: + sql = (OWIlixSQLQuery.from_templates("pq_select") + .select(select_clause) + .where("") + .groupby("") + .partitioned_by("") + .postfix("") + .limit(None)) + counter = Counter() + db = OWIDuckDBSelectExecutor( + pq_files, + sql, + pq_batch_size=10, + batch_size=1000, + prefetch=1, + ) + try: + for results in db.query_aggregator(): + if not results.success: + continue + for row in results.rows: + if select_clause == "url": + url = row.get("url") if isinstance(row, dict) else getattr(row, "url", None) + if not url: + continue + parts = extract_domain_components(str(url)) + if not parts: + continue + host = _build_host( + parts.get("url_subdomain"), + parts.get("url_domain"), + parts.get("url_suffix"), + ) + else: + if isinstance(row, dict): + subdomain = row.get("url_subdomain") + domain = row.get("url_domain") + suffix = row.get("url_suffix") else: - if isinstance(row, dict): - subdomain = row.get("url_subdomain") - domain = row.get("url_domain") - suffix = row.get("url_suffix") - else: - subdomain = getattr(row, "url_subdomain", None) - domain = getattr(row, "url_domain", None) - suffix = getattr(row, "url_suffix", None) - host = _build_host(subdomain, domain, suffix) - if host: - counter[host] += 1 - finally: - _close_executor(db) - return counter + subdomain = getattr(row, "url_subdomain", None) + domain = getattr(row, "url_domain", None) + suffix = getattr(row, "url_suffix", None) + host = _build_host(subdomain, domain, suffix) + if host: + counter[host] += 1 + finally: + _close_executor(db) + return counter + try: + has_parts = _parquet_has_columns( + pq_files, + ["url_subdomain", "url_domain", "url_suffix"], + ) + if has_parts is False: + counter = run_query("url") + else: + counter = run_query("url_subdomain, url_domain, url_suffix") + except Exception as e: + if console: + console.print(f"[yellow]Host query failed ({e}); falling back to URL parsing.[/yellow]") try: - has_parts = _parquet_has_columns( - pq_files, - ["url_subdomain", "url_domain", "url_suffix"], - ) - if has_parts is False: - counter = run_query("url") - else: - counter = run_query("url_subdomain, url_domain, url_suffix") - except Exception as e: + counter = run_query("url") + except Exception as inner: if console: - console.print(f"[yellow]Host query failed ({e}); falling back to URL parsing.[/yellow]") - try: - counter = run_query("url") - except Exception as inner: - if console: - console.print(f"[red]Host query failed ({inner}); no host data generated.[/red]") - counter = Counter() + console.print(f"[red]Host query failed ({inner}); no host data generated.[/red]") + counter = Counter() - _upsert_hosts_counter_db(conn, dataset_id, counter) - return counter - finally: - if own_conn: - conn.close() + _write_hosts_counter(cache_path, counter) + return counter def _generate_stats_json( @@ -2146,226 +2359,192 @@ def remote_readme_summary( if output_file is None: output_file = _summary_output_path(getattr(manager, "owi_path", None), command=command) hosts_cache_dir = _hosts_cache_root(getattr(manager, "owi_path", None)) - stats_db = _stats_db_path(getattr(manager, "owi_path", None)) - db_conn = _open_stats_db(stats_db) - - try: - spec = manager.parse_specifier(specifier) - datasets = manager.remote_data.list( - spec.pop("data_center", None), - spec.get("query", {}).pop("access", "public"), - day=spec.pop("day", None), - duration=spec.pop("duration", 0), - query=spec.pop("query", {}), - ) - console.print(f"Found {len(datasets)} datasets") - if not datasets: - console.print("[yellow]No datasets found matching specifier[/yellow]") - return CommandResult(success=True, object=None, msg="No datasets to summarize") - - aggregated_by_language: Dict[str, Dict] = {} - aggregated_by_language_topk: Dict[str, Dict] = {} - dataset_summaries: List[Dict] = [] - grouped: Dict[str, Dict] = {} - group_dataset_ids: Dict[str, List[str]] = {} - detail_rows: List[Dict[str, Any]] = [] - totals_metadata = {"size": 0, "files": 0, "objects": 0} - totals_stats = {"size": 0, "files": 0, "objects": 0} - processed = 0 - errors = 0 - - topk_collection_name = None - if topk_collection is not None: - candidate = str(topk_collection).strip() - if candidate and candidate.lower() not in ("all", "*", "any", "none"): - topk_collection_name = candidate - - migration = _migrate_hosts_cache_json_to_db( - cache_dir=hosts_cache_dir, - conn=db_conn, - dataset_ids={_dataset_id(d) for d in datasets}, - ) - if migration["migrated"] > 0: - console.print( - f"[green]Migrated {migration['migrated']} host cache files into {stats_db}[/green]" - ) - - with currentItemProgress() as progress: - task = progress.add_task("Collecting statistics", total=len(datasets), current_item="Setup") - for d in datasets: - ds_id = _dataset_id(d) - title = d.metadata.get("title", "Untitled") - data_center = getattr(d, "dataCenter", None) or d.metadata.get("dataCenter", "unknown") - collection_name = d.metadata.get("collectionName", "unknown") - collection_name_norm = str(collection_name) - start_date = str(d.metadata.get("startDate", "?"))[:10] - end_date = str(d.metadata.get("endDate", "?"))[:10] - progress.update(task, current_item=f"Processing {title[:30]}...") + spec = manager.parse_specifier(specifier) + datasets = manager.remote_data.list( + spec.pop("data_center", None), + spec.get("query", {}).pop("access", "public"), + day=spec.pop("day", None), + duration=spec.pop("duration", 0), + query=spec.pop("query", {}), + ) + console.print(f"Found {len(datasets)} datasets") + if not datasets: + console.print("[yellow]No datasets found matching specifier[/yellow]") + return CommandResult(success=True, object=None, msg="No datasets to summarize") + + aggregated_by_language: Dict[str, Dict] = {} + aggregated_by_language_topk: Dict[str, Dict] = {} + dataset_summaries: List[Dict] = [] + grouped: Dict[str, Dict] = {} + group_dataset_ids: Dict[str, List[str]] = {} + detail_rows: List[Dict[str, Any]] = [] + totals_metadata = {"size": 0, "files": 0, "objects": 0} + totals_stats = {"size": 0, "files": 0, "objects": 0} + processed = 0 + errors = 0 - stats = None - stats_generated = False - if not force: - stats = _read_stats_db(db_conn, ds_id, hosts_topk=hosts_topk) - if stats: - console.print(f" [green]✓[/green] Read DuckDB stats for {title[:40]}...") + topk_collection_name = None + if topk_collection is not None: + candidate = str(topk_collection).strip() + if candidate and candidate.lower() not in ("all", "*", "any", "none"): + topk_collection_name = candidate - if not force and stats is None: - stats = _read_stats_json(d, console) - if stats: - _upsert_dataset_stats_db(db_conn, d, stats, source="stats.json", replace_hosts=False) - console.print(f" [green]✓[/green] Migrated stats.json for {title[:40]}...") + with currentItemProgress() as progress: + task = progress.add_task("Collecting statistics", total=len(datasets), current_item="Setup") + for d in datasets: + ds_id = _dataset_id(d) + title = d.metadata.get("title", "Untitled") + data_center = getattr(d, "dataCenter", None) or d.metadata.get("dataCenter", "unknown") + collection_name = d.metadata.get("collectionName", "unknown") + collection_name_norm = str(collection_name) + start_date = str(d.metadata.get("startDate", "?"))[:10] + end_date = str(d.metadata.get("endDate", "?"))[:10] + progress.update(task, current_item=f"Processing {title[:30]}...") - try: - if stats is None: - console.print(f" [cyan]○[/cyan] Generating stats for {title[:40]}...") - stats = _generate_stats_json( - d, - console=console, - hosts_topk=hosts_topk, - cache_dir=hosts_cache_dir, - force=force, - db_conn=db_conn, - ) - stats_generated = True - elif hosts_topk and hosts_topk > 0 and not stats.get("hosts"): - stats = _generate_stats_json( - d, - console=console, - hosts_topk=hosts_topk, - cache_dir=hosts_cache_dir, - force=force, - db_conn=db_conn, - ) - stats_generated = True - except Exception as e: - console.print(f" [red]✗[/red] Error generating stats for {title}: {e}") - errors += 1 - progress.update(task, advance=1) - continue + stats = None + if not force: + stats = _read_stats_json(d, console) - if stats: - _upsert_dataset_stats_db( - db_conn, + try: + if stats is None: + console.print(f" [cyan]○[/cyan] Generating stats for {title[:40]}...") + stats = _generate_stats_json( d, - stats, - source="generated" if stats_generated else "cache", - replace_hosts=bool(stats_generated), + console=console, + hosts_topk=hosts_topk, + cache_dir=hosts_cache_dir, + force=force, + db_conn=None, + ) + elif hosts_topk and hosts_topk > 0 and not stats.get("hosts"): + stats = _generate_stats_json( + d, + console=console, + hosts_topk=hosts_topk, + cache_dir=hosts_cache_dir, + force=force, + db_conn=None, ) + except Exception as e: + console.print(f" [red]✗[/red] Error generating stats for {title}: {e}") + errors += 1 + progress.update(task, advance=1) + continue - stats_info = _stats_totals(stats) - lang_values = stats_info["languages"] - ds_total_files = stats_info["totals"]["files"] - ds_total_size = stats_info["totals"]["size"] - ds_total_objects = stats_info["totals"]["objects"] + stats_info = _stats_totals(stats) + lang_values = stats_info["languages"] + ds_total_files = stats_info["totals"]["files"] + ds_total_size = stats_info["totals"]["size"] + ds_total_objects = stats_info["totals"]["objects"] + + for lang, data in lang_values.items(): + files_count = data.get("files", 0) + size = data.get("size", 0) + objects = data.get("objects", 0) + if lang not in aggregated_by_language: + aggregated_by_language[lang] = {"files": 0, "size": 0, "objects": 0} + aggregated_by_language[lang]["files"] += files_count + aggregated_by_language[lang]["size"] += size + aggregated_by_language[lang]["objects"] += objects + if topk_collection_name is None or collection_name_norm.lower() == topk_collection_name.lower(): + if lang not in aggregated_by_language_topk: + aggregated_by_language_topk[lang] = {"files": 0, "size": 0, "objects": 0} + aggregated_by_language_topk[lang]["files"] += files_count + aggregated_by_language_topk[lang]["size"] += size + aggregated_by_language_topk[lang]["objects"] += objects + + dataset_summaries.append( + { + "id": ds_id, + "title": title, + "dataCenter": data_center, + "collectionName": collection_name, + "startDate": start_date, + "endDate": end_date, + "statistics": lang_values, + "totals": {"files": ds_total_files, "size": ds_total_size, "objects": ds_total_objects}, + } + ) + meta_totals = _metadata_totals(d.metadata) + if meta_totals.get("size") is not None: + totals_metadata["size"] += meta_totals["size"] + if meta_totals.get("objects") is not None: + totals_metadata["objects"] += meta_totals["objects"] + if meta_totals.get("files") is not None: + totals_metadata["files"] += meta_totals["files"] + + totals_stats["size"] += ds_total_size + totals_stats["objects"] += ds_total_objects + totals_stats["files"] += ds_total_files + + if group_by: + group_value = d.metadata.get(group_by) + if group_value is None: + group_value = "unknown" + group_key = str(group_value) + if group_key not in grouped: + grouped[group_key] = { + "datasetsCount": 0, + "totals": {"files": 0, "size": 0, "objects": 0}, + "byLanguage": {}, + "datasets": [], + } + grouped[group_key]["datasetsCount"] += 1 + grouped[group_key]["datasets"].append(ds_id) + group_dataset_ids.setdefault(group_key, []).append(ds_id) + grouped[group_key]["totals"]["files"] += ds_total_files + grouped[group_key]["totals"]["size"] += ds_total_size + grouped[group_key]["totals"]["objects"] += ds_total_objects for lang, data in lang_values.items(): - files_count = data.get("files", 0) - size = data.get("size", 0) - objects = data.get("objects", 0) - if lang not in aggregated_by_language: - aggregated_by_language[lang] = {"files": 0, "size": 0, "objects": 0} - aggregated_by_language[lang]["files"] += files_count - aggregated_by_language[lang]["size"] += size - aggregated_by_language[lang]["objects"] += objects - if topk_collection_name is None or collection_name_norm.lower() == topk_collection_name.lower(): - if lang not in aggregated_by_language_topk: - aggregated_by_language_topk[lang] = {"files": 0, "size": 0, "objects": 0} - aggregated_by_language_topk[lang]["files"] += files_count - aggregated_by_language_topk[lang]["size"] += size - aggregated_by_language_topk[lang]["objects"] += objects - - dataset_summaries.append( + if lang not in grouped[group_key]["byLanguage"]: + grouped[group_key]["byLanguage"][lang] = {"files": 0, "size": 0, "objects": 0} + grouped[group_key]["byLanguage"][lang]["files"] += data.get("files", 0) + grouped[group_key]["byLanguage"][lang]["size"] += data.get("size", 0) + grouped[group_key]["byLanguage"][lang]["objects"] += data.get("objects", 0) + + if details: + dataset_label = f"{title} ({ds_id})" + if not create_readme: + dataset_label = f"[red]{dataset_label}[/red]" + meta_size = meta_totals.get("size") + meta_objects = meta_totals.get("objects") + meta_files = meta_totals.get("files") + detail_rows.append( { - "id": ds_id, - "title": title, - "dataCenter": data_center, - "collectionName": collection_name, - "startDate": start_date, - "endDate": end_date, - "statistics": lang_values, - "totals": {"files": ds_total_files, "size": ds_total_size, "objects": ds_total_objects}, + "dataset": dataset_label, + "source": "metadata", + "size": _format_size(meta_size) if meta_size is not None else "n/a", + "objects": f"{meta_objects:,}" if meta_objects is not None else "n/a", + "files": f"{meta_files:,}" if meta_files is not None else "n/a", + "languages": "", + } + ) + size_style = _diff_style(meta_size, ds_total_size) + objects_style = _diff_style(meta_objects, ds_total_objects) + files_style = _diff_style(meta_files, ds_total_files) + size_value = _format_size(ds_total_size) + objects_value = f"{ds_total_objects:,}" + files_value = f"{ds_total_files:,}" + if size_style: + size_value = f"[{size_style}]{size_value}[/{size_style}]" + if objects_style: + objects_value = f"[{objects_style}]{objects_value}[/{objects_style}]" + if files_style: + files_value = f"[{files_style}]{files_value}[/{files_style}]" + detail_rows.append( + { + "dataset": dataset_label, + "source": "stats.json", + "size": size_value, + "objects": objects_value, + "files": files_value, + "languages": _format_top_languages(lang_values), } ) - meta_totals = _metadata_totals(d.metadata) - if meta_totals.get("size") is not None: - totals_metadata["size"] += meta_totals["size"] - if meta_totals.get("objects") is not None: - totals_metadata["objects"] += meta_totals["objects"] - if meta_totals.get("files") is not None: - totals_metadata["files"] += meta_totals["files"] - - totals_stats["size"] += ds_total_size - totals_stats["objects"] += ds_total_objects - totals_stats["files"] += ds_total_files - - if group_by: - group_value = d.metadata.get(group_by) - if group_value is None: - group_value = "unknown" - group_key = str(group_value) - if group_key not in grouped: - grouped[group_key] = { - "datasetsCount": 0, - "totals": {"files": 0, "size": 0, "objects": 0}, - "byLanguage": {}, - "datasets": [], - } - grouped[group_key]["datasetsCount"] += 1 - grouped[group_key]["datasets"].append(ds_id) - group_dataset_ids.setdefault(group_key, []).append(ds_id) - grouped[group_key]["totals"]["files"] += ds_total_files - grouped[group_key]["totals"]["size"] += ds_total_size - grouped[group_key]["totals"]["objects"] += ds_total_objects - for lang, data in lang_values.items(): - if lang not in grouped[group_key]["byLanguage"]: - grouped[group_key]["byLanguage"][lang] = {"files": 0, "size": 0, "objects": 0} - grouped[group_key]["byLanguage"][lang]["files"] += data.get("files", 0) - grouped[group_key]["byLanguage"][lang]["size"] += data.get("size", 0) - grouped[group_key]["byLanguage"][lang]["objects"] += data.get("objects", 0) - - if details: - dataset_label = f"{title} ({ds_id})" - if not create_readme: - dataset_label = f"[red]{dataset_label}[/red]" - meta_size = meta_totals.get("size") - meta_objects = meta_totals.get("objects") - meta_files = meta_totals.get("files") - detail_rows.append( - { - "dataset": dataset_label, - "source": "metadata", - "size": _format_size(meta_size) if meta_size is not None else "n/a", - "objects": f"{meta_objects:,}" if meta_objects is not None else "n/a", - "files": f"{meta_files:,}" if meta_files is not None else "n/a", - "languages": "", - } - ) - size_style = _diff_style(meta_size, ds_total_size) - objects_style = _diff_style(meta_objects, ds_total_objects) - files_style = _diff_style(meta_files, ds_total_files) - size_value = _format_size(ds_total_size) - objects_value = f"{ds_total_objects:,}" - files_value = f"{ds_total_files:,}" - if size_style: - size_value = f"[{size_style}]{size_value}[/{size_style}]" - if objects_style: - objects_value = f"[{objects_style}]{objects_value}[/{objects_style}]" - if files_style: - files_value = f"[{files_style}]{files_value}[/{files_style}]" - detail_rows.append( - { - "dataset": dataset_label, - "source": "stats.json", - "size": size_value, - "objects": objects_value, - "files": files_value, - "languages": _format_top_languages(lang_values), - } - ) - - processed += 1 - progress.update(task, advance=1) + processed += 1 + progress.update(task, advance=1) topk_dataset_ids = [_dataset_id(ds) for ds in datasets] if topk_collection_name is not None: @@ -2407,11 +2586,12 @@ def remote_readme_summary( summary["topkCollectionDatasetsCount"] = len(topk_dataset_ids) summary["byLanguageTopkCollection"] = dict(sorted(aggregated_by_language_topk.items())) - hosts_summary = _aggregate_hosts_db( - conn=db_conn, - dataset_ids=topk_dataset_ids, + hosts_summary = _aggregate_hosts_caches( + datasets=[ds for ds in datasets if _dataset_id(ds) in set(topk_dataset_ids)], topk=hosts_summary_topk, domains_topk=domains_topk, + cache_dir=hosts_cache_dir, + console=console, ) if hosts_summary: if topk_collection_name is not None: @@ -2421,11 +2601,12 @@ def remote_readme_summary( if group_by: summary["groupBy"] = group_by summary["groups"] = dict(sorted(grouped.items())) - group_topk = _aggregate_group_topk_db( - conn=db_conn, - group_dataset_ids=group_dataset_ids, + group_topk = _aggregate_group_topk_caches( + datasets=datasets, + group_by=group_by, hosts_topk=hosts_summary_topk, domains_topk=domains_topk, + cache_dir=hosts_cache_dir, ) if group_topk: summary["topkByGroup"] = group_topk @@ -2451,7 +2632,8 @@ def remote_readme_summary( console.print(f" Unique hosts: {hosts_summary['uniqueHosts']:,}") console.print( " Host data coverage: " - f"{hosts_summary.get('datasetsWithDb', 0):,} DuckDB, " + f"{hosts_summary.get('datasetsWithCache', 0):,} cache files, " + f"{hosts_summary.get('datasetsWithStats', 0):,} stats.json, " f"{hosts_summary['datasetsMissingHosts']:,} missing" ) @@ -2505,11 +2687,12 @@ def remote_readme_summary( md_lines.append("### Hosts Summary") md_lines.append( _markdown_table( - ["Unique Hosts", "Unique Domains", "Datasets in DuckDB", "Datasets Missing Hosts"], + ["Unique Hosts", "Unique Domains", "Datasets in Cache", "Datasets in stats.json", "Datasets Missing Hosts"], [[ f"{hosts_summary.get('uniqueHosts', 0):,}", f"{hosts_summary.get('uniqueDomains', 0):,}", - f"{hosts_summary.get('datasetsWithDb', 0):,}", + f"{hosts_summary.get('datasetsWithCache', 0):,}", + f"{hosts_summary.get('datasetsWithStats', 0):,}", f"{hosts_summary.get('datasetsMissingHosts', 0):,}", ]], ) @@ -2518,148 +2701,626 @@ def remote_readme_summary( console.print("\n" + "\n".join(md_lines), markup=False) return CommandResult(success=True, object=summary, msg=msg) - finally: - db_conn.close() -def local_hosts_summary( +def extract_domain_tldextract(host: str) -> Optional[str]: + """ + Extract domain using tldextract library for proper PSL support. + + Examples: + - example.co.uk → example.co.uk (not co.uk) + - www.example.github.io → example.github.io (not github.io) + - mail.example.ac.uk → example.ac.uk + """ + if not host: + return None + + extractor = _get_tld_extractor() + extracted = extractor(host) + + # Combine domain + suffix + if extracted.domain and extracted.suffix: + return f"{extracted.domain}.{extracted.suffix}" + elif extracted.domain: + return extracted.domain + else: + return None + + +def _dataset_id_from_metadata(metadata: Any) -> Optional[str]: + if metadata is None: + return None + if isinstance(metadata, dict): + value = metadata.get("internalID") or metadata.get("id") + return str(value) if value is not None else None + get_fn = getattr(metadata, "get", None) + if callable(get_fn): + value = get_fn("internalID") or get_fn("id") + if value is not None: + return str(value) + value = getattr(metadata, "internalID", None) or getattr(metadata, "id", None) + return str(value) if value is not None else None + + +def _collection_name_from_metadata(metadata: Any) -> str: + if metadata is None: + return "main" + if isinstance(metadata, dict): + return str(metadata.get("collectionName") or "main") + get_fn = getattr(metadata, "get", None) + if callable(get_fn): + return str(get_fn("collectionName", "main") or "main") + return str(getattr(metadata, "collectionName", "main") or "main") + + +def _safe_collection_filename(collection_name: str) -> str: + safe = re.sub(r"[^A-Za-z0-9._-]+", "-", collection_name.strip()) + return safe or "unknown" + + +def _host_to_surl(host: str) -> Optional[str]: + """ + Build a sort-friendly host key by reversing labels. + + Example: + - www.example.com -> com.example.www + """ + host_norm = str(host or "").strip().strip(".").lower() + if not host_norm: + return None + parts = [part for part in host_norm.split(".") if part] + if not parts: + return None + return ".".join(reversed(parts)) + + +def _surl_to_host(surl: str) -> Optional[str]: + value = str(surl or "").strip().strip(".").lower() + if not value: + return None + parts = [part for part in value.split(".") if part] + if not parts: + return None + return ".".join(reversed(parts)) + + +def _aggregate_collection_on_disk( + collection: str, + raw_tsv: str, + collection_output_jsonl: str, + temp_root: str, + hosts_topk: int, + domains_topk: int, + on_stage: Optional[Any] = None, +) -> Dict[str, Any]: + """ + Aggregate one collection using disk-based GNU tools. + """ + collection_tmp = os.path.join(temp_root, f"agg-{_safe_collection_filename(collection)}") + os.makedirs(collection_tmp, exist_ok=True) + + sorted_by_surl = os.path.join(collection_tmp, "hosts-sorted-surl.tsv") + aggregated_hosts = os.path.join(collection_tmp, "hosts-aggregated.tsv") + sorted_by_count = os.path.join(collection_tmp, "hosts-sorted-count.tsv") + domains_input = os.path.join(collection_tmp, "domains-input.tsv") + domains_sorted = os.path.join(collection_tmp, "domains-sorted.tsv") + domains_aggregated = os.path.join(collection_tmp, "domains-aggregated.tsv") + domains_sorted_count = os.path.join(collection_tmp, "domains-sorted-count.tsv") + + if on_stage: + on_stage("sort-by-surl") + subprocess.run( + ["sort", "-t", "\t", "-k1,1", "-k2,2", "-S", "1G", "-T", collection_tmp, raw_tsv, "-o", sorted_by_surl], + check=True, + ) + + awk_hosts = r""" +BEGIN { FS="\t"; OFS="\t" } +NR == 1 { + prev = $1; host = $2; domain = $3; cnt = $4 + 0; + next; +} +{ + if ($1 == prev) { + cnt += ($4 + 0); + } else { + print prev, host, domain, cnt; + prev = $1; host = $2; domain = $3; cnt = $4 + 0; + } +} +END { + if (NR > 0) { + print prev, host, domain, cnt; + } +} +""" + if on_stage: + on_stage("aggregate-hosts") + subprocess.run(["awk", awk_hosts, sorted_by_surl], check=True, stdout=open(aggregated_hosts, "w")) + + if on_stage: + on_stage("sort-hosts-by-count") + subprocess.run( + ["sort", "-t", "\t", "-k4,4nr", "-k1,1", "-S", "1G", "-T", collection_tmp, aggregated_hosts, "-o", sorted_by_count], + check=True, + ) + + # Convert sorted host TSV -> aggregated per-collection JSONL and top-K stats. + unique_hosts = 0 + hosts_topk_rows: List[Dict[str, Any]] = [] + if on_stage: + on_stage("write-jsonl-and-topk-hosts") + with open(sorted_by_count, "r") as in_handle, open(collection_output_jsonl, "w") as out_handle: + for line in in_handle: + line = line.rstrip("\n") + if not line: + continue + parts = line.split("\t") + if len(parts) < 4: + continue + surl, _host, domain, count_str = parts[0], parts[1], parts[2], parts[3] + try: + count = int(count_str) + except Exception: + continue + out_handle.write( + json.dumps( + {"surlHost": surl, "domain": domain or None, "count": count}, + ensure_ascii=False, + ) + + "\n" + ) + unique_hosts += 1 + if hosts_topk > 0 and len(hosts_topk_rows) < hosts_topk: + host = _surl_to_host(surl) + hosts_topk_rows.append({"host": host or surl, "count": count}) + + # Build domain counts on disk from aggregated host rows. + awk_domains_extract = r'BEGIN { FS="\t"; OFS="\t" } { if ($3 != "") print $3, $4 }' + if on_stage: + on_stage("extract-domains") + subprocess.run(["awk", awk_domains_extract, aggregated_hosts], check=True, stdout=open(domains_input, "w")) + if on_stage: + on_stage("sort-domains") + subprocess.run( + ["sort", "-t", "\t", "-k1,1", "-S", "1G", "-T", collection_tmp, domains_input, "-o", domains_sorted], + check=True, + ) + + awk_domains_agg = r""" +BEGIN { FS="\t"; OFS="\t" } +NR == 1 { prev = $1; cnt = $2 + 0; next } +{ + if ($1 == prev) { + cnt += ($2 + 0); + } else { + print prev, cnt; + prev = $1; cnt = $2 + 0; + } +} +END { if (NR > 0) print prev, cnt } +""" + if on_stage: + on_stage("aggregate-domains") + subprocess.run(["awk", awk_domains_agg, domains_sorted], check=True, stdout=open(domains_aggregated, "w")) + + if on_stage: + on_stage("sort-domains-by-count") + subprocess.run( + ["sort", "-t", "\t", "-k2,2nr", "-k1,1", "-S", "1G", "-T", collection_tmp, domains_aggregated, "-o", domains_sorted_count], + check=True, + ) + + unique_domains = 0 + domains_topk_rows: List[Dict[str, Any]] = [] + if on_stage: + on_stage("topk-domains") + with open(domains_sorted_count, "r") as handle: + for line in handle: + line = line.rstrip("\n") + if not line: + continue + parts = line.split("\t") + if len(parts) < 2: + continue + domain, count_str = parts[0], parts[1] + try: + count = int(count_str) + except Exception: + continue + unique_domains += 1 + if domains_topk > 0 and len(domains_topk_rows) < domains_topk: + domains_topk_rows.append({"domain": domain, "count": count}) + + if on_stage: + on_stage("done") + return { + "uniqueHosts": unique_hosts, + "uniqueDomains": unique_domains, + "hostsTopk": hosts_topk_rows, + "domainsTopk": domains_topk_rows, + } + + +def summarize_local_host_stats( manager: Any, + specifier: str = "all", output_file: Optional[str] = None, hosts_topk: int = 1000, domains_topk: int = 1000, + collection_filter: Optional[str] = None, + create_missing_stats: bool = False, command: Optional[str] = None, markdown_report: bool = False, console: Optional[Console] = None, ) -> CommandResult: """ - Analyze local host cache summaries and write a stats-summary.json file. + Summarize local host stats by matching remote metadata to local stats.json files. """ if console is None: console = Console() - cache_dir = _hosts_cache_root(getattr(manager, "owi_path", None)) - stats_db = _stats_db_path(getattr(manager, "owi_path", None)) - db_conn = _open_stats_db(stats_db) + console.print("[bold]Phase 1:[/bold] Fetching dataset metadata from remote...") + spec = manager.parse_specifier(specifier) + query = dict(spec.get("query", {})) + access = query.pop("access", "public") + remote_datasets = manager.remote_data.list( + spec.pop("data_center", None), + access, + day=spec.pop("day", None), + duration=spec.pop("duration", 0), + query=query, + ) + if collection_filter: + remote_datasets = [ + d for d in remote_datasets + if _collection_name_from_metadata(getattr(d, "metadata", None)) == collection_filter + ] + console.print(f"Remote datasets found: {len(remote_datasets):,}") + + console.print("[bold]Phase 2:[/bold] Matching remote datasets to local stats.json files...") + local_datasets = manager.local.list(access=access, query={}) + hosts_cache_dir = _hosts_cache_root(getattr(manager, "owi_path", None)) + local_by_exact: Dict[tuple[str, str], Any] = {} + local_by_id: Dict[str, Any] = {} + for dataset in local_datasets: + md = getattr(dataset, "metadata", None) + ds_id = _dataset_id_from_metadata(md) + if not ds_id: + continue + collection = _collection_name_from_metadata(md) + local_by_exact[(ds_id, collection)] = dataset + local_by_id.setdefault(ds_id, dataset) + + datasets_with_stats_by_collection: Dict[str, int] = {} + missing_datasets: List[Dict[str, Any]] = [] + datasets_with_local_stats = 0 + datasets_considered = 0 + owi_root = getattr(manager, "owi_path", None) or os.path.expanduser("~/.owi") + aggregation_tmp_root = os.path.join(owi_root, "summaries", "collections") + os.makedirs(aggregation_tmp_root, exist_ok=True) + collection_raw_files: Dict[str, str] = {} + try: - migration = _migrate_hosts_cache_json_to_db(cache_dir=cache_dir, conn=db_conn, dataset_ids=None) - if migration["migrated"] > 0: - console.print(f"[green]Migrated {migration['migrated']} cache files into {stats_db}[/green]") - - dataset_ids = [row[0] for row in db_conn.execute("SELECT DISTINCT dataset_id FROM host_stats").fetchall()] - db_summary = _aggregate_hosts_db( - conn=db_conn, - dataset_ids=dataset_ids, - topk=hosts_topk, - domains_topk=domains_topk, - ) - finally: - db_conn.close() + with currentItemProgress() as progress: + match_task = progress.add_task( + "Matching remote datasets", + total=max(1, len(remote_datasets)), + current_item="Starting", + ) + for remote_dataset in remote_datasets: + remote_md = getattr(remote_dataset, "metadata", None) + dataset_id = _dataset_id_from_metadata(remote_md) + collection = _collection_name_from_metadata(remote_md) + title = ( + remote_md.get("title", "Unknown") + if isinstance(remote_md, dict) + else (getattr(remote_md, "get", lambda *_: "Unknown")("title", "Unknown")) + ) + label = (dataset_id or title or "unknown")[:60] + progress.update(match_task, current_item=f"Checking {label}") + datasets_considered += 1 - summary = None - if db_summary: - loaded_datasets = len(dataset_ids) - summary = { - "cacheDir": cache_dir, - "cacheFiles": { - "total": migration.get("total", 0), - "loaded": loaded_datasets, - "skipped": migration.get("skipped", 0), - }, - "hosts": { - "unique": db_summary.get("uniqueHosts", 0), - "topk": hosts_topk, - "values": db_summary.get("values", []), - }, - "domains": { - "unique": db_summary.get("uniqueDomains", 0), - "topk": domains_topk, - "values": db_summary.get("domains", []), - }, - } - else: - summary = _aggregate_local_hosts_cache( - cache_dir=cache_dir, - hosts_topk=hosts_topk, - domains_topk=domains_topk, - console=console, - ) + if not dataset_id: + missing_datasets.append( + { + "title": title, + "collectionName": collection, + "reason": "missing-remote-dataset-id", + } + ) + progress.update(match_task, advance=1, current_item=f"Missing id: {label}") + continue - if not summary: - return CommandResult(success=True, object=None, msg="No local host summary data found") + cache_path = os.path.join(hosts_cache_dir, f"{dataset_id}.jsonl") + cache_counter = _load_hosts_counter(cache_path) if os.path.exists(cache_path) else None + hosts_counter_for_dataset: Optional[Counter] = None - summary["generated"] = __import__("datetime").datetime.now().isoformat() - summary["command"] = command + # Prefer full cache counters when present; fall back to local stats.json. + if cache_counter: + hosts_counter_for_dataset = Counter(cache_counter) + else: + local_dataset = local_by_exact.get((dataset_id, collection)) or local_by_id.get(dataset_id) + if local_dataset is None: + missing_datasets.append( + { + "datasetId": dataset_id, + "title": title, + "collectionName": collection, + "reason": "local-dataset-not-found", + } + ) + progress.update(match_task, advance=1, current_item=f"Missing local dataset: {label}") + continue - if output_file is None: - output_file = _summary_output_path( - getattr(manager, "owi_path", None), - command=command, - prefix="stats-summary", + stats_file = os.path.join(getattr(local_dataset, "path", ""), "stats.json") + if not stats_file or not os.path.exists(stats_file): + if create_missing_stats: + try: + generated_stats = _generate_stats_json( + local_dataset, + hosts_topk=max(hosts_topk, 1000), + cache_dir=hosts_cache_dir, + ) + if generated_stats: + with open(stats_file, "w") as handle: + json.dump(generated_stats, handle, indent=2) + hosts_values = generated_stats.get("hosts", {}).get("values", []) + if isinstance(hosts_values, list) and hosts_values: + hosts_counter_for_dataset = Counter() + for entry in hosts_values: + host = str(entry.get("host", "")).strip() + if not host: + continue + try: + count = int(entry.get("count", 0)) + except Exception: + count = 0 + if count > 0: + hosts_counter_for_dataset[host] += count + except Exception: + hosts_counter_for_dataset = None + if not hosts_counter_for_dataset: + missing_datasets.append( + { + "datasetId": dataset_id, + "title": title, + "collectionName": collection, + "reason": "local-stats-json-missing", + } + ) + progress.update(match_task, advance=1, current_item=f"Missing stats.json: {label}") + continue + + try: + if hosts_counter_for_dataset is None: + with open(stats_file, "r") as handle: + stats_payload = json.load(handle) + else: + stats_payload = None + except Exception: + if create_missing_stats: + try: + generated_stats = _generate_stats_json( + local_dataset, + hosts_topk=max(hosts_topk, 1000), + cache_dir=hosts_cache_dir, + ) + with open(stats_file, "w") as handle: + json.dump(generated_stats, handle, indent=2) + hosts_values = generated_stats.get("hosts", {}).get("values", []) + if isinstance(hosts_values, list) and hosts_values: + hosts_counter_for_dataset = Counter() + for entry in hosts_values: + host = str(entry.get("host", "")).strip() + if not host: + continue + try: + count = int(entry.get("count", 0)) + except Exception: + count = 0 + if count > 0: + hosts_counter_for_dataset[host] += count + except Exception: + hosts_counter_for_dataset = None + if not hosts_counter_for_dataset: + missing_datasets.append( + { + "datasetId": dataset_id, + "title": title, + "collectionName": collection, + "reason": "local-stats-json-invalid", + } + ) + progress.update(match_task, advance=1, current_item=f"Invalid stats.json: {label}") + continue + + if hosts_counter_for_dataset is None: + hosts_values = stats_payload.get("hosts", {}).get("values", []) + if (not isinstance(hosts_values, list) or len(hosts_values) == 0) and create_missing_stats: + try: + generated_stats = _generate_stats_json( + local_dataset, + hosts_topk=max(hosts_topk, 1000), + cache_dir=hosts_cache_dir, + ) + with open(stats_file, "w") as handle: + json.dump(generated_stats, handle, indent=2) + hosts_values = generated_stats.get("hosts", {}).get("values", []) + except Exception: + pass + if not isinstance(hosts_values, list) or len(hosts_values) == 0: + missing_datasets.append( + { + "datasetId": dataset_id, + "title": title, + "collectionName": collection, + "reason": "local-host-stats-missing", + } + ) + progress.update(match_task, advance=1, current_item=f"No host stats: {label}") + continue + + hosts_counter_for_dataset = Counter() + for entry in hosts_values: + host = str(entry.get("host", "")).strip() + if not host: + continue + try: + count = int(entry.get("count", 0)) + except Exception: + count = 0 + if count > 0: + hosts_counter_for_dataset[host] += count + + if not hosts_counter_for_dataset: + missing_datasets.append( + { + "datasetId": dataset_id, + "title": title, + "collectionName": collection, + "reason": "local-host-stats-missing", + } + ) + progress.update(match_task, advance=1, current_item=f"No host stats: {label}") + continue + + raw_tsv = collection_raw_files.get(collection) + if raw_tsv is None: + raw_tsv = os.path.join(aggregation_tmp_root, f"raw-{_safe_collection_filename(collection)}.tsv") + collection_raw_files[collection] = raw_tsv + + with open(raw_tsv, "a") as raw_handle: + for host, count in hosts_counter_for_dataset.items(): + surl = _host_to_surl(host) + if not surl: + continue + domain = extract_domain_tldextract(host) or "" + raw_handle.write(f"{surl}\t{host}\t{domain}\t{int(count)}\n") + + datasets_with_local_stats += 1 + datasets_with_stats_by_collection[collection] = datasets_with_stats_by_collection.get(collection, 0) + 1 + progress.update(match_task, advance=1, current_item=f"Matched {label}") + + console.print( + f"Datasets with local host stats: {datasets_with_local_stats:,} " + f"({len(missing_datasets):,} missing local stats)" ) - try: + console.print("[bold]Phase 3:[/bold] Aggregating and sorting on disk...") + if output_file is None: + output_file = _summary_output_path( + getattr(manager, "owi_path", None), + command=command, + prefix="stats-summary-local-host-stats", + ) + output_dir = os.path.dirname(output_file) if output_dir: os.makedirs(output_dir, exist_ok=True) - with open(output_file, "w") as handle: - json.dump(summary, handle, indent=2) - console.print(f"\n[green]Local summary written to {output_file}[/green]") - except Exception as e: - console.print(f"[red]Error writing local summary file: {e}[/red]") - return CommandResult(success=False, msg=f"Failed to write output file: {e}") - - console.print("\n[bold]Local Host Summary:[/bold]") - console.print(f" Cache files: {summary['cacheFiles']['loaded']:,}/{summary['cacheFiles']['total']:,} loaded") - console.print(f" Unique hosts: {summary['hosts']['unique']:,}") - console.print(f" Unique domains: {summary['domains']['unique']:,}") - - if markdown_report: - md_lines = [] - md_lines.append("## OWI Local Hosts Summary") - md_lines.append("") - md_lines.append(f"- Generated: {summary.get('generated')}") - if command: - md_lines.append(f"- Command: `{command}`") - md_lines.append("") - md_lines.append("### Cache Coverage") - md_lines.append(_markdown_table( - ["Total Cache Files", "Loaded", "Skipped"], - [[ - f"{summary['cacheFiles']['total']:,}", - f"{summary['cacheFiles']['loaded']:,}", - f"{summary['cacheFiles']['skipped']:,}", - ]], - )) - md_lines.append("") - md_lines.append("### Unique Counts") - md_lines.append(_markdown_table( - ["Unique Hosts", "Unique Domains"], - [[ - f"{summary['hosts']['unique']:,}", - f"{summary['domains']['unique']:,}", - ]], - )) - md_lines.append("") - domain_rows = [] - for item in summary.get("domains", {}).get("values", [])[:domains_topk]: - domain_rows.append([item.get("domain"), f"{item.get('count', 0):,}"]) - if domain_rows: - md_lines.append("### Top Domains") - md_lines.append(_markdown_table(["Domain", "Count"], domain_rows)) - md_lines.append("") + collection_files: Dict[str, str] = {} + collections_summary: Dict[str, Dict[str, Any]] = {} + collection_names = sorted(collection_raw_files.keys()) + with currentItemProgress() as progress: + phase3_task = progress.add_task( + "Aggregating collections on disk", + total=max(1, len(collection_names)), + current_item="Starting", + ) + for collection in collection_names: + raw_tsv = collection_raw_files[collection] + collection_file = os.path.join( + output_dir, + f"stats-hosts-{_safe_collection_filename(collection)}.jsonl", + ) - host_rows = [] - for item in summary.get("hosts", {}).get("values", [])[:hosts_topk]: - host_rows.append([item.get("host"), f"{item.get('count', 0):,}"]) - if host_rows: - md_lines.append("### Top Hosts") - md_lines.append(_markdown_table(["Host", "Count"], host_rows)) - md_lines.append("") + def _stage_update(stage: str) -> None: + progress.update( + phase3_task, + current_item=f"{collection}: {stage}", + ) + + agg_result = _aggregate_collection_on_disk( + collection=collection, + raw_tsv=raw_tsv, + collection_output_jsonl=collection_file, + temp_root=aggregation_tmp_root, + hosts_topk=max(0, hosts_topk), + domains_topk=max(0, domains_topk), + on_stage=_stage_update, + ) + collection_files[collection] = collection_file + collections_summary[collection] = { + "datasetsWithLocalStats": datasets_with_stats_by_collection.get(collection, 0), + "uniqueHosts": agg_result["uniqueHosts"], + "uniqueDomains": agg_result["uniqueDomains"], + "hostsTopk": agg_result["hostsTopk"], + "domainsTopk": agg_result["domainsTopk"], + } + progress.update( + phase3_task, + advance=1, + current_item=f"{collection}: done ({agg_result['uniqueHosts']:,} hosts)", + ) + + summary = { + "generated": __import__("datetime").datetime.now().isoformat(), + "command": command, + "method": "remote-metadata-local-host-stats-disk", + "specifier": specifier, + "collectionFilter": collection_filter, + "createMissingStats": bool(create_missing_stats), + "remoteDatasetsFound": len(remote_datasets), + "datasetsConsidered": datasets_considered, + "datasetsWithLocalStats": datasets_with_local_stats, + "datasetsMissingLocalStats": len(missing_datasets), + "missingDatasets": missing_datasets, + "collectionsProcessed": len(collections_summary), + "collectionFiles": collection_files, + "collections": collections_summary, + } + + try: + with open(output_file, "w") as handle: + json.dump(summary, handle, indent=2) + console.print(f"\n[green]Summary written to {output_file}[/green]") + except Exception as e: + console.print(f"[red]Error writing summary file: {e}[/red]") + return CommandResult(success=False, msg=f"Failed to write output file: {e}") + + console.print("\n[bold]Summary:[/bold]") + console.print(f" Remote datasets: {len(remote_datasets):,}") + console.print(f" Local stats found: {datasets_with_local_stats:,}") + console.print(f" Missing local stats: {len(missing_datasets):,}") + console.print(f" Collections: {len(collections_summary):,}") + for collection in sorted(collections_summary.keys()): + coll = collections_summary[collection] + console.print( + f" {collection}: {coll['uniqueHosts']:,} unique hosts, " + f"{coll['uniqueDomains']:,} unique domains " + f"({coll['datasetsWithLocalStats']:,} datasets)" + ) - console.print("\n" + "\n".join(md_lines), markup=False) + if markdown_report: + md_lines = [] + md_lines.append("## OWI Local Host Stats Summary") + md_lines.append("") + md_lines.append(f"- Generated: {summary.get('generated')}") + if command: + md_lines.append(f"- Command: `{command}`") + md_lines.append(f"- Specifier: `{specifier}`") + if collection_filter: + md_lines.append(f"- Collection filter: `{collection_filter}`") + md_lines.append(f"- Remote datasets found: {len(remote_datasets):,}") + md_lines.append(f"- Datasets with local stats: {datasets_with_local_stats:,}") + md_lines.append(f"- Datasets missing local stats: {len(missing_datasets):,}") + md_lines.append("") + console.print("\n" + "\n".join(md_lines), markup=False) - return CommandResult(success=True, object=summary, msg="Local host summary generated") + return CommandResult(success=True, object=summary, msg="Local host stats summary generated") + finally: + pass def remote_diff( diff --git a/tests/owilix/core/tasks/test_remote.py b/tests/owilix/core/tasks/test_remote.py index 4ef721f..7f6fbd6 100644 --- a/tests/owilix/core/tasks/test_remote.py +++ b/tests/owilix/core/tasks/test_remote.py @@ -36,6 +36,8 @@ from owilix.core.tasks.remote import ( _format_top_languages, _aggregate_local_hosts_cache, _generate_readme_markdown, + extract_domain_tldextract, + summarize_local_host_stats, remote_doctor, remote_pull, remote_push, @@ -672,3 +674,190 @@ class TestRemoteRemove: result = remote_remove(manager, specifier="dc1/public", auto_yes=True, console=MagicMock()) assert result.success is True ds.repository.delete.assert_called_once_with(ds) + + +# --------------------------------------------------------------------------- +# extract_domain_tldextract +# --------------------------------------------------------------------------- +class TestExtractDomainTldextract: + """Test proper TLD extraction with tldextract library.""" + + def test_standard_com(self): + assert extract_domain_tldextract("example.com") == "example.com" + + def test_www_subdomain(self): + assert extract_domain_tldextract("www.example.com") == "example.com" + + def test_co_uk_tld(self): + """Should return example.co.uk, not co.uk""" + assert extract_domain_tldextract("example.co.uk") == "example.co.uk" + + def test_subdomain_co_uk(self): + """Should return example.co.uk, not co.uk""" + assert extract_domain_tldextract("a.b.example.co.uk") == "example.co.uk" + + def test_github_io(self): + """Should return example.github.io, not github.io""" + assert extract_domain_tldextract("www.example.github.io") == "example.github.io" + + def test_ac_uk(self): + """Should return example.ac.uk, not ac.uk""" + assert extract_domain_tldextract("mail.example.ac.uk") == "example.ac.uk" + + def test_single_part(self): + """Single part hostnames return as-is""" + assert extract_domain_tldextract("localhost") == "localhost" + + def test_empty(self): + assert extract_domain_tldextract("") is None + assert extract_domain_tldextract(None) is None + + def test_ip_address(self): + """IP addresses should be handled gracefully""" + result = extract_domain_tldextract("192.168.1.1") + # tldextract may return the IP or None depending on version + assert result is None or result == "192.168.1.1" + + +# --------------------------------------------------------------------------- +# summarize_local_host_stats +# --------------------------------------------------------------------------- +class TestSummarizeLocalHostStats: + def test_aggregates_from_local_stats_using_remote_metadata(self, tmp_path): + output_file = tmp_path / "summary.json" + + # Local datasets with stats.json files + local_ds1_path = tmp_path / "public" / "main" / "ds1" + local_ds1_path.mkdir(parents=True) + with open(local_ds1_path / "stats.json", "w") as f: + json.dump({"hosts": {"values": [{"host": "a.com", "count": 3}, {"host": "b.org", "count": 1}]}}, f) + + local_ds2_path = tmp_path / "public" / "main" / "ds2" + local_ds2_path.mkdir(parents=True) + with open(local_ds2_path / "stats.json", "w") as f: + json.dump({"hosts": {"values": [{"host": "a.com", "count": 2}]}}, f) + + # Remote list result contains two datasets, only one has local stats + remote_ds1 = SimpleNamespace(metadata={"internalID": "ds1", "collectionName": "main", "title": "One"}) + remote_ds2 = SimpleNamespace(metadata={"internalID": "ds2", "collectionName": "main", "title": "Two"}) + remote_ds3 = SimpleNamespace(metadata={"internalID": "ds3", "collectionName": "main", "title": "Three"}) + + local_ds1 = SimpleNamespace(metadata={"internalID": "ds1", "collectionName": "main"}, path=str(local_ds1_path)) + local_ds2 = SimpleNamespace(metadata={"internalID": "ds2", "collectionName": "main"}, path=str(local_ds2_path)) + + manager = MagicMock() + manager.owi_path = str(tmp_path) + manager.parse_specifier.return_value = {"data_center": None, "query": {"access": "public"}, "day": None, "duration": 0} + manager.remote_data.list.return_value = [remote_ds1, remote_ds2, remote_ds3] + manager.local.list.return_value = [local_ds1, local_ds2] + + # Remove ds2 stats to simulate missing local stats + os.remove(local_ds2_path / "stats.json") + + result = summarize_local_host_stats( + manager=manager, + specifier="all", + output_file=str(output_file), + hosts_topk=10, + domains_topk=10, + console=MagicMock(), + ) + + assert result.success is True + assert output_file.exists() + + with open(output_file) as f: + summary = json.load(f) + + assert summary["method"] == "remote-metadata-local-host-stats-disk" + assert summary["remoteDatasetsFound"] == 3 + assert summary["datasetsWithLocalStats"] == 1 + assert summary["datasetsMissingLocalStats"] == 2 + assert "main" in summary["collections"] + assert summary["collections"]["main"]["uniqueHosts"] == 2 + + top_hosts = summary["collections"]["main"]["hostsTopk"] + assert top_hosts[0]["host"] == "a.com" + assert top_hosts[0]["count"] == 3 + + assert "main" in summary["collectionFiles"] + assert os.path.exists(summary["collectionFiles"]["main"]) + + def test_collection_filter(self, tmp_path): + output_file = tmp_path / "summary-filtered.json" + + main_path = tmp_path / "public" / "main" / "ds-main" + main_path.mkdir(parents=True) + with open(main_path / "stats.json", "w") as f: + json.dump({"hosts": {"values": [{"host": "main.com", "count": 4}]}}, f) + + other_path = tmp_path / "public" / "other" / "ds-other" + other_path.mkdir(parents=True) + with open(other_path / "stats.json", "w") as f: + json.dump({"hosts": {"values": [{"host": "other.com", "count": 9}]}}, f) + + remote_main = SimpleNamespace(metadata={"internalID": "ds-main", "collectionName": "main"}) + remote_other = SimpleNamespace(metadata={"internalID": "ds-other", "collectionName": "other"}) + local_main = SimpleNamespace(metadata={"internalID": "ds-main", "collectionName": "main"}, path=str(main_path)) + local_other = SimpleNamespace(metadata={"internalID": "ds-other", "collectionName": "other"}, path=str(other_path)) + + manager = MagicMock() + manager.owi_path = str(tmp_path) + manager.parse_specifier.return_value = {"data_center": None, "query": {"access": "public"}, "day": None, "duration": 0} + manager.remote_data.list.return_value = [remote_main, remote_other] + manager.local.list.return_value = [local_main, local_other] + + result = summarize_local_host_stats( + manager=manager, + specifier="all", + output_file=str(output_file), + collection_filter="main", + console=MagicMock(), + ) + + assert result.success is True + with open(output_file) as f: + summary = json.load(f) + assert summary["remoteDatasetsFound"] == 1 + assert "main" in summary["collections"] + assert "other" not in summary["collections"] + + def test_uses_hosts_cache_when_local_dataset_missing(self, tmp_path): + output_file = tmp_path / "summary-cache.json" + + cache_dir = tmp_path / "summaries" / "hosts" + cache_dir.mkdir(parents=True) + _write_hosts_counter( + str(cache_dir / "450feb34-2bb4-11f0-9523-0242ac140003.jsonl"), + Counter({"example.org": 7, "www.example.org": 3}), + ) + + remote_ds = SimpleNamespace( + metadata={ + "internalID": "450feb34-2bb4-11f0-9523-0242ac140003", + "collectionName": "cefal", + "title": "OWI-Open Web Index-cefal.owi@lrz-2025-05-07:2025-05-07", + } + ) + + manager = MagicMock() + manager.owi_path = str(tmp_path) + manager.parse_specifier.return_value = {"data_center": None, "query": {"access": "public"}, "day": None, "duration": 0} + manager.remote_data.list.return_value = [remote_ds] + manager.local.list.return_value = [] + + result = summarize_local_host_stats( + manager=manager, + specifier="all", + output_file=str(output_file), + console=MagicMock(), + ) + + assert result.success is True + with open(output_file) as f: + summary = json.load(f) + + assert summary["datasetsWithLocalStats"] == 1 + assert summary["datasetsMissingLocalStats"] == 0 + assert "cefal" in summary["collections"] + assert summary["collections"]["cefal"]["uniqueHosts"] == 2 -- 2.51.2 From 4500f5ad7e200a35aac04e50a6824395671e77b4 Mon Sep 17 00:00:00 2001 From: mgrani Date: Tue, 17 Feb 2026 09:22:50 +0100 Subject: [PATCH 3/6] Add admin fs diagnostics and harden remote upload/storage handling --- docs/source/commands.md | 8 +- docs/source/details/admin.md | 124 +++++ docs/source/details/remote.md | 129 +++++ owilix/cli/admin.py | 208 +++++++- owilix/cli/remote.py | 90 ++++ owilix/core/repository/lexis.py | 201 ++++++-- owilix/core/tasks/remote.py | 448 ++++++++++++++++++ .../owilix/core/repository/test_repository.py | 66 +++ tests/owilix/core/tasks/test_remote.py | 234 +++++++++ 9 files changed, 1463 insertions(+), 45 deletions(-) create mode 100644 docs/source/details/admin.md diff --git a/docs/source/commands.md b/docs/source/commands.md index cc30f9c..b393f8e 100644 --- a/docs/source/commands.md +++ b/docs/source/commands.md @@ -10,12 +10,13 @@ The OWIlix CLI provides a suite of commands for managing datasets, interacting w | **[remote](details/remote.md)** | Interact with remote data centers (pull, push, list). | [Read More](details/remote.md) | | **[query](details/query.md)** | SQL-based analysis and filtering (slice, stats, less). | [Read More](details/query.md) | | **[config](details/config.md)** | View and modify CLI configuration. | [Read More](details/config.md) | -| **admin** | Administrative tasks (logs, system stats). | *See below* | +| **[admin](details/admin.md)** | Administrative tasks, repository/path diagnostics, and filesystem inspection. | [Read More](details/admin.md) | ## Detailed Documentation - **[Local Commands](details/local.md)**: `ls`, `rm`, `analyze`, `init` -- **[Remote Commands](details/remote.md)**: `ls`, `pull`, `push`, `diff`, `summarize`, `summarize-hosts`, `doctor`, `logout` +- **[Remote Commands](details/remote.md)**: `ls`, `pull`, `push`, `upload`, `diff`, `summarize`, `summarize-hosts`, `doctor`, `logout` +- **[Admin Commands](details/admin.md)**: `repos`, `path`, `fs`, `logs`, `stats`, `check` - **[Query Commands](details/query.md)**: - **[Slice](details/query_slice.md)**: Create new datasets from queries. - **[WARC](details/warc.md)**: WARC file extraction. @@ -62,6 +63,9 @@ Admin commands are used for troubleshooting and system monitoring. - `owi admin logs [level]`: View logs. - `owi admin stats`: View system statistics. +- `owi admin repos`: List configured repositories and their status. +- `owi admin path [--repository ]`: Check whether a path exists in one/all repositories. +- `owi admin fs [options]`: Run low-level filesystem commands (`ls`, `exists`, `info`, `find`, `glob`, `cat`, `mkdir`, `cp`, `mv`, `rm`). ## Publishing to OpenSearch diff --git a/docs/source/details/admin.md b/docs/source/details/admin.md new file mode 100644 index 0000000..0aee637 --- /dev/null +++ b/docs/source/details/admin.md @@ -0,0 +1,124 @@ +# Admin Commands + +Admin commands (`owi admin`) support troubleshooting, repository inspection, and low-level filesystem diagnostics. + +## `admin repos` + +List configured repositories and their connection status. + +### Usage + +```bash +owi admin repos +``` + +--- + +## `admin path` + +Check whether a given path exists in one repository or across all configured repositories. + +### Usage + +```bash +owi admin path PATH [OPTIONS] +``` + +### Options + +- `--repository REPO` (`-r`): Restrict checks to a single repository. + +### Examples + +```bash +owi admin path /IT4ILexisV2/public/ --repository lexis +owi admin path /OWILRZZONE/LEXIS//public/ +``` + +--- + +## `admin fs` + +Run low-level filesystem operations on repository backends. + +### Usage + +```bash +owi admin fs OPERATION PATH [OPTIONS] +``` + +### Operations + +- Read-only: `ls`, `exists`, `info`, `find`, `glob`, `cat` +- Modifying: `mkdir`, `cp`, `mv`, `rm` + +### Options + +- `--repository REPO` (`-r`): Repository name (default: `lexis`) +- `--target-path PATH` (`-t`): Destination path for `cp` and `mv` +- `--pattern GLOB` (`-p`): Optional filter for `find`/`glob` +- `--recursive`: Recursive behavior for supported ops (`rm`, some copy/move/list cases) +- `--max-items N`: Max printed items for list-like ops (default: `50`) +- `--yes` (`-y`): Skip confirmation for modifying operations + +### Safeguards + +- Modifying operations (`mkdir`, `cp`, `mv`, `rm`) require an interactive confirmation unless `--yes` is used. + +### Examples + +```bash +owi admin fs exists /IT4ILexisV2/public/ --repository lexis +owi admin fs ls /OWILRZZONE/LEXIS//public/ --repository lexis --max-items 20 +owi admin fs glob "/OWILRZZONE/LEXIS//public//**/*.parquet" --repository lexis +``` + +--- + +## `admin logs` + +View application logs. + +### Usage + +```bash +owi admin logs [LOG_TYPE] [OPTIONS] +``` + +### Examples + +```bash +owi admin logs error +owi admin logs all --lines 100 +``` + +--- + +## `admin stats` + +Display usage statistics. + +### Usage + +```bash +owi admin stats +``` + +--- + +## `admin check` + +Validate and optionally repair metadata for datasets matching a specifier. + +### Usage + +```bash +owi admin check SPECIFIER [OPTIONS] +``` + +### Examples + +```bash +owi admin check lrz:latest +owi admin check all/collectionName=main --repair=infer --update=update +``` diff --git a/docs/source/details/remote.md b/docs/source/details/remote.md index 2b0d1e8..e3500bc 100644 --- a/docs/source/details/remote.md +++ b/docs/source/details/remote.md @@ -99,6 +99,135 @@ owi remote push "all/id=my-local-id" --datacenter it4i --- +## `remote upload` + +Create a dataset from metadata and upload all files from a local directory, or upload into an existing dataset ID. + +### Usage + +```bash +owi remote upload DIRECTORY [OPTIONS] +``` + +### Options + +- `--dataset-id ID`: Upload into an existing dataset (skips dataset creation). +- `--metadata-file FILE` (`-m`): Metadata JSON for dataset creation mode. +- `--update-metadata-from-file FILE`: Update dataset metadata from JSON file before upload. +- `--repository REPO` (`-r`): Target repository key (required in creation mode, optional with `--dataset-id`). +- `--zone ZONE` (`-z`): Target iRODS zone (required in creation mode). +- `--storage-name NAME`: Explicit storage name override for creation mode. +- `--storage-resource RESOURCE`: Explicit storage resource override for creation mode. +- `--collection NAME` (`-c`): `collectionName` for dataset metadata (defaults to `main` in creation mode). +- `--access ACCESS`: Visibility when creating dataset (`public` or `project`). +- `--yes` (`-y`): Skip confirmation. + +### Modes and required flags + +- **Create + upload mode** + - Required: `DIRECTORY`, `--repository`, `--zone`, `--metadata-file` + - Optional: `--collection`, `--access`, `--yes` +- **Existing dataset upload mode** + - Required: `DIRECTORY`, `--dataset-id` + - Optional: `--repository`, `--update-metadata-from-file`, `--collection`, `--yes` + +Notes: +- For existing datasets, `--access` is informational only; actual access is read from remote metadata. +- If `--repository` is omitted with `--dataset-id`, OWI tries to auto-detect the repository. +- `--collection` is written to `additionalMetadata.collectionName` during create mode and during `--update-metadata-from-file` updates. +- Metadata payloads are sanitized to remove empty-string/null fields before submission. +- `--storage-name` and `--storage-resource` must be provided together and override automatic zone-based storage selection. + +### Examples + +**Create a dataset and upload a folder:** +```bash +owi remote upload ./imprints \ + --repository lexis \ + --zone IT4ILexisV2 \ + --collection special \ + --access project \ + --metadata-file ./metadata.json +``` + +**Create with explicit storage name/resource override:** +```bash +owi remote upload ./imprints \ + --repository lexis \ + --zone IT4IZone \ + --storage-name "iRODS IT4I" \ + --storage-resource "ATR-25-2" \ + --metadata-file ./metadata.json +``` + +**Upload to an existing dataset ID only:** +```bash +owi remote upload ./imprints --dataset-id 450feb34-2bb4-11f0-9523-0242ac140003 +``` + +**Upload to existing dataset with explicit repository:** +```bash +owi remote upload ./imprints \ + --dataset-id 450feb34-2bb4-11f0-9523-0242ac140003 \ + --repository lexis +``` + +**Update metadata and upload to existing dataset:** +```bash +owi remote upload ./imprints \ + --dataset-id 450feb34-2bb4-11f0-9523-0242ac140003 \ + --update-metadata-from-file ./metadata.json +``` + +**Metadata-only update (empty directory):** +```bash +mkdir -p /tmp/owi-empty-upload +owi remote upload /tmp/owi-empty-upload \ + --dataset-id 450feb34-2bb4-11f0-9523-0242ac140003 \ + --update-metadata-from-file ./metadata.json +``` + +### Metadata file format + +- `--metadata-file` and `--update-metadata-from-file` accept JSON payloads compatible with LEXIS DataCite metadata. +- In create mode with `--access public`, DataCite metadata is required. +- You can include `additionalMetadata` (e.g. `collectionName`, `totalSize`, `fileCount`, `objectCount`). + +### Upload behavior + +- Upload is recursive for all files under `DIRECTORY`. +- Progress bars show both source item and resolved remote target path. +- Failed files are reported in the `Failed uploads` progress line and the command exits non-zero if any file failed. + +### Troubleshooting zone/resource placement + +If a dataset appears in an unexpected zone/resource, use admin diagnostics: + +```bash +# List configured repositories +owi admin repos + +# Check expected and observed paths +owi admin path /IT4ILexisV2/LEXIS//public/ --repository lexis +owi admin path /OWILRZZONE/LEXIS//public/ --repository lexis + +# Inspect files under a path +owi admin fs ls /OWILRZZONE/LEXIS//public/ --repository lexis +``` + +For deterministic placement in create mode, set storage explicitly: + +```bash +owi remote upload ./imprints \ + --repository lexis \ + --zone IT4IZone \ + --storage-name "iRODS IT4I" \ + --storage-resource "ATR-25-2" \ + --metadata-file ./metadata.json +``` + +--- + ## `remote diff` Compare datasets between local and remote repositories to see what is missing or different. diff --git a/owilix/cli/admin.py b/owilix/cli/admin.py index 7594f65..487583a 100644 --- a/owilix/cli/admin.py +++ b/owilix/cli/admin.py @@ -1,15 +1,13 @@ """ Admin Commands - Typer CLI for administrative operations. - -Commands: -- logs: View error and operation logs -- stats: Display usage statistics """ from typing import Optional from pathlib import Path import typer +from rich.table import Table from ._common.context import CLIContext +from ._common.ui import ask_yes_no # Create sub-app for admin commands app = typer.Typer( @@ -19,6 +17,207 @@ app = typer.Typer( ) +def _all_repo_names(cli_ctx: CLIContext) -> list[str]: + try: + return sorted(cli_ctx.owi.remote_data.get_repo_names()) + except Exception: + return [] + + +def _get_repo(cli_ctx: CLIContext, repository: str): + repo = cli_ctx.owi.remote_data.get_single_repo(repository) + if repo is None: + raise ValueError(f"Repository '{repository}' not found") + return repo + + +@app.command("repos") +def repos(ctx: typer.Context): + """ + List configured repositories and status. + """ + cli_ctx: CLIContext = ctx.obj + names = _all_repo_names(cli_ctx) + if not names: + cli_ctx.console.print("[yellow]No repositories configured[/yellow]") + return + + table = Table(title="Configured Repositories") + table.add_column("Name", style="cyan") + table.add_column("Backend", style="magenta") + table.add_column("Status", style="green") + table.add_column("Details", style="dim") + + for name in names: + try: + repo = _get_repo(cli_ctx, name) + backend = getattr(repo, "_backend_name", "unknown") + st = repo.status() if hasattr(repo, "status") else {"status": None, "message": "n/a"} + ok = bool(st.get("status")) + status_text = "ok" if ok else "error" + details = str(st.get("message") or "").strip() + table.add_row(name, str(backend), status_text, details) + except Exception as exc: + table.add_row(name, "unknown", "error", str(exc)) + + cli_ctx.console.print(table) + + +@app.command("path") +def path_check( + ctx: typer.Context, + path: str = typer.Argument(..., help="Remote path to check (e.g., /IT4ILexisV2/public/)"), + repository: Optional[str] = typer.Option(None, "--repository", "-r", help="Repository name (default: check all)"), +): + """ + Check whether a path exists in one repository or all configured repositories. + """ + cli_ctx: CLIContext = ctx.obj + names = [repository] if repository else _all_repo_names(cli_ctx) + if not names: + cli_ctx.console.print("[yellow]No repositories configured[/yellow]") + return + + table = Table(title=f"Path Check: {path}") + table.add_column("Repository", style="cyan") + table.add_column("Exists", style="green") + table.add_column("Type", style="yellow") + table.add_column("Info", style="dim") + + for name in names: + try: + repo = _get_repo(cli_ctx, name) + fs = getattr(repo, "fs", None) + if fs is None: + table.add_row(name, "no", "n/a", "repository has no filesystem") + continue + exists = fs.exists(path) + if not exists: + table.add_row(name, "no", "-", "") + continue + is_dir = fs.isdir(path) + info = fs.info(path) + kind = "directory" if is_dir else "file" + info_bits = [] + if isinstance(info, dict): + size = info.get("size") + if size is not None: + info_bits.append(f"size={size}") + table.add_row(name, "yes", kind, ", ".join(info_bits)) + except Exception as exc: + table.add_row(name, "error", "-", str(exc)) + + cli_ctx.console.print(table) + + +@app.command("fs") +def filesystem( + ctx: typer.Context, + operation: str = typer.Argument(..., help="Operation: ls, exists, info, find, glob, cat, mkdir, cp, mv, rm"), + path: str = typer.Argument(..., help="Path (source path for cp/mv)"), + repository: str = typer.Option("lexis", "--repository", "-r", help="Repository name"), + target_path: Optional[str] = typer.Option(None, "--target-path", "-t", help="Target path for cp/mv"), + pattern: Optional[str] = typer.Option(None, "--pattern", "-p", help="Pattern for glob/find"), + recursive: bool = typer.Option(False, "--recursive", help="Recursive remove or listing behavior where applicable"), + max_items: int = typer.Option(50, "--max-items", help="Maximum items to display for ls/find/glob"), + yes: bool = typer.Option(False, "--yes", "-y", help="Skip confirmation for modifying operations"), +): + """ + Run low-level filesystem operations on repository backends. + """ + cli_ctx: CLIContext = ctx.obj + op = operation.strip().lower() + mutable_ops = {"mkdir", "cp", "mv", "rm"} + read_ops = {"ls", "exists", "info", "find", "glob", "cat"} + allowed = mutable_ops | read_ops + if op not in allowed: + cli_ctx.console.print(f"[red]Unsupported operation '{operation}'. Allowed: {', '.join(sorted(allowed))}[/red]") + raise typer.Exit(code=1) + + repo = _get_repo(cli_ctx, repository) + fs = getattr(repo, "fs", None) + if fs is None: + cli_ctx.console.print(f"[red]Repository '{repository}' has no filesystem[/red]") + raise typer.Exit(code=1) + + if op in mutable_ops and not yes: + action = f"{op} on repository={repository} path={path}" + if target_path: + action += f" target={target_path}" + if not ask_yes_no(cli_ctx.console, f"Proceed with {action}?"): + cli_ctx.console.print("[yellow]Aborted[/yellow]") + raise typer.Exit(code=1) + + try: + if op == "exists": + cli_ctx.console.print(str(fs.exists(path))) + return + if op == "info": + cli_ctx.console.print(fs.info(path)) + return + if op == "ls": + items = fs.ls(path, detail=False) + for item in items[:max_items]: + cli_ctx.console.print(item) + if len(items) > max_items: + cli_ctx.console.print(f"[dim]... {len(items) - max_items} more[/dim]") + return + if op == "find": + found = fs.find(path, withdirs=recursive) + if pattern: + import fnmatch + found = [f for f in found if fnmatch.fnmatch(str(f), pattern)] + for item in found[:max_items]: + cli_ctx.console.print(item) + if len(found) > max_items: + cli_ctx.console.print(f"[dim]... {len(found) - max_items} more[/dim]") + return + if op == "glob": + pat = pattern or path + items = fs.glob(pat) + for item in items[:max_items]: + cli_ctx.console.print(item) + if len(items) > max_items: + cli_ctx.console.print(f"[dim]... {len(items) - max_items} more[/dim]") + return + if op == "cat": + with fs.open(path, "rb") as handle: + data = handle.read(4096) + try: + cli_ctx.console.print(data.decode("utf-8", errors="replace")) + except Exception: + cli_ctx.console.print(data) + return + + if op == "mkdir": + fs.makedirs(path, exist_ok=True) + cli_ctx.console.print(f"[green]created:[/green] {path}") + return + if op == "cp": + if not target_path: + raise ValueError("--target-path is required for cp") + if not hasattr(fs, "cp"): + raise ValueError("Filesystem does not support cp") + fs.cp(path, target_path, recursive=recursive) + cli_ctx.console.print(f"[green]copied:[/green] {path} -> {target_path}") + return + if op == "mv": + if not target_path: + raise ValueError("--target-path is required for mv") + if not hasattr(fs, "mv"): + raise ValueError("Filesystem does not support mv") + fs.mv(path, target_path, recursive=recursive) + cli_ctx.console.print(f"[green]moved:[/green] {path} -> {target_path}") + return + if op == "rm": + fs.rm(path, recursive=recursive) + cli_ctx.console.print(f"[green]removed:[/green] {path}") + return + except Exception as exc: + cli_ctx.console.print(f"[red]filesystem operation failed:[/red] {exc}") + raise typer.Exit(code=1) + + @app.command() def logs( ctx: typer.Context, @@ -286,4 +485,3 @@ def check( count_error += 1 cli_ctx.console.print(f"Check complete. {count_ok} OK, {count_error} issues found.") - diff --git a/owilix/cli/remote.py b/owilix/cli/remote.py index bd41158..caa5f04 100644 --- a/owilix/cli/remote.py +++ b/owilix/cli/remote.py @@ -264,6 +264,96 @@ def push( raise typer.Exit(code=1) +@app.command() +def upload( + ctx: typer.Context, + directory: str = typer.Argument(..., help="Local directory whose files should be uploaded recursively"), + repository: Optional[str] = typer.Option( + None, + "--repository", + "-r", + help="Target remote repository key (required for creation mode; optional with --dataset-id)", + ), + zone: Optional[str] = typer.Option( + None, + "--zone", + "-z", + help="Target iRODS zone (required for creation mode)", + ), + collection: Optional[str] = typer.Option( + None, + "--collection", + "-c", + help="collectionName in additionalMetadata (defaults to 'main' in creation mode)", + ), + access: str = typer.Option("project", "--access", help="Dataset visibility: public or project"), + metadata_file: Optional[str] = typer.Option( + None, + "--metadata-file", + "-m", + help="Path to metadata JSON file (required when creating a new dataset)", + ), + dataset_id: Optional[str] = typer.Option( + None, + "--dataset-id", + help="Upload into an existing remote dataset ID (skips dataset creation)", + ), + update_metadata_from_file: Optional[str] = typer.Option( + None, + "--update-metadata-from-file", + help="Update dataset metadata from JSON file before upload (works with --dataset-id)", + ), + storage_name: Optional[str] = typer.Option( + None, + "--storage-name", + help="Override storage name for dataset creation (must be used with --storage-resource)", + ), + storage_resource: Optional[str] = typer.Option( + None, + "--storage-resource", + help="Override storage resource for dataset creation (must be used with --storage-name)", + ), + yes: bool = typer.Option(False, "--yes", "-y", help="Skip confirmation prompt"), +): + """ + Upload files from a local directory to a remote dataset. + + Modes: + 1. Create + upload: provide --metadata-file (no --dataset-id) + 2. Existing dataset upload: provide --dataset-id (metadata file optional/ignored) + 3. Metadata update: provide --update-metadata-from-file to push DataCite/metadata JSON before upload + + Examples: + owi remote upload ./data --repository lexis --zone IT4ILexisV2 --collection special --metadata-file ./metadata.json + owi remote upload ./data --repository lexis --zone IT4IZone --storage-name \"iRODS IT4I\" --storage-resource \"ATR-25-2\" --metadata-file ./metadata.json + owi remote upload ./data --dataset-id + owi remote upload ./data --repository lexis --dataset-id + owi remote upload ./data --dataset-id --update-metadata-from-file ./metadata.json + """ + cli_ctx: CLIContext = ctx.obj + from owilix.core.tasks.remote import remote_upload + + result = remote_upload( + manager=cli_ctx.owi, + directory=directory, + metadata_file=metadata_file, + dataset_id=dataset_id, + update_metadata_from_file=update_metadata_from_file, + storage_name_override=storage_name, + storage_resource_override=storage_resource, + repository=repository, + zone=zone, + collection_name=collection, + access=access, + yes=yes or cli_ctx.auto_yes, + console=cli_ctx.console, + ) + + if not result.success: + cli_ctx.console.print(f"[red]Upload failed: {result.msg}[/red]") + raise typer.Exit(code=1) + + @app.command() def diff( ctx: typer.Context, diff --git a/owilix/core/repository/lexis.py b/owilix/core/repository/lexis.py index a7e7609..853974b 100644 --- a/owilix/core/repository/lexis.py +++ b/owilix/core/repository/lexis.py @@ -11,6 +11,7 @@ Uses: import logging import os import time +import fnmatch from datetime import datetime from typing import List, Union, Sequence @@ -22,6 +23,23 @@ from owilix.core.models.dataset import Dataset logger = logging.getLogger("owilix") +def _match_glob_pattern(rel_path: str, pattern: str) -> bool: + """ + Match glob patterns against a dataset-relative path. + Supports `**/` prefix as optional directory depth, so patterns like + `**/*.*` also match root files such as `README.md`. + """ + rel_path = (rel_path or "").lstrip("/") + pattern = (pattern or "").strip() + if not pattern: + return False + if fnmatch.fnmatch(rel_path, pattern): + return True + if pattern.startswith("**/"): + return fnmatch.fnmatch(rel_path, pattern[3:]) + return False + + class LexisRepository(AbstractRepository): """ No host/port/zone configuration needed - everything comes from LexisSession. @@ -58,6 +76,14 @@ class LexisRepository(AbstractRepository): self._ddi_api = None self._filesystem = None self._zone = kwargs.get("zone", None) # Optional zone override + self._default_zone = kwargs.get("default_zone", None) or self._zone + self._zones = kwargs.get("zones", {}) or {} + self._default_location_name = ( + kwargs.get("default_location_name", None) or kwargs.get("location_name", None) + ) + self._zone_filesystems = {} + self._irods_wrappers = {} + self._warned_unknown_zones = set() @property def lexis_session(self): @@ -77,41 +103,101 @@ class LexisRepository(AbstractRepository): @property def fs(self): - """Lazy-init Http2IrodsFileSystem with automatic token refresh.""" + """Lazy-init a routing filesystem that resolves per-zone Http2IrodsFileSystem.""" if self._filesystem is None: - from owilix.core.fsspec import Http2IrodsFileSystem + from owilix.core.fsspec import Http2IrodsFileSystem, ZoneRoutingFileSystem from owilix.core.repository.irods import IRODSOWI - - # Use IRODSOWI wrapper for automatic token refresh - # get_session_callable() returns a callable that checks token before each call - irods_wrapper = IRODSOWI(self.lexis_session) - - # Get base URL from session - irods_url = self.lexis_session.irods_http_api_url - - # Create filesystem with callable that refreshes token on each access - self._filesystem = Http2IrodsFileSystem( - irods_client=irods_wrapper.get_client(), - url_base=irods_url, - irods_client_factory=irods_wrapper.get_session_callable() + + def _get_fs_for_zone(zone: str) -> Http2IrodsFileSystem: + if zone not in self._zone_filesystems: + irods_url = self._get_zone_irods_url(zone) + irods_wrapper = IRODSOWI( + self.lexis_session, + irods_http_api_url=irods_url, + user_zone=zone + ) + self._zone_filesystems[zone] = Http2IrodsFileSystem( + irods_client=irods_wrapper.get_client(), + url_base=irods_url, + irods_client_factory=irods_wrapper.get_session_callable() + ) + self._irods_wrappers[zone] = irods_wrapper + return self._zone_filesystems[zone] + + known_zones = list(self._zones.keys()) if self._zones else None + self._filesystem = ZoneRoutingFileSystem( + get_fs_for_zone=_get_fs_for_zone, + default_zone=self.zone, + known_zones=known_zones ) - - # Store wrapper for potential future use - self._irods_wrapper = irods_wrapper return self._filesystem @property def zone(self) -> str: """Get iRODS zone from session or override.""" - if self._zone: - return self._zone + if self._default_zone: + return self._default_zone # Extract zone from session URL or use default # The zone is typically part of the iRODS path structure return getattr(self.lexis_session, '_zone', 'IT4ILexisV2') + + def _parse_zone_from_path(self, path: str | None) -> str | None: + if not path: + return None + if not path.startswith("/"): + return None + parts = path.lstrip("/").split("/", 1) + return parts[0] if parts and parts[0] else None + + def _zone_config(self, zone: str) -> dict: + if isinstance(self._zones, dict): + return self._zones.get(zone, {}) or {} + return {} + + def _get_zone_irods_url(self, zone: str) -> str: + zone_config = self._zone_config(zone) + if not zone_config and zone not in self._warned_unknown_zones: + logger.warning( + "Zone '%s' not configured; using default iRODS HTTP API URL. " + "Add it to config.zones to set api_url/location_name.", + zone, + ) + self._warned_unknown_zones.add(zone) + api_url = zone_config.get("api_url") or self.lexis_session.irods_http_api_url + try: + from urllib.parse import urlparse + parsed = urlparse(api_url) + if parsed.scheme and parsed.netloc and parsed.path in ("", "/"): + normalized = api_url.rstrip("/") + "/" + logger.warning( + "Zone '%s' api_url has no path; using '%s'. " + "Set a full api_url to override.", + zone, + normalized, + ) + return normalized + except Exception: + pass + return api_url + + def _get_zone_location_name(self, zone: str) -> str | None: + zone_config = self._zone_config(zone) + return zone_config.get("location_name") or self._default_location_name + + def _resolve_zone(self, *, zone: str | None = None, path: str | None = None, dataset: Dataset | None = None) -> str: + if zone: + return zone + if dataset and dataset.zone: + return dataset.zone + zone_from_path = self._parse_zone_from_path(path) + if zone_from_path: + return zone_from_path + return self.zone - def _get_irods_path(self, access: str, dataset_id: str = None) -> str: + def _get_irods_path(self, access: str, dataset_id: str = None, zone: str | None = None) -> str: """Build iRODS path for a dataset.""" - base = f"/{self.zone}/{access}" + resolved_zone = self._resolve_zone(zone=zone) + base = f"/{resolved_zone}/{access}" if dataset_id: return f"{base}/{dataset_id}" return base @@ -120,7 +206,8 @@ class LexisRepository(AbstractRepository): """Get iRODS path for a dataset.""" if dataset.path: return dataset.path - return self._get_irods_path(dataset.access, dataset.metadata["id"]) + resolved_zone = self._resolve_zone(dataset=dataset) + return self._get_irods_path(dataset.access, dataset.metadata["id"], zone=resolved_zone) def status(self) -> dict: """Check repository connectivity.""" @@ -171,11 +258,27 @@ class LexisRepository(AbstractRepository): for record in items: try: + raw_path = record.get("absolute_path") or record.get("absolutePath") + record_zone = record.get("zone") or self._parse_zone_from_path(raw_path) + record_zone = record_zone or self.zone + if "zone" not in record: + record["zone"] = record_zone + if not record.get("location_name"): + location_name = self._get_zone_location_name(record_zone) + if location_name: + record["location_name"] = location_name + if not raw_path: + record["absolute_path"] = self._get_irods_path( + access, + record.get("dataset_id", record.get("id", "")), + zone=record_zone + ) + record_path = record.get("absolute_path") or record.get("absolutePath") # Pass the full record to Dataset like old implementation # This preserves all metadata fields (startDate, endDate, objectCount, etc.) ds = Dataset( repository=self, - path=record.get("absolute_path", self._get_irods_path(access, record.get("dataset_id", record.get("id", "")))), + path=record_path, **record ) datasets.append(ds) @@ -198,8 +301,6 @@ class LexisRepository(AbstractRepository): OPTIMIZED: Uses fs.find() which fetches entire tree in single HTTP request, then applies fnmatch patterns locally. """ - import fnmatch - path = self._get_path(dataset) if files_glob is None: @@ -237,15 +338,7 @@ class LexisRepository(AbstractRepository): rel_path = entry for pat in patterns: - # Handle ** patterns - if "**" in pat: - # fnmatch doesn't support **, use simple suffix matching - suffix = pat.replace("**/*", "").replace("**/", "").lstrip("/") - if suffix and entry.endswith(suffix): - hits.add(entry) - elif not suffix: # Pattern is just "**/*" - hits.add(entry) - elif fnmatch.fnmatch(rel_path, pat): + if _match_glob_pattern(rel_path, pat): hits.add(entry) return sorted(hits) @@ -311,8 +404,40 @@ class LexisRepository(AbstractRepository): # Upload file start_time = time.time() file_size = os.path.getsize(local_path) - - self.fs.put(local_path, remote_path) + chunk_size = 8 * 1024 * 1024 + written = 0 + with open(local_path, "rb") as source: + first_chunk = source.read(chunk_size) + if first_chunk == b"": + result = self.fs.data_objects.write(b"", lpath=remote_path, truncate=1, append=0, offset=0) + if isinstance(result, dict) and int(result.get("status_code", 500)) // 100 != 2: + raise IOError(f"Failed to upload empty file to {remote_path}: {result}") + else: + result = self.fs.data_objects.write( + first_chunk, + lpath=remote_path, + truncate=1, + append=0, + offset=0, + ) + if isinstance(result, dict) and int(result.get("status_code", 500)) // 100 != 2: + raise IOError(f"Failed to upload {local_path} to {remote_path}: {result}") + written += len(first_chunk) + + while True: + chunk = source.read(chunk_size) + if chunk == b"": + break + result = self.fs.data_objects.write( + chunk, + lpath=remote_path, + truncate=0, + append=1, + offset=written, + ) + if isinstance(result, dict) and int(result.get("status_code", 500)) // 100 != 2: + raise IOError(f"Failed to append upload chunk for {local_path} to {remote_path}: {result}") + written += len(chunk) elapsed = time.time() - start_time self.update_performance_stats("upload", elapsed, file_size) @@ -371,4 +496,4 @@ class LexisRepository(AbstractRepository): return [str(e)] def __str__(self): - return f"LexisRepository(zone={self.zone})" + return f"LexisRepository(default_zone={self.zone})" diff --git a/owilix/core/tasks/remote.py b/owilix/core/tasks/remote.py index 335a884..acad6ee 100644 --- a/owilix/core/tasks/remote.py +++ b/owilix/core/tasks/remote.py @@ -7,6 +7,7 @@ import inspect import os import json import re +import time import subprocess import tempfile from collections import Counter @@ -443,6 +444,453 @@ def remote_push( return CommandResult(success=True, object=datasets, msg=f"Pushed {len(datasets)} datasets") +def _select_storage_for_zone( + storages: List[Dict[str, Any]], + zone: str, + preferred_location_name: Optional[str] = None, +) -> Optional[Dict[str, Any]]: + if not storages: + return None + + zone_norm = str(zone or "").strip().lower() + preferred_norm = str(preferred_location_name or "").strip().lower() + + def _norm_token(value: str) -> str: + v = re.sub(r"[^a-z0-9]", "", str(value or "").lower()) + if v.startswith("irods"): + v = v[5:] + return v + + def _zone_aliases(value: str) -> List[str]: + base = _norm_token(value) + aliases = [base] if base else [] + if base.endswith("zone") and len(base) > 4: + aliases.append(base[:-4]) + if base.startswith("owi") and len(base) > 3: + aliases.append(base[3:]) + return [a for a in aliases if a] + + def storage_name(s: Dict[str, Any]) -> str: + return str(s.get("storage_name") or s.get("storageName") or "").strip() + + def storage_resource(s: Dict[str, Any]) -> str: + return str( + s.get("storage_resource") + or s.get("storage_resouce") + or s.get("storageResource") + or "" + ).strip() + + # 1) Preferred location_name match (exact and normalized/contains). + if preferred_norm: + pref_aliases = _zone_aliases(preferred_norm) + for s in storages: + s_name = storage_name(s).lower() + s_res = storage_resource(s).lower() + if s_name == preferred_norm or s_res == preferred_norm: + return s + s_name_norm = _norm_token(s_name) + s_res_norm = _norm_token(s_res) + for alias in pref_aliases: + if alias and (alias == s_name_norm or alias == s_res_norm or alias in s_name_norm or alias in s_res_norm): + return s + + # 2) Exact zone match on storage_name or storage_resource. + zone_aliases = _zone_aliases(zone_norm) + for s in storages: + s_name = storage_name(s).lower() + s_res = storage_resource(s).lower() + if s_name == zone_norm or s_res == zone_norm: + return s + s_name_norm = _norm_token(s_name) + s_res_norm = _norm_token(s_res) + for alias in zone_aliases: + if alias and (alias == s_name_norm or alias == s_res_norm): + return s + + # 3) Contains zone aliases (case-insensitive/normalized). + for s in storages: + s_name = storage_name(s).lower() + s_res = storage_resource(s).lower() + s_name_norm = _norm_token(s_name) + s_res_norm = _norm_token(s_res) + for alias in zone_aliases: + if alias and ( + alias in s_name + or alias in s_res + or alias in s_name_norm + or alias in s_res_norm + ): + return s + + # 4) Fallback to first storage. + return storages[0] + + +def _prune_empty_values(value: Any) -> Any: + """Recursively drop None/empty-string values from dict/list payloads.""" + if isinstance(value, dict): + out: Dict[str, Any] = {} + for k, v in value.items(): + cleaned = _prune_empty_values(v) + if cleaned is None: + continue + if isinstance(cleaned, str) and cleaned.strip() == "": + continue + if isinstance(cleaned, (dict, list)) and len(cleaned) == 0: + continue + out[k] = cleaned + return out + if isinstance(value, list): + out_list: List[Any] = [] + for item in value: + cleaned = _prune_empty_values(item) + if cleaned is None: + continue + if isinstance(cleaned, str) and cleaned.strip() == "": + continue + if isinstance(cleaned, (dict, list)) and len(cleaned) == 0: + continue + out_list.append(cleaned) + return out_list + return value + + +def remote_upload( + manager: Any, + directory: str, + repository: Optional[str] = None, + zone: Optional[str] = None, + collection_name: Optional[str] = None, + metadata_file: Optional[str] = None, + dataset_id: Optional[str] = None, + update_metadata_from_file: Optional[str] = None, + storage_name_override: Optional[str] = None, + storage_resource_override: Optional[str] = None, + access: str = "project", + yes: bool = False, + console: Optional[Console] = None, +) -> CommandResult: + """ + Create a new remote dataset and upload all files from a local directory. + """ + if console is None: + console = Console() + + access_norm = str(access).strip().lower() + if access_norm not in ("public", "project"): + return CommandResult(success=False, msg="Access must be 'public' or 'project'") + + source_dir = os.path.abspath(os.path.expanduser(directory)) + if not os.path.isdir(source_dir): + return CommandResult(success=False, msg=f"Directory not found: {source_dir}") + + metadata_payload = None + if not dataset_id: + if not metadata_file: + return CommandResult(success=False, msg="Either dataset_id or metadata_file is required") + metadata_path = os.path.abspath(os.path.expanduser(metadata_file)) + if not os.path.isfile(metadata_path): + return CommandResult(success=False, msg=f"Metadata file not found: {metadata_path}") + try: + with open(metadata_path, "r", encoding="utf-8") as handle: + metadata_payload = json.load(handle) + except Exception as e: + return CommandResult(success=False, msg=f"Failed to parse metadata JSON: {e}") + + metadata_update_payload: Optional[Dict[str, Any]] = None + if update_metadata_from_file: + metadata_update_path = os.path.abspath(os.path.expanduser(update_metadata_from_file)) + if not os.path.isfile(metadata_update_path): + return CommandResult(success=False, msg=f"Update metadata file not found: {metadata_update_path}") + try: + with open(metadata_update_path, "r", encoding="utf-8") as handle: + metadata_update_payload = json.load(handle) + except Exception as e: + return CommandResult(success=False, msg=f"Failed to parse update metadata JSON: {e}") + if not isinstance(metadata_update_payload, dict): + return CommandResult(success=False, msg="Update metadata JSON must be an object") + + repo = None + repo_name: Optional[str] = repository + existing_record: Optional[Dict[str, Any]] = None + + if dataset_id and not repository: + for candidate_name in sorted(manager.remote_data.get_repo_names()): + candidate = manager.remote_data.get_single_repo(candidate_name) + if candidate is None or not hasattr(candidate, "ddi_api"): + continue + try: + record = candidate.ddi_api.get_dataset_info_by_id(dataset_id, content_as_pandas=False) + except Exception: + continue + if record: + repo = candidate + repo_name = candidate_name + existing_record = record + break + if repo is None: + return CommandResult( + success=False, + msg=f"Could not resolve dataset '{dataset_id}' in configured repositories; pass --repository explicitly", + ) + else: + if not repository: + return CommandResult(success=False, msg="Repository is required when creating a dataset") + repo = manager.remote_data.get_single_repo(repository) + if repo is None: + available = ", ".join(sorted(manager.remote_data.get_repo_names())) + return CommandResult( + success=False, + msg=f"Repository '{repository}' not found. Available: {available}", + ) + repo_name = repository + + if not hasattr(repo, "ddi_api"): + return CommandResult(success=False, msg=f"Repository '{repo_name}' does not support DDI dataset creation") + + if dataset_id and existing_record is None: + try: + existing_record = repo.ddi_api.get_dataset_info_by_id(dataset_id, content_as_pandas=False) + except Exception: + existing_record = {} + + datacite_payload = None + additional_metadata = {"collectionName": collection_name or "main"} + title = None + if metadata_payload is not None: + datacite_payload = dict(metadata_payload) + additional_metadata = datacite_payload.pop("additionalMetadata", {}) or {} + additional_metadata["collectionName"] = collection_name or additional_metadata.get("collectionName") or "main" + additional_metadata = _prune_empty_values(additional_metadata) or {} + additional_metadata["collectionName"] = additional_metadata.get("collectionName") or "main" + datacite_payload = _prune_empty_values(datacite_payload) + + titles = datacite_payload.get("titles", []) if isinstance(datacite_payload, dict) else [] + if isinstance(titles, list) and titles: + first_title = titles[0] + if isinstance(first_title, dict): + title = first_title.get("title") + if not title: + title = datacite_payload.get("title") if isinstance(datacite_payload, dict) else None + if not title: + title = f"OWI Upload {__import__('datetime').datetime.now().strftime('%Y-%m-%d %H:%M:%S')}" + + project_name = getattr(manager, "name", None) or os.getenv("OWILIX_LEXIS_PROJECT_NAME", "openwebsearch") + storage_name = None + storage_resource = None + if not dataset_id: + if not zone: + return CommandResult(success=False, msg="Zone is required when creating a dataset") + if (storage_name_override and not storage_resource_override) or (storage_resource_override and not storage_name_override): + return CommandResult( + success=False, + msg="Both storage_name_override and storage_resource_override are required together", + ) + if storage_name_override and storage_resource_override: + storage_name = storage_name_override + storage_resource = storage_resource_override + else: + storages = repo.ddi_api.get_project_storages(project_name) + preferred_location = None + if hasattr(repo, "_get_zone_location_name"): + preferred_location = repo._get_zone_location_name(zone) + selected_storage = _select_storage_for_zone(storages, zone=zone, preferred_location_name=preferred_location) + if not selected_storage: + return CommandResult(success=False, msg=f"No storage found for project '{project_name}'") + + storage_name = selected_storage.get("storage_name") or selected_storage.get("storageName") + storage_resource = ( + selected_storage.get("storage_resource") + or selected_storage.get("storage_resouce") + or selected_storage.get("storageResource") + ) + if not storage_name or not storage_resource: + return CommandResult(success=False, msg=f"Selected storage is missing required fields: {selected_storage}") + + local_files: List[tuple[str, str]] = [] + for root, _dirs, files in os.walk(source_dir): + for filename in files: + abs_path = os.path.join(root, filename) + rel_path = os.path.relpath(abs_path, source_dir).replace(os.sep, "/") + local_files.append((abs_path, rel_path)) + local_files.sort(key=lambda item: item[1]) + + if not local_files and metadata_update_payload is None: + return CommandResult(success=False, msg=f"No files found in directory: {source_dir}") + + effective_access = access_norm + if dataset_id and isinstance(existing_record, dict): + _record_access = str(existing_record.get("access") or "").strip().lower() + if _record_access in ("public", "project"): + effective_access = _record_access + + console.print("[bold]Dataset Upload Plan[/bold]") + console.print(f" Repository: {repo_name}") + if zone: + console.print(f" Zone: {zone}") + console.print(f" Access: {effective_access}") + if collection_name: + console.print(f" Collection: {collection_name}") + if dataset_id: + console.print(f" Dataset ID: {dataset_id} (existing)") + if effective_access != access_norm: + console.print(f" Access flag ignored for existing dataset (requested {access_norm}, actual {effective_access})") + else: + console.print(f" Project: {project_name}") + console.print(f" Storage name: {storage_name}") + console.print(f" Storage resource: {storage_resource}") + console.print(f" Title: {title}") + console.print(f" Files: {len(local_files):,}") + if metadata_update_payload is not None: + console.print(" Metadata update: yes (from file)") + + action_label = "Upload all files to existing dataset?" if dataset_id else "Create remote dataset and upload all files?" + if not yes and not ask_yes_no(console, action_label): + return CommandResult(success=False, msg="Upload cancelled") + + created_dataset = False + if not dataset_id: + ddi_prev_reraise = getattr(repo.ddi_api, "_reraise_exceptions", None) + if ddi_prev_reraise is not None: + setattr(repo.ddi_api, "_reraise_exceptions", True) + try: + creation = repo.ddi_api.create_dataset( + access=access_norm, + project=project_name, + storage_name=storage_name, + storage_resource=storage_resource, + dataset_type="dataset", + additional_metadata=additional_metadata, + datacite=datacite_payload, + title=title, + ) + except Exception as e: + return CommandResult(success=False, msg=f"Dataset creation failed: {e}") + finally: + if ddi_prev_reraise is not None: + setattr(repo.ddi_api, "_reraise_exceptions", ddi_prev_reraise) + + if not isinstance(creation, dict): + return CommandResult( + success=False, + msg=( + "Dataset creation failed: API returned no dataset descriptor. " + "Check metadata JSON validity (DataCite fields) and server logs." + ), + ) + dataset_id = str(creation.get("dataset_id") or creation.get("id") or "").strip() + if not dataset_id: + return CommandResult(success=False, msg=f"Dataset creation returned no dataset_id: {creation}") + created_dataset = True + + if metadata_update_payload is not None: + try: + update_payload = dict(metadata_update_payload) + if collection_name: + additional = update_payload.get("additionalMetadata") + if not isinstance(additional, dict): + additional = {} + additional["collectionName"] = collection_name + update_payload["additionalMetadata"] = additional + update_payload = _prune_empty_values(update_payload) or {} + repo.ddi_api.update_dataset_metadata(dataset_id=dataset_id, metadata=update_payload) + console.print("[green]Metadata updated from file[/green]") + except Exception as e: + return CommandResult(success=False, msg=f"Metadata update failed: {e}") + + if existing_record is None: + record = {} + # Newly created datasets can be briefly unavailable in DDI search/index. + # Retry so we always resolve the authoritative absolute_path/zone. + max_attempts = 8 if created_dataset else 3 + for attempt in range(max_attempts): + try: + candidate = repo.ddi_api.get_dataset_info_by_id(dataset_id, content_as_pandas=False) or {} + except Exception: + candidate = {} + if candidate and (candidate.get("absolute_path") or candidate.get("absolutePath")): + record = candidate + break + if attempt < max_attempts - 1: + time.sleep(0.5) + else: + record = existing_record or {} + + if not record or not (record.get("absolute_path") or record.get("absolutePath")): + if created_dataset: + return CommandResult( + success=False, + object={"datasetId": dataset_id}, + msg=( + f"Dataset {dataset_id} was created but metadata lookup did not return absolute_path. " + "Aborting upload to avoid writing to a wrong zone/path. Re-run with --dataset-id once metadata is visible." + ), + ) + return CommandResult( + success=False, + object={"datasetId": dataset_id}, + msg=f"Could not resolve dataset absolute_path for {dataset_id}; aborting upload.", + ) + + record = record or {} + record.setdefault("id", dataset_id) + record.setdefault("dataset_id", dataset_id) + record.setdefault("access", access_norm) + if collection_name: + record.setdefault("collectionName", collection_name) + if zone and not record.get("zone"): + record["zone"] = zone + if hasattr(repo, "_get_zone_location_name") and not record.get("location_name"): + location_name = repo._get_zone_location_name(zone) + if location_name: + record["location_name"] = location_name + from owilix.core.models.dataset import Dataset + dataset = Dataset( + repository=repo, + path=record.get("absolute_path"), + **record, + ) + + uploaded = 0 + failed = 0 + if local_files: + dataset_base_path = record.get("absolute_path") or dataset.path or "" + with currentItemProgress() as progress: + upload_task = progress.add_task("Files uploaded", total=max(1, len(local_files)), current_item="Starting") + error_task = progress.add_task("Failed uploads", total=max(1, len(local_files)), current_item="None") + for abs_path, rel_path in local_files: + try: + target_path = f"{dataset_base_path.rstrip('/')}/{rel_path}" if dataset_base_path else rel_path + progress.update(upload_task, current_item=f"Uploading {rel_path} -> {target_path}") + manager.remote_data.put(dataset, abs_path, rel_path) + uploaded += 1 + progress.update(upload_task, advance=1, current_item=f"Uploaded -> {target_path}") + except Exception as e: + failed += 1 + progress.update(upload_task, advance=1, current_item=f"Failed -> {target_path}") + progress.update(error_task, advance=1, current_item=f"{rel_path[:50]}: {e}") + + if failed > 0: + return CommandResult( + success=False, + object={"datasetId": dataset_id, "uploaded": uploaded, "failed": failed}, + msg=( + f"{'Created dataset' if created_dataset else 'Using dataset'} {dataset_id}, " + f"but {failed}/{len(local_files)} uploads failed" + ), + ) + if created_dataset: + console.print(f"[green]Dataset created and uploaded successfully: {dataset_id}[/green]") + else: + console.print(f"[green]Upload to existing dataset completed: {dataset_id}[/green]") + return CommandResult( + success=True, + object={"datasetId": dataset_id, "uploaded": uploaded, "failed": failed}, + msg=f"Uploaded {uploaded} files to dataset {dataset_id}", + ) + + def remote_catalog( manager: Any, specifier: str, diff --git a/tests/owilix/core/repository/test_repository.py b/tests/owilix/core/repository/test_repository.py index ed3a17c..8f8ce7f 100644 --- a/tests/owilix/core/repository/test_repository.py +++ b/tests/owilix/core/repository/test_repository.py @@ -117,3 +117,69 @@ class TestCoreExports: """Test LexisRepository accessible from core.""" from owilix.core import LexisRepository assert LexisRepository is not None + + +class TestLexisRepositoryRouting: + """Unit tests for LexisRepository zone helpers.""" + + class DummySession: + def __init__(self, zone="IT4ILexisV2"): + self._zone = zone + self.irods_http_api_url = "https://example.com/irods" + + class DummyManager: + def __init__(self, zone="IT4ILexisV2"): + class SessionWrapper: + def __init__(self, session): + self.lexis = session + self.session = SessionWrapper(TestLexisRepositoryRouting.DummySession(zone=zone)) + + def test_parse_zone_from_path(self): + from owilix.core.repository import LexisRepository + repo = LexisRepository(self.DummyManager()) + + assert repo._parse_zone_from_path("/IT4ILexisV2/public") == "IT4ILexisV2" + assert repo._parse_zone_from_path("relative/path") is None + assert repo._parse_zone_from_path(None) is None + + def test_resolve_zone_prefers_dataset_zone(self): + from owilix.core.repository import LexisRepository + from owilix.core.models.dataset import Dataset + + repo = LexisRepository(self.DummyManager()) + dataset = Dataset(repository=repo, path="/OTHER/public/x", zone="OTHER", access="public", internalID="x") + + assert repo._resolve_zone(dataset=dataset) == "OTHER" + + def test_get_irods_path_uses_zone(self): + from owilix.core.repository import LexisRepository + repo = LexisRepository(self.DummyManager()) + + path = repo._get_irods_path("public", "abc", zone="LRZ") + assert path == "/LRZ/public/abc" + + def test_zone_location_name_fallback(self): + from owilix.core.repository import LexisRepository + repo = LexisRepository( + self.DummyManager(), + default_location_name="it4i", + zones={"IT4ILexisV2": {"location_name": "override"}} + ) + + assert repo._get_zone_location_name("IT4ILexisV2") == "override" + assert repo._get_zone_location_name("UNKNOWN") == "it4i" + + +class TestLexisGlobMatching: + def test_double_star_dot_star_matches_nested(self): + from owilix.core.repository.lexis import _match_glob_pattern + assert _match_glob_pattern("documents/metadata_265.parquet", "**/*.*") is True + + def test_double_star_dot_star_matches_root_file(self): + from owilix.core.repository.lexis import _match_glob_pattern + assert _match_glob_pattern("README.md", "**/*.*") is True + + def test_parquet_pattern(self): + from owilix.core.repository.lexis import _match_glob_pattern + assert _match_glob_pattern("parquet/batch0001.parquet", "**/*.parquet") is True + assert _match_glob_pattern("parquet/batch0001.txt", "**/*.parquet") is False diff --git a/tests/owilix/core/tasks/test_remote.py b/tests/owilix/core/tasks/test_remote.py index 7f6fbd6..72d94d2 100644 --- a/tests/owilix/core/tasks/test_remote.py +++ b/tests/owilix/core/tasks/test_remote.py @@ -18,6 +18,7 @@ from owilix.core.tasks.remote import ( _markdown_table, _extract_language_from_path, _build_host, + _prune_empty_values, _slugify_command, _summary_output_path, _hosts_cache_root, @@ -36,9 +37,11 @@ from owilix.core.tasks.remote import ( _format_top_languages, _aggregate_local_hosts_cache, _generate_readme_markdown, + _select_storage_for_zone, extract_domain_tldextract, summarize_local_host_stats, remote_doctor, + remote_upload, remote_pull, remote_push, remote_remove, @@ -128,6 +131,52 @@ class TestBuildHost: assert _build_host(" www ", " example ", " com ") == "www.example.com" +# --------------------------------------------------------------------------- +# _prune_empty_values +# --------------------------------------------------------------------------- +class TestPruneEmptyValues: + def test_prunes_empty_values_recursively(self): + payload = { + "a": "", + "b": None, + "c": "ok", + "d": {"x": "", "y": 1}, + "e": [{"k": ""}, {"k": "v"}, "", None], + } + assert _prune_empty_values(payload) == {"c": "ok", "d": {"y": 1}, "e": [{"k": "v"}]} + + +# --------------------------------------------------------------------------- +# _select_storage_for_zone +# --------------------------------------------------------------------------- +class TestSelectStorageForZone: + def test_prefers_location_name_exact(self): + storages = [ + {"storage_name": "OtherStorage", "storage_resource": "IT4IOther"}, + {"storage_name": "IT4ILexisV2", "storage_resource": "IT4ILexisV2Res"}, + ] + selected = _select_storage_for_zone(storages, zone="it4ilexisv2", preferred_location_name="IT4ILexisV2") + assert selected["storage_name"] == "IT4ILexisV2" + + def test_matches_storage_resource(self): + storages = [{"storage_name": "LRZ", "storage_resource": "IT4ILexisV2"}] + selected = _select_storage_for_zone(storages, zone="it4ilexisv2") + assert selected["storage_resource"] == "IT4ILexisV2" + + def test_fallback_first(self): + storages = [{"storage_name": "A"}, {"storage_name": "B"}] + selected = _select_storage_for_zone(storages, zone="missing-zone") + assert selected["storage_name"] == "A" + + def test_matches_zone_alias_it4izone_to_irods_it4i(self): + storages = [ + {"storage_name": "iRODS LRZ OWSeu", "storage_resource": "pn36le-DATA"}, + {"storage_name": "iRODS IT4I", "storage_resource": "ATR-25-2"}, + ] + selected = _select_storage_for_zone(storages, zone="IT4IZone") + assert selected["storage_name"] == "iRODS IT4I" + + # --------------------------------------------------------------------------- # _slugify_command # --------------------------------------------------------------------------- @@ -861,3 +910,188 @@ class TestSummarizeLocalHostStats: assert summary["datasetsMissingLocalStats"] == 0 assert "cefal" in summary["collections"] assert summary["collections"]["cefal"]["uniqueHosts"] == 2 + + +# --------------------------------------------------------------------------- +# remote_upload +# --------------------------------------------------------------------------- +class TestRemoteUpload: + def test_create_mode_requires_repository(self, tmp_path): + data_dir = tmp_path / "data" + data_dir.mkdir() + (data_dir / "part-000.txt").write_text("x", encoding="utf-8") + + metadata_file = tmp_path / "metadata.json" + metadata_file.write_text('{"titles":[{"title":"Demo"}]}', encoding="utf-8") + + manager = MagicMock() + result = remote_upload( + manager=manager, + directory=str(data_dir), + repository=None, + zone="IT4ILexisV2", + collection_name="main", + metadata_file=str(metadata_file), + yes=True, + console=MagicMock(), + ) + assert result.success is False + assert "Repository is required" in result.msg + + def test_create_mode_requires_zone(self, tmp_path): + data_dir = tmp_path / "data" + data_dir.mkdir() + (data_dir / "part-000.txt").write_text("x", encoding="utf-8") + + metadata_file = tmp_path / "metadata.json" + metadata_file.write_text('{"titles":[{"title":"Demo"}]}', encoding="utf-8") + + repo = MagicMock() + repo.ddi_api = MagicMock() + manager = MagicMock() + manager.remote_data.get_single_repo.return_value = repo + + result = remote_upload( + manager=manager, + directory=str(data_dir), + repository="lexis", + zone=None, + collection_name="main", + metadata_file=str(metadata_file), + yes=True, + console=MagicMock(), + ) + assert result.success is False + assert "Zone is required" in result.msg + + def test_create_mode_handles_none_creation_response(self, tmp_path): + data_dir = tmp_path / "data" + data_dir.mkdir() + (data_dir / "part-000.txt").write_text("x", encoding="utf-8") + + metadata_file = tmp_path / "metadata.json" + metadata_file.write_text('{"titles":[{"title":"Demo"}]}', encoding="utf-8") + + repo = MagicMock() + repo.ddi_api = MagicMock() + repo.ddi_api.get_project_storages.return_value = [ + {"storage_name": "IT4ILexisV2", "storage_resource": "ATR-25-2"} + ] + repo.ddi_api.create_dataset.return_value = None + + manager = MagicMock() + manager.remote_data.get_single_repo.return_value = repo + + result = remote_upload( + manager=manager, + directory=str(data_dir), + repository="lexis", + zone="IT4ILexisV2", + collection_name="main", + metadata_file=str(metadata_file), + yes=True, + console=MagicMock(), + ) + + assert result.success is False + assert "returned no dataset descriptor" in result.msg + + def test_create_mode_storage_override_requires_pair(self, tmp_path): + data_dir = tmp_path / "data" + data_dir.mkdir() + (data_dir / "part-000.txt").write_text("x", encoding="utf-8") + metadata_file = tmp_path / "metadata.json" + metadata_file.write_text('{"titles":[{"title":"Demo"}]}', encoding="utf-8") + + repo = MagicMock() + repo.ddi_api = MagicMock() + manager = MagicMock() + manager.remote_data.get_single_repo.return_value = repo + + result = remote_upload( + manager=manager, + directory=str(data_dir), + repository="lexis", + zone="IT4IZone", + collection_name="main", + metadata_file=str(metadata_file), + storage_name_override="iRODS IT4I", + storage_resource_override=None, + yes=True, + console=MagicMock(), + ) + assert result.success is False + assert "required together" in result.msg + + def test_dataset_id_mode_auto_resolves_repository(self, tmp_path): + data_dir = tmp_path / "data" + data_dir.mkdir() + (data_dir / "folder").mkdir() + (data_dir / "folder" / "a.txt").write_text("abc", encoding="utf-8") + + repo1 = MagicMock() + repo1.ddi_api = MagicMock() + repo1.ddi_api.get_dataset_info_by_id.side_effect = Exception("not found") + + repo2 = MagicMock() + repo2.ddi_api = MagicMock() + repo2.ddi_api.get_dataset_info_by_id.return_value = {"id": "ds-1", "absolute_path": "/irods/path/ds-1"} + + manager = MagicMock() + manager.remote_data.get_repo_names.return_value = ["repo1", "repo2"] + + def _repo_by_name(name): + return repo1 if name == "repo1" else repo2 + + manager.remote_data.get_single_repo.side_effect = _repo_by_name + + result = remote_upload( + manager=manager, + directory=str(data_dir), + repository=None, + zone=None, + collection_name=None, + dataset_id="ds-1", + yes=True, + console=MagicMock(), + ) + + assert result.success is True + assert result.object["datasetId"] == "ds-1" + manager.remote_data.put.assert_called_once() + put_args = manager.remote_data.put.call_args.args + assert put_args[2] == "folder/a.txt" + + def test_dataset_id_mode_updates_metadata_from_file(self, tmp_path): + data_dir = tmp_path / "data" + data_dir.mkdir() + (data_dir / "a.txt").write_text("abc", encoding="utf-8") + + update_file = tmp_path / "update.json" + update_file.write_text('{"datacite":{"titles":[{"title":"Updated"}]}}', encoding="utf-8") + + repo = MagicMock() + repo.ddi_api = MagicMock() + repo.ddi_api.get_dataset_info_by_id.return_value = {"id": "ds-2", "absolute_path": "/irods/path/ds-2"} + + manager = MagicMock() + manager.remote_data.get_repo_names.return_value = ["lexis"] + manager.remote_data.get_single_repo.return_value = repo + + result = remote_upload( + manager=manager, + directory=str(data_dir), + repository=None, + zone=None, + collection_name=None, + dataset_id="ds-2", + update_metadata_from_file=str(update_file), + yes=True, + console=MagicMock(), + ) + + assert result.success is True + repo.ddi_api.update_dataset_metadata.assert_called_once() + kwargs = repo.ddi_api.update_dataset_metadata.call_args.kwargs + assert kwargs["dataset_id"] == "ds-2" + assert "datacite" in kwargs["metadata"] -- 2.51.2 From 694c1dc4219c0de4c6c0f0442909c12161300500 Mon Sep 17 00:00:00 2001 From: mgrani Date: Tue, 17 Feb 2026 09:37:48 +0100 Subject: [PATCH 4/6] feat(multifs): improve filesystem integration and align backlog/changelog workflow --- .agent/workflows/release.md | 62 ++------- .gitignore | 2 +- CHANGELOG.md | 12 ++ docs/AI_DEVELOPMENT.md | 14 ++ docs/branch/main.md | 10 +- docs/changes.md | 13 ++ docs/epics.md | 1 + docs/source/config.md | 11 +- owilix/cli/__init__.py | 5 + owilix/cli/_common/context.py | 3 +- owilix/cli/_common/output.py | 11 +- owilix/cli/local.py | 2 +- owilix/cli/plugin.py | 3 +- owilix/core/fsspec/__init__.py | 4 +- owilix/core/fsspec/http2irods.py | 129 ++++++++++++++++++ owilix/core/manager/manager.py | 20 ++- owilix/core/tasks/query.py | 2 + pyproject.toml | 1 + .../core/fsspec/test_core_fsspec_unit.py | 124 ++++++++++++++++- uv.lock | 32 ++++- 20 files changed, 394 insertions(+), 67 deletions(-) diff --git a/.agent/workflows/release.md b/.agent/workflows/release.md index 87a8891..bdbe829 100644 --- a/.agent/workflows/release.md +++ b/.agent/workflows/release.md @@ -1,50 +1,10 @@ --- -description: Semantic release workflow for owilix +description: Manual release workflow for owilix --- # Release Workflow -## Semantic Versioning - -owilix uses [Python Semantic Release](https://python-semantic-release.readthedocs.io/) with Angular commit conventions. - -## Commit Message Format - -``` -(): - - - -