From 9118220863a13cfba6cc476c8ae8fcc4c9d5a642 Mon Sep 17 00:00:00 2001 From: Matthew Suozzo Date: Wed, 30 Sep 2026 11:25:06 -0400 Subject: [PATCH] README: condense and reorganize --- README.md | 225 ++++++++++++++++++------------------------------------ 1 file changed, 74 insertions(+), 151 deletions(-) diff --git a/README.md b/README.md index f17a1f3..9385359 100644 --- a/README.md +++ b/README.md @@ -1,74 +1,40 @@ # isomorph -isomorph maps every source package in four Linux distributions to the upstream repository -it is built from. Through that repository it finds the same software in the other -distributions: -- Debian sid; -- Arch (core, extra, multilib); -- openSUSE Tumbleweed; -- Alpine edge. +isomorph works out which upstream repository each source package in a Linux distribution is +built from. Packages that come from the same repository are the same software, whatever +their distributions call them: Debian's `requests`, Arch's `python-requests` and Alpine's +`py3-requests` all resolve to `github.com/psf/requests`. It also works the other way. The +`fff` in Debian and the `fff` in openSUSE are unrelated projects, and they resolve to different +repositories. -The mapping never relies on package names. `py3-requests`, `python-requests` and -`python3-requests` meet at `github.com/psf/requests`, and two unrelated projects that happen -to share a name stay apart. +It covers Debian sid, Arch Linux, openSUSE Tumbleweed and Alpine edge. Every mapping is made +from the distribution's own packaging (a watch file, an Arch `source=` array, an openSUSE spec) +and records the evidence it came from. -Every mapping is made from the distribution's own packaging data, and records the evidence -and the kind of evidence behind it. [docs/EVALUATION.md](docs/EVALUATION.md) measures how far -each kind can be trusted. +## Usage -## What you get - -`isomorph export` writes a snapshot: +```sh +go build ./cmd/isomorph +isomorph bulk debian +isomorph bulk arch -crawl 20000 +isomorph bulk opensuse -crawl 7000 +isomorph bulk alpine +GITHUB_TOKEN=... isomorph bulk canonicalize +isomorph export ``` -export/ - LATEST the newest snapshot id - 20260928-946df9558260/ - manifest.json every file with its sha256, plus where each slice came from - debian-sid.jsonl one row per source package, per distribution suite - arch-core.jsonl arch-extra.jsonl arch-multilib.jsonl - opensuse-tumbleweed.jsonl - alpine-edge-main.jsonl alpine-edge-community.jsonl - packages.sqlite the same rows as a SQLite database - package.schema.json manifest.schema.json -``` -- **The JSONL files are the canonical record.** Each line is one source package - ([schema](schema/package.schema.json)): - - ```json - {"distro": "arch", "suite": "core", "package": "json-c", "version": "0.19-1", - "binaries": ["json-c"], "homepage": "https://github.com/json-c/json-c/wiki", - "upstream_key": "github.com/json-c/json-c", "repo": "https://github.com/json-c/json-c", - "status": "resolved", "basis": "vcs-source", - "evidence": [ - {"source": "SRCINFO:source:git", "value": "git+https://github.com/json-c/json-c#tag=json-c-0.19-20260627", - "key": "github.com/json-c/json-c", "weight": 0.9}, - {"source": "SRCINFO:url", "value": "https://github.com/json-c/json-c/wiki", - "key": "github.com/json-c/json-c", "weight": 0.5}]} - ``` - - - `upstream_key` identifies the repository: packages with equal keys are the same software. - - `repo` is a clone URL, ready for an OSV `GIT` range. - - `osv_ecosystem` is set where OSV defines one (`Debian:sid`, `openSUSE:Tumbleweed`). -- **The SQLite database** is a convenience built from the same run. Python reads it with the - standard library. - - Its `counterparts` view pairs every package with the packages that share its upstream. - - `binaries` maps binary package names to source packages. - - `aliases` records which keys were merged, and why. -- **The snapshot id** is the export's UTC date and a hash of the JSONL files. Re-exporting - unchanged data gives the same id. -- **Publishing a snapshot** means copying its directory (and `LATEST`) wherever consumers - read from, such as `gcloud storage rsync -r data/export gs://BUCKET/snapshots`. Consumers - check files against `manifest.json`. - -`isomorph lookup` answers queries against the database. A query can be: -- `distro[/suite]:package`; -- a source or binary package name, in any distribution; -- a repository URL or key. - -It reads the queries from its arguments, or one per line on stdin. With `-json` it writes one -answer per line. +Debian and Alpine take seconds. Arch and openSUSE need one or two files per package from their +build infrastructure, which isomorph fetches at about a request a second and caches, so the +first crawl takes a few hours and later runs are quick. + +`canonicalize` merges keys that name the same repository: renamed GitHub projects, GitLab +project IDs, PyPI and crates.io entries, KDE paths without a group. The GitHub token gets it +past the anonymous limit of 60 lookups an hour. + +Everything goes under `./data` unless you pass `-data`. + +To look something up: ``` $ isomorph lookup libjson-c5 @@ -80,97 +46,54 @@ libjson-c5: = arch/multilib:lib32-json-c 0.19-1 github.com/json-c/json-c (vcs-source) ``` -## Building the table +A query can be a source package (`arch:curl`, `debian/sid:curl`), a binary package name, or a +repository URL. With no arguments, `lookup` reads queries from stdin, one per line. `-json` +writes one result per line. -```sh -go build ./cmd/isomorph -isomorph bulk debian # one query of the Ultimate Debian Database -isomorph bulk arch -crawl 20000 # .SRCINFO per pkgbase, paced; later runs read the cache -isomorph bulk opensuse -crawl 7000 # spec and _service per unresolved package, paced -isomorph bulk alpine # APKINDEX and one streamed download of aports -GITHUB_TOKEN=... isomorph bulk canonicalize -isomorph bulk compare -all # agreement between distributions, as confusion matrices -isomorph export +## Output + +`isomorph export` writes a snapshot directory. It holds one JSONL file per distribution suite, +a SQLite copy of the same data, the JSON Schemas, and a `manifest.json` with each file's +sha256. Each line of a JSONL file is one source package: + +```json +{"distro": "arch", "suite": "core", "package": "json-c", "version": "0.19-1", + "upstream_key": "github.com/json-c/json-c", "repo": "https://github.com/json-c/json-c", + "status": "resolved", "basis": "vcs-source", "evidence": [...]} ``` -- **Storage:** the database is `data/packages.sqlite` (`-data DIR` moves it). HTTP responses - are cached in the user cache directory (`-cache DIR`). -- **Slices:** each `bulk ` run replaces that distribution's slices, so they can be - refreshed independently. -- **After a slice changes:** run `canonicalize` again. It maps keys that name the same - repository onto one: - - GitHub renames, and repositories that no longer exist; - - GitLab project ids; - - PyPI and crates.io metadata; - - KDE's project list; - - known mirrors. - -## Sources and evidence - -| distribution | where the mapping comes from | -|---|---| -| Debian | UDD: `Homepage`, DEP-12 `Repository`/`Repository-Browse`/`Bug-Database`, the watch URL, `.orig` tarball checksums | -| Arch | pacman databases, the packaging state repository (released commit per pkgbase), `.SRCINFO` or `PKGBUILD` at that commit | -| openSUSE | repository metadata (spec `Url:`), rb.zq1.de giturls, the OBS spec (`Source`, `#Git-Clone`) and `_service` | -| Alpine | the edge APKINDEX, and each APKBUILD's `url`, `source` and `_repo_url` | - -Each observation is weighted by how much its field can be trusted, and observations of the -same repository combine. A project site or download area that names no repository is mapped -by host rules (GNU and nongnu to Savannah, cgit to GitLab, Xfce, KDE, netfilter, Apache). - -A row's `basis` is the strongest kind of evidence behind its mapping: - -| basis | example | -|---|---| -| metadata | DEP-12 `Repository` | -| vcs-source | a git source in `.SRCINFO`, an openSUSE `_service` | -| release-url | a watch URL, a release tarball | -| derived | rb.zq1.de giturls | -| bug-tracker | DEP-12 `Bug-Database` | -| homepage | `Homepage`, `url` | - -Other distributions never decide a mapping. They are used only to measure agreement, and to -choose which keys are worth a GitHub lookup. +`upstream_key` is the field to join on. + +`basis` records what kind of evidence decided the mapping: a DEP-12 `Repository` field, a git +source, a release URL, a homepage. They are not equally reliable, and the evaluation measures +each one. The full format is in [schema/package.schema.json](schema/package.schema.json). + +In the SQLite database, the `counterparts` view pairs each package with its counterparts in +the other distributions. ## How well it works -From [docs/EVALUATION.md](docs/EVALUATION.md), 2026-09-28: - -- **Coverage:** Arch 85.3%, Alpine 78.3%, Debian 77.9%, openSUSE 75.0% of source packages map - to a repository. -- **Debian's DEP-12 `Repository` as ground truth:** without it, the other fields resolve 94.6% - of those packages and agree with it on 96.8%. -- **Same-named packages** in two distributions map to the same repository 90.7% to 93.8% of - the time. The name join also pairs unrelated software, so this understates agreement. -- **Packages built from the identical release tarball** (Debian and Arch) agree on 96.4%. - -## Crawling politely - -- **Pacing:** every request goes through one client with a disk cache and per-host pacing. - crates.io and the OBS API get one request a second, and gitlab.archlinux.org stays under - its published anonymous limit. -- **Rate limits:** the client waits out short rate-limit windows and stops at long ones - rather than retrying into them. -- **Contact:** set `ISOMORPH_CONTACT`, or write `~/.config/isomorph/contact`, so - operators can reach you. It is appended to the User-Agent, and crates.io is not queried - without it. -- **Tokens:** - - `GITHUB_TOKEN` lifts GitHub's anonymous limit of 60 lookups an hour. - - `ARCH_GITLAB_TOKEN` lifts gitlab.archlinux.org's anonymous limit. The crawl does not need - it. - -## Limitations - -- **Packages whose only URL is a project website stay unresolved.** Nothing in their - packaging names a repository. -- **Stale packaging data wins.** When a distribution's data points at a project's old home - (a SourceForge download, a repository that moved), the mapping follows it. The evaluation - counts these cases. -- **One repository can build many packages,** and a package built from a monorepo maps to the - monorepo. -- **Arch and Alpine keep no source order,** so a dependency's git source can outrank the main - tarball. - -## Licence - -MIT; see [LICENSE](LICENSE). +Between 75% and 85% of source packages resolve, depending on the distribution. + +Many Debian packages declare their repository in DEP-12 metadata. When isomorph is given the +rest of such a package's fields and not that one, it names the same repository 96.8% of the +time. + +Same-named packages in two distributions resolve to the same repository 91 to 94% of the +time. That understates agreement, since some of those pairs are different software. + +[docs/EVALUATION.md](docs/EVALUATION.md) has the details, including an error analysis. + +Most packages that don't resolve name nothing but a project website. isomorph doesn't copy one +distribution's answer onto another's package, so if Arch's packaging says where a project lives +and openSUSE's doesn't, the openSUSE package stays unresolved. + +## Crawling + +Set `ISOMORPH_CONTACT`, or write `~/.config/isomorph/contact`, to an address or URL where +server operators can reach you. It goes in the User-Agent, and isomorph won't query crates.io +without it. Responses are cached in your user cache directory. + +## License + +MIT -- 2.51.2