isomorph #

isomorph works out which upstream repository each source package in a Linux distribution is built from. Packages that come from the same repository are the same software, whatever their distributions call them: Debian's requests, Arch's python-requests and Alpine's py3-requests all resolve to github.com/psf/requests. It also works the other way. The fff in Debian and the fff in openSUSE are unrelated projects, and they resolve to different repositories.

It covers Debian sid, Arch Linux, openSUSE Tumbleweed and Alpine edge. Every mapping is made from the distribution's own packaging (a watch file, an Arch source= array, an openSUSE spec) and records the evidence it came from.

Usage #

go build ./cmd/isomorph

isomorph bulk debian
isomorph bulk arch -crawl 20000
isomorph bulk opensuse -crawl 7000
isomorph bulk alpine
GITHUB_TOKEN=... isomorph bulk canonicalize
isomorph export

Debian and Alpine take seconds. Arch and openSUSE need one or two files per package from their build infrastructure, which isomorph fetches at about a request a second and caches, so the first crawl takes a few hours and later runs are quick.

canonicalize merges keys that name the same repository: renamed GitHub projects, GitLab project IDs, PyPI and crates.io entries, KDE paths without a group. The GitHub token gets it past the anonymous limit of 60 lookups an hour.

Everything goes under ./data unless you pass -data.

To look something up:

$ isomorph lookup libjson-c5
libjson-c5:
  debian/sid:json-c 0.19+ds-1  github.com/json-c/json-c (release-url) [via binary]
  opensuse/tumbleweed:json-c 0.19-1.2  github.com/json-c/json-c (derived) [via binary]
    = alpine/edge-main:json-c 0.19-r1  github.com/json-c/json-c (homepage)
    = arch/core:json-c 0.19-1  github.com/json-c/json-c (vcs-source)
    = arch/multilib:lib32-json-c 0.19-1  github.com/json-c/json-c (vcs-source)

A query can be a source package (arch:curl, debian/sid:curl), a binary package name, or a repository URL. With no arguments, lookup reads queries from stdin, one per line. -json writes one result per line.

Output #

isomorph export writes a snapshot directory. It holds one JSONL file per distribution suite, a SQLite copy of the same data, the JSON Schemas, and a manifest.json with each file's sha256. Each line of a JSONL file is one source package:

{"distro": "arch", "suite": "core", "package": "json-c", "version": "0.19-1",
 "upstream_key": "github.com/json-c/json-c", "repo": "https://github.com/json-c/json-c",
 "status": "resolved", "basis": "vcs-source", "evidence": [...]}

upstream_key is the field to join on.

basis records what kind of evidence decided the mapping: a DEP-12 Repository field, a git source, a release URL, a homepage. They are not equally reliable, and the evaluation measures each one. The full format is in schema/package.schema.json.

In the SQLite database, the counterparts view pairs each package with its counterparts in the other distributions.

How well it works #

Between 74% and 86% of source packages resolve, depending on the distribution.

Many Debian packages declare their repository in DEP-12 metadata. When isomorph is given the rest of such a package's fields and not that one, it names the same repository 96.8% of the time.

Same-named packages in two distributions resolve to the same repository 91 to 94% of the time. That understates agreement, since some of those pairs are different software.

docs/EVALUATION.md has the details, including an error analysis.

Most packages that don't resolve name nothing but a project website. isomorph doesn't copy one distribution's answer onto another's package, so if Arch's packaging says where a project lives and openSUSE's doesn't, the openSUSE package stays unresolved.

Crawling #

Set ISOMORPH_CONTACT, or write ~/.config/isomorph/contact, to an address or URL where server operators can reach you. It goes in the User-Agent, and isomorph won't query crates.io without it. Responses are cached in your user cache directory.

License #

MIT