# isomorph

isomorph works out which upstream repository each source package in a Linux distribution is
built from. Packages that come from the same repository are the same software, whatever
their distributions call them: Debian's `requests`, Arch's `python-requests` and Alpine's
`py3-requests` all resolve to `github.com/psf/requests`. It also works the other way. The
`fff` in Debian and the `fff` in openSUSE are unrelated projects, and they resolve to different
repositories.

It covers Debian sid, Arch Linux, openSUSE Tumbleweed and Alpine edge. Every mapping is made
from the distribution's own packaging (a watch file, an Arch `source=` array, an openSUSE spec)
and records the evidence it came from.

## Usage

```sh
go build ./cmd/isomorph

isomorph bulk debian
isomorph bulk arch -crawl 20000
isomorph bulk opensuse -crawl 7000
isomorph bulk alpine
GITHUB_TOKEN=... isomorph bulk canonicalize
isomorph export
```

Debian and Alpine take seconds. Arch and openSUSE need one or two files per package from their
build infrastructure, which isomorph fetches at about a request a second and caches, so the
first crawl takes a few hours and later runs are quick.

`canonicalize` merges keys that name the same repository: renamed GitHub projects, GitLab
project IDs, PyPI and crates.io entries, KDE paths without a group. The GitHub token gets it
past the anonymous limit of 60 lookups an hour.

Everything goes under `./data` unless you pass `-data`.

To look something up:

```
$ isomorph lookup libjson-c5
libjson-c5:
  debian/sid:json-c 0.19+ds-1  github.com/json-c/json-c (release-url) [via binary]
  opensuse/tumbleweed:json-c 0.19-1.2  github.com/json-c/json-c (derived) [via binary]
    = alpine/edge-main:json-c 0.19-r1  github.com/json-c/json-c (homepage)
    = arch/core:json-c 0.19-1  github.com/json-c/json-c (vcs-source)
    = arch/multilib:lib32-json-c 0.19-1  github.com/json-c/json-c (vcs-source)
```

A query can be a source package (`arch:curl`, `debian/sid:curl`), a binary package name, or a
repository URL. With no arguments, `lookup` reads queries from stdin, one per line. `-json`
writes one result per line.

## Output

`isomorph export` writes a snapshot directory. It holds one JSONL file per distribution suite,
a SQLite copy of the same data, the JSON Schemas, and a `manifest.json` with each file's
sha256. Each line of a JSONL file is one source package:

```json
{"distro": "arch", "suite": "core", "package": "json-c", "version": "0.19-1",
 "upstream_key": "github.com/json-c/json-c", "repo": "https://github.com/json-c/json-c",
 "status": "resolved", "basis": "vcs-source", "evidence": [...]}
```

`upstream_key` is the field to join on.

`basis` records what kind of evidence decided the mapping: a DEP-12 `Repository` field, a git
source, a release URL, a homepage. They are not equally reliable, and the evaluation measures
each one. The full format is in [schema/package.schema.json](schema/package.schema.json).

In the SQLite database, the `counterparts` view pairs each package with its counterparts in
the other distributions.

## How well it works

Between 74% and 86% of source packages resolve, depending on the distribution.

Many Debian packages declare their repository in DEP-12 metadata. When isomorph is given the
rest of such a package's fields and not that one, it names the same repository 96.8% of the
time.

Same-named packages in two distributions resolve to the same repository 91 to 94% of the
time. That understates agreement, since some of those pairs are different software.

[docs/EVALUATION.md](docs/EVALUATION.md) has the details, including an error analysis.

Most packages that don't resolve name nothing but a project website. isomorph doesn't copy one
distribution's answer onto another's package, so if Arch's packaging says where a project lives
and openSUSE's doesn't, the openSUSE package stays unresolved.

## Crawling

Set `ISOMORPH_CONTACT`, or write `~/.config/isomorph/contact`, to an address or URL where
server operators can reach you. It goes in the User-Agent, and isomorph won't query crates.io
without it. Responses are cached in your user cache directory.

## License

MIT