# Evaluation

How well does each distribution's mapping name the right upstream repository, and how
often do distributions agree? This document measures three things:

1. **Coverage:** the share of source packages mapped to a repository.
2. **Agreement with a ground truth:** Debian's DEP-12 `Repository` field is the one field
   whose purpose is to name the upstream repository. Its packages are resolved again without
   it, and the result is compared with it.
3. **Agreement between distributions:** same-named packages, and packages built from the
   identical release tarball, should map to the same repository.

The numbers below come from one run on 2026-09-28 over Debian sid/main, Arch (core, extra,
multilib), openSUSE Tumbleweed and Alpine edge (main, community). "How to reproduce" at the
end has the commands.

Every mapping comes from its own distribution's data: the packaging, and the upstream
projects' own services (GitHub, KDE's project list, PyPI, crates.io). Other distributions
are used only to measure agreement, and to choose which keys are worth a GitHub lookup.

## Coverage

| distribution | source packages | mapped to a repository |
|---|---:|---:|
| Arch | 12,358 | 85.3% |
| Alpine edge | 9,494 | 78.3% |
| Debian sid | 41,417 | 77.9% |
| openSUSE Tumbleweed | 17,211 | 75.0% |

Every mapping records its **basis**, the strongest kind of evidence behind it:

| basis | Debian | Arch | openSUSE | Alpine |
|---|---:|---:|---:|---:|
| metadata (DEP-12 `Repository`) | 11,404 | | | |
| vcs-source (a git source, `_service` scm, `_repo_url`) | | 4,367 | 564 | 453 |
| release-url (watch URL, release tarball) | 15,984 | 5,014 | 1,192 | 6,693 |
| derived (rb.zq1.de giturls) | | | 6,399 | |
| bug-tracker (DEP-12 `Bug-Database`) | 68 | | | |
| homepage | 4,790 | 1,157 | 4,757 | 285 |

The basis stands in for a confidence score. Measured agreement per basis, below, says how
far each kind of evidence can be trusted.

- **Arch is a moving target.** About 1,100 pkgbases were released after the `.SRCINFO`
  crawl, so their new packaging commits were not cached for this run and those rows use the
  repository database's url.

## Agreement with Debian's DEP-12 `Repository`

10,872 Debian packages have a DEP-12 `Repository` that names a repository. Each is resolved
again without `Repository` and `Repository-Browse`:

- The remaining fields (watch URL, Bug-Database, Homepage) resolve **10,283 (94.6%)**.
- They agree with `Repository` on **9,950 (96.8%)**.

| basis of the fallback | resolved | agree |
|---|---:|---:|
| release-url | 6,547 | 96.7% |
| bug-tracker | 3,624 | 97.3% |
| homepage | 112 | 81.2% |

- **Why homepage agreement is lower:** most of its misses are GitHub renames (this check does
  not apply aliases), umbrella project pages (gtk2-perl's SourceForge project), and the GNU
  Octave packages repository.
- **Limits:** DEP-12 is itself sometimes stale or wrong, so 96.8% is a lower bound on the
  fallbacks' precision. Packages with DEP-12 metadata are also better maintained than the
  average.

The same field scores the other distributions, for their same-named packages:

| distribution | basis | packages | agree with DEP-12 |
|---|---|---:|---:|
| Arch | release-url | 427 | 94.4% |
| Arch | vcs-source | 415 | 93.3% |
| Arch | homepage | 16 | 87.5% |
| Alpine | release-url | 272 | 92.6% |
| Alpine | homepage | 7 | 57.1% |
| openSUSE | derived | 556 | 95.3% |
| openSUSE | homepage | 227 | 88.5% |
| openSUSE | release-url | 70 | 87.1% |
| openSUSE | vcs-source | 29 | 93.1% |

Most remaining misses are projects that moved (soundtouch to Codeberg, pmbootstrap to
postmarketOS's GitLab), where the other distribution is newer than Debian's field, or
different software with the same name.

## Agreement between distributions

`isomorph bulk compare -all` joins every pair of distributions twice, independently of
the mappings:
- by package name;
- for Debian and Arch, by identical source tarball: the sha256 of Debian's `.orig` tarball
  and of Arch's release source.

Each join is shown as a confusion matrix of one distribution's outcome against the other's.

**By package name.** Each row flattens one matrix.

| a vs b | both, agree | both, differ | only a | only b | neither | agree where both |
|---|---:|---:|---:|---:|---:|---:|
| Arch vs Debian | 4,584 | 304 | 611 | 156 | 574 | 93.8% |
| Alpine vs Arch | 2,857 | 192 | 54 | 281 | 681 | 93.7% |
| Alpine vs Debian | 2,694 | 194 | 236 | 133 | 478 | 93.3% |
| Arch vs openSUSE | 3,882 | 356 | 355 | 89 | 321 | 91.6% |
| Alpine vs openSUSE | 2,318 | 231 | 138 | 123 | 378 | 90.9% |
| Debian vs openSUSE | 3,898 | 401 | 376 | 367 | 694 | 90.7% |

- **The name join understates agreement.** It pairs different software that shares a name
  (fff, hyperlink, forks of mg), so "differ" overstates real disagreement.
- **Arch fills in for Debian.** 611 shared names resolve in Arch but not in Debian, against
  156 the other way.

**By identical source tarball (Debian vs Arch):** 1,162 pairs; 668 resolved on both sides,
of which 644 (96.4%) agree. This join needs no names. It stays small because Debian repacks
tarballs and Arch builds many packages from git.

### Before the error-analysis fixes

The morning run on 2026-09-28 came before the fixes below and before the openSUSE spec crawl.

| | before | after |
|---|---:|---:|
| Arch coverage | 83.8% | 85.3% |
| Alpine coverage | 75.4% | 78.3% |
| Debian coverage | 76.7% | 77.9% |
| openSUSE coverage | 65.3% | 75.0% |
| Arch vs Debian | 93.0% of 4,662 | 93.8% of 4,888 |
| Alpine vs Arch | 92.1% of 2,809 | 93.7% of 3,049 |
| Alpine vs Debian | 92.7% of 2,631 | 93.3% of 2,888 |
| Arch vs openSUSE | 89.2% of 3,547 | 91.6% of 4,238 |
| Alpine vs openSUSE | 89.6% of 1,972 | 90.9% of 2,549 |
| Debian vs openSUSE | 90.2% of 3,608 | 90.7% of 4,299 |

## Error analysis

**Sample.** It was drawn from the morning run with seed 20260928, and stratified:
- 8 pairs from each of the six "both, differ" cells, 48 in total;
- 50 packages that are unresolved while a same-named package elsewhere resolves: 20 from
  openSUSE and 10 from each other distribution.

**Method.** Each item was labelled from its evidence in all four distributions, and some were
checked against the GitHub API. Shares are weighted by stratum size; with about 50 items per
sample they carry roughly ±10 points.
- The sampler is `eval/sample.py`.
- The labels are in `eval/errors-2026-09-28.tsv`.

**Why two resolved packages differed** (1,684 pairs):

| cause | sampled | share | examples |
|---|---:|---:|---|
| stale or dead distribution data: the row names only an old home | 12 | 25% | SourceForge downloads for GitHub projects, notabug to Codeberg, repositories that 404 |
| wrong repository in openSUSE's giturls, or an umbrella homepage | 7 | 18% | awesome-releases, the parent hfst, dead github.com/freedesktop mirrors |
| selection: the right repository was in the row but lost | 9 | 17% | a SourceForge or Launchpad homepage beat a GitHub release URL |
| different software sharing a name | 8 | 15% | hyperlink, fff, forks of mg, each distribution's ca-certificates |
| same project, both keys valid | 7 | 15% | GitHub mirrors, renames not yet canonicalised |
| URL mapping | 5 | 10% | `*.wiki.kernel.org` to Linux, `download.samba.org/pub/rsync` to Samba |

**Why a package was unresolved while a counterpart resolved** (3,456 packages):

| cause | sampled | share |
|---|---:|---:|
| only a project website, no repository or forge URL (openSUSE: 16 of 20) | 27 | 55% |
| a host rule would map the homepage (docs.xfce.org, gnu.org/software, netfilter.org, apps.kde.org) | 11 | 23% |
| the evidence named the repository but the key was incomplete (KDE tarballs name no group; an unevaluated `case` in APKBUILDs) | 8 | 12% |
| no URL at all, a Debian-native package, or a registry entry naming no repository | 4 | 10% |

**What was fixed.** Each fix was counted over all packages, not only the sample.

1. **KDE.**
   - Keys without a group take it from KDE's project list; 489 packages were unresolved this
     way.
   - The APKBUILD evaluator takes the matching branch of a `case` over a known variable
     (47 Alpine Plasma packages), and reads `_repo_url`.
2. **Host rules.** The findvcs-style rules the openSUSE resolver used (GNU and nongnu to
   Savannah, cgit to GitLab, Apache, GitHub Pages) now apply to every distribution. About 435
   rows had a homepage one of them covers. New rules cover docs.xfce.org, netfilter.org and
   freedesktop.org/software/pulseaudio.
3. **SourceForge and Launchpad homepages.** A homepage on these hosts no longer beats a
   release or VCS source naming another repository (130 packages).
   - Against DEP-12, where such a homepage and the watch URL disagreed, the watch URL was
     right 22 times out of 23.
   - GitHub homepages split 16 to 17, so the rule is limited to these two hosts.
4. **URL mapping:**
   - kernel.org project subdomains (66 packages) and samba.org's other projects (23) no longer
     hint at Linux or Samba;
   - GitLab API URLs that name a project by path now name it.
5. **Canonicalisation** checks conflicts across every pair of distributions. It aliases
   repositories that no longer exist (GitHub answers 404) to nothing, so the package falls back
   to its next candidate.
6. **openSUSE's gap was missing data.** Its repository metadata carries only the spec
   `Url:`. The OBS spec and `_service` of the ~5,900 unresolved packages are now read, one
   request a second; coverage went from 65.3% to 75.0%.
7. **Secondary spec sources** (`Source1` and later: configuration files, completions, test
   data) corroborate a candidate but never decide one. Of the 43 packages they alone
   resolved, about a third named another project.

**Measured and not changed.** When openSUSE's giturls and spec `Url:` disagree (214
packages), neither is reliably right. On the 22 with a DEP-12 `Repository`, giturls matches
in 12 and `Url:` in 9, so their precedence stays.

**What remains:**
- **Stale or wrong distribution data** was about 43% of the disagreements before the fixes.
  Manual overrides or cross-distribution consensus could address it; neither is built. Each
  distribution's mapping stays its own.
- **Packages whose only URL is a project website.**
- **Arch and Alpine source order.** Neither records which source is primary, so a
  dependency's git source can still outrank the main tarball (telegram-desktop).

## Fixes from the osvrb joins (2026-09-30)

Joining osvrb's data by upstream (osvrb's `validation/2026-09-30-identity-join.md`) turned up
mappings where a site stands for more projects than the one it names:
- **python.org:** PyPy's releases on `downloads.python.org` and PyPI-era pages on
  `packages.python.org` mapped to CPython.
- **gnupg.org:** its pages for gpgme, libgcrypt and gpa mapped to GnuPG.
- **Octave Forge:** every package mapped to one repository.
- **Flagship sites:** `curl.se/trurl` mapped to curl, and `sphinx-doc.org` to Sphinx for the
  sphinxcontrib packages.
- **PyPI descriptions:** python-barbicanclient's description, which cites the barbican
  server, mapped it to barbican.

The fixes:
- **Catalogue sites name no repository:** Octave Forge, the Octave package index, PEAR, and
  homalg's package pages. Octave Forge's per-package repositories
  (`sourceforge.net/p/octave/<package>`) are keyed as their own.
- **python.org's other subdomains:** PyPI-era pages become PyPI entries, and PyPy's releases
  map to PyPy.
- **gnupg.org's project pages** map to each project's repository.
- **A project site** decides a package only if the package is named like the site's
  repository (`bulk/flagship.go`). A forge page or a source URL still decides any package.
- **PyPI descriptions:** a repository cited in a description counts only if it is named like
  the project.
- **OpenStack:** opendev.org is recognised, with OpenStack's GitHub mirrors mapped to it.
- **Registry aliases** are recomputed in full, so stale ones are replaced.

| | before | after |
|---|---:|---:|
| Debian coverage | 77.9% | 77.8% |
| Arch coverage | 85.6% | 85.6% |
| openSUSE coverage | 75.0% | 74.3% |
| Alpine coverage | 78.3% | 78.2% |
| DEP-12 held-out agreement | 96.8% | 96.8% |
| held-out, homepage basis | 81.2% (112) | 84.9% (106) |
| Arch vs Debian | 93.8% of 4,904 | 94.1% of 4,900 |
| Alpine vs Arch | 93.8% of 3,061 | 94.0% of 3,058 |
| Alpine vs Debian | 93.3% of 2,889 | 93.7% of 2,886 |
| Arch vs openSUSE | 91.6% of 4,255 | 92.1% of 4,276 |
| Alpine vs openSUSE | 91.0% of 2,551 | 91.6% of 2,548 |
| Debian vs openSUSE | 90.7% of 4,300 | 91.2% of 4,357 |

- **Coverage** falls slightly: a package a site wrongly claimed is now unresolved instead.
- **Agreement** rises in every pair.
- **Known misses:**
  - The name guard lets through three openSUSE flavour packages (`python313-pyside6`,
    `-shiboken6`, `-packaging`), which still map to CPython.
  - openSUSE's giturls still names another project's repository for a few packages, such as
    java-11-openj9.

## How to reproduce

```sh
isomorph bulk debian                # one UDD query
isomorph bulk arch -crawl 20000     # paced .SRCINFO crawl; later runs read the cache
isomorph bulk opensuse -crawl 7000  # paced OBS spec/_service crawl
isomorph bulk alpine                # one streamed aports download
GITHUB_TOKEN=... isomorph bulk canonicalize
isomorph bulk compare -all
python3 eval/sample.py differ         # the error-analysis sample
```

- **Held-out check:** `bulk debian` prints it. `-heldout FILE` writes each disagreement.
- **`GITHUB_TOKEN`:** anonymous GitHub API use is capped at 60 requests an hour.