Evaluation #

How well does each distribution's mapping name the right upstream repository, and how often do distributions agree? This document measures three things:

1. Coverage: the share of source packages mapped to a repository.
2. Agreement with a ground truth: Debian's DEP-12 Repository field is the one field whose purpose is to name the upstream repository. Its packages are resolved again without it, and the result is compared with it.
3. Agreement between distributions: same-named packages, and packages built from the identical release tarball, should map to the same repository.

The numbers below come from one run on 2026-09-28 over Debian sid/main, Arch (core, extra, multilib), openSUSE Tumbleweed and Alpine edge (main, community). "How to reproduce" at the end has the commands.

Every mapping comes from its own distribution's data: the packaging, and the upstream projects' own services (GitHub, KDE's project list, PyPI, crates.io). Other distributions are used only to measure agreement, and to choose which keys are worth a GitHub lookup.

Coverage #

distribution	source packages	mapped to a repository
Arch	12,358	85.3%
Alpine edge	9,494	78.3%
Debian sid	41,417	77.9%
openSUSE Tumbleweed	17,211	75.0%

Every mapping records its basis, the strongest kind of evidence behind it:

basis	Debian	Arch	openSUSE	Alpine
metadata (DEP-12 Repository)	11,404
vcs-source (a git source, _service scm, _repo_url)		4,367	564	453
release-url (watch URL, release tarball)	15,984	5,014	1,192	6,693
derived (rb.zq1.de giturls)			6,399
bug-tracker (DEP-12 Bug-Database)	68
homepage	4,790	1,157	4,757	285

The basis stands in for a confidence score. Measured agreement per basis, below, says how far each kind of evidence can be trusted.

• Arch is a moving target. About 1,100 pkgbases were released after the .SRCINFO crawl, so their new packaging commits were not cached for this run and those rows use the repository database's url.

Agreement with Debian's DEP-12 Repository #

10,872 Debian packages have a DEP-12 Repository that names a repository. Each is resolved again without Repository and Repository-Browse:

• The remaining fields (watch URL, Bug-Database, Homepage) resolve 10,283 (94.6%).
• They agree with Repository on 9,950 (96.8%).

basis of the fallback	resolved	agree
release-url	6,547	96.7%
bug-tracker	3,624	97.3%
homepage	112	81.2%

• Why homepage agreement is lower: most of its misses are GitHub renames (this check does not apply aliases), umbrella project pages (gtk2-perl's SourceForge project), and the GNU Octave packages repository.
• Limits: DEP-12 is itself sometimes stale or wrong, so 96.8% is a lower bound on the fallbacks' precision. Packages with DEP-12 metadata are also better maintained than the average.

The same field scores the other distributions, for their same-named packages:

distribution	basis	packages	agree with DEP-12
Arch	release-url	427	94.4%
Arch	vcs-source	415	93.3%
Arch	homepage	16	87.5%
Alpine	release-url	272	92.6%
Alpine	homepage	7	57.1%
openSUSE	derived	556	95.3%
openSUSE	homepage	227	88.5%
openSUSE	release-url	70	87.1%
openSUSE	vcs-source	29	93.1%

Most remaining misses are projects that moved (soundtouch to Codeberg, pmbootstrap to postmarketOS's GitLab), where the other distribution is newer than Debian's field, or different software with the same name.

Agreement between distributions #

isomorph bulk compare -all joins every pair of distributions twice, independently of the mappings:

• by package name;
• for Debian and Arch, by identical source tarball: the sha256 of Debian's .orig tarball and of Arch's release source.

Each join is shown as a confusion matrix of one distribution's outcome against the other's.

By package name. Each row flattens one matrix.

a vs b	both, agree	both, differ	only a	only b	neither	agree where both
Arch vs Debian	4,584	304	611	156	574	93.8%
Alpine vs Arch	2,857	192	54	281	681	93.7%
Alpine vs Debian	2,694	194	236	133	478	93.3%
Arch vs openSUSE	3,882	356	355	89	321	91.6%
Alpine vs openSUSE	2,318	231	138	123	378	90.9%
Debian vs openSUSE	3,898	401	376	367	694	90.7%

• The name join understates agreement. It pairs different software that shares a name (fff, hyperlink, forks of mg), so "differ" overstates real disagreement.
• Arch fills in for Debian. 611 shared names resolve in Arch but not in Debian, against 156 the other way.

By identical source tarball (Debian vs Arch): 1,162 pairs; 668 resolved on both sides, of which 644 (96.4%) agree. This join needs no names. It stays small because Debian repacks tarballs and Arch builds many packages from git.

Before the error-analysis fixes #

The morning run on 2026-09-28 came before the fixes below and before the openSUSE spec crawl.

	before	after
Arch coverage	83.8%	85.3%
Alpine coverage	75.4%	78.3%
Debian coverage	76.7%	77.9%
openSUSE coverage	65.3%	75.0%
Arch vs Debian	93.0% of 4,662	93.8% of 4,888
Alpine vs Arch	92.1% of 2,809	93.7% of 3,049
Alpine vs Debian	92.7% of 2,631	93.3% of 2,888
Arch vs openSUSE	89.2% of 3,547	91.6% of 4,238
Alpine vs openSUSE	89.6% of 1,972	90.9% of 2,549
Debian vs openSUSE	90.2% of 3,608	90.7% of 4,299

Error analysis #

Sample. It was drawn from the morning run with seed 20260928, and stratified:

• 8 pairs from each of the six "both, differ" cells, 48 in total;
• 50 packages that are unresolved while a same-named package elsewhere resolves: 20 from openSUSE and 10 from each other distribution.

Method. Each item was labelled from its evidence in all four distributions, and some were checked against the GitHub API. Shares are weighted by stratum size; with about 50 items per sample they carry roughly ±10 points.

• The sampler is eval/sample.py.
• The labels are in eval/errors-2026-09-28.tsv.

Why two resolved packages differed (1,684 pairs):

cause	sampled	share	examples
stale or dead distribution data: the row names only an old home	12	25%	SourceForge downloads for GitHub projects, notabug to Codeberg, repositories that 404
wrong repository in openSUSE's giturls, or an umbrella homepage	7	18%	awesome-releases, the parent hfst, dead github.com/freedesktop mirrors
selection: the right repository was in the row but lost	9	17%	a SourceForge or Launchpad homepage beat a GitHub release URL
different software sharing a name	8	15%	hyperlink, fff, forks of mg, each distribution's ca-certificates
same project, both keys valid	7	15%	GitHub mirrors, renames not yet canonicalised
URL mapping	5	10%	*.wiki.kernel.org to Linux, download.samba.org/pub/rsync to Samba

Why a package was unresolved while a counterpart resolved (3,456 packages):

cause	sampled	share
only a project website, no repository or forge URL (openSUSE: 16 of 20)	27	55%
a host rule would map the homepage (docs.xfce.org, gnu.org/software, netfilter.org, apps.kde.org)	11	23%
the evidence named the repository but the key was incomplete (KDE tarballs name no group; an unevaluated case in APKBUILDs)	8	12%
no URL at all, a Debian-native package, or a registry entry naming no repository	4	10%

What was fixed. Each fix was counted over all packages, not only the sample.

1. KDE.
   • Keys without a group take it from KDE's project list; 489 packages were unresolved this way.
   • The APKBUILD evaluator takes the matching branch of a case over a known variable (47 Alpine Plasma packages), and reads _repo_url.
2. Host rules. The findvcs-style rules the openSUSE resolver used (GNU and nongnu to Savannah, cgit to GitLab, Apache, GitHub Pages) now apply to every distribution. About 435 rows had a homepage one of them covers. New rules cover docs.xfce.org, netfilter.org and freedesktop.org/software/pulseaudio.
3. SourceForge and Launchpad homepages. A homepage on these hosts no longer beats a release or VCS source naming another repository (130 packages).
   • Against DEP-12, where such a homepage and the watch URL disagreed, the watch URL was right 22 times out of 23.
   • GitHub homepages split 16 to 17, so the rule is limited to these two hosts.
4. URL mapping:
   • kernel.org project subdomains (66 packages) and samba.org's other projects (23) no longer hint at Linux or Samba;
   • GitLab API URLs that name a project by path now name it.
5. Canonicalisation checks conflicts across every pair of distributions. It aliases repositories that no longer exist (GitHub answers 404) to nothing, so the package falls back to its next candidate.
6. openSUSE's gap was missing data. Its repository metadata carries only the spec Url:. The OBS spec and _service of the ~5,900 unresolved packages are now read, one request a second; coverage went from 65.3% to 75.0%.
7. Secondary spec sources (Source1 and later: configuration files, completions, test data) corroborate a candidate but never decide one. Of the 43 packages they alone resolved, about a third named another project.

Measured and not changed. When openSUSE's giturls and spec Url: disagree (214 packages), neither is reliably right. On the 22 with a DEP-12 Repository, giturls matches in 12 and Url: in 9, so their precedence stays.

What remains:

• Stale or wrong distribution data was about 43% of the disagreements before the fixes. Manual overrides or cross-distribution consensus could address it; neither is built. Each distribution's mapping stays its own.
• Packages whose only URL is a project website.
• Arch and Alpine source order. Neither records which source is primary, so a dependency's git source can still outrank the main tarball (telegram-desktop).

Fixes from the osvrb joins (2026-09-30) #

Joining osvrb's data by upstream (osvrb's validation/2026-09-30-identity-join.md) turned up mappings where a site stands for more projects than the one it names:

• python.org: PyPy's releases on downloads.python.org and PyPI-era pages on packages.python.org mapped to CPython.
• gnupg.org: its pages for gpgme, libgcrypt and gpa mapped to GnuPG.
• Octave Forge: every package mapped to one repository.
• Flagship sites: curl.se/trurl mapped to curl, and sphinx-doc.org to Sphinx for the sphinxcontrib packages.
• PyPI descriptions: python-barbicanclient's description, which cites the barbican server, mapped it to barbican.

The fixes:

• Catalogue sites name no repository: Octave Forge, the Octave package index, PEAR, and homalg's package pages. Octave Forge's per-package repositories (sourceforge.net/p/octave/<package>) are keyed as their own.
• python.org's other subdomains: PyPI-era pages become PyPI entries, and PyPy's releases map to PyPy.
• gnupg.org's project pages map to each project's repository.
• A project site decides a package only if the package is named like the site's repository (bulk/flagship.go). A forge page or a source URL still decides any package.
• PyPI descriptions: a repository cited in a description counts only if it is named like the project.
• OpenStack: opendev.org is recognised, with OpenStack's GitHub mirrors mapped to it.
• Registry aliases are recomputed in full, so stale ones are replaced.

	before	after
Debian coverage	77.9%	77.8%
Arch coverage	85.6%	85.6%
openSUSE coverage	75.0%	74.3%
Alpine coverage	78.3%	78.2%
DEP-12 held-out agreement	96.8%	96.8%
held-out, homepage basis	81.2% (112)	84.9% (106)
Arch vs Debian	93.8% of 4,904	94.1% of 4,900
Alpine vs Arch	93.8% of 3,061	94.0% of 3,058
Alpine vs Debian	93.3% of 2,889	93.7% of 2,886
Arch vs openSUSE	91.6% of 4,255	92.1% of 4,276
Alpine vs openSUSE	91.0% of 2,551	91.6% of 2,548
Debian vs openSUSE	90.7% of 4,300	91.2% of 4,357

• Coverage falls slightly: a package a site wrongly claimed is now unresolved instead.
• Agreement rises in every pair.
• Known misses:
  • The name guard lets through three openSUSE flavour packages (python313-pyside6, -shiboken6, -packaging), which still map to CPython.
  • openSUSE's giturls still names another project's repository for a few packages, such as java-11-openj9.

How to reproduce #

isomorph bulk debian                # one UDD query
isomorph bulk arch -crawl 20000     # paced .SRCINFO crawl; later runs read the cache
isomorph bulk opensuse -crawl 7000  # paced OBS spec/_service crawl
isomorph bulk alpine                # one streamed aports download
GITHUB_TOKEN=... isomorph bulk canonicalize
isomorph bulk compare -all
python3 eval/sample.py differ         # the error-analysis sample

• Held-out check: bulk debian prints it. -heldout FILE writes each disagreement.
• GITHUB_TOKEN: anonymous GitHub API use is capped at 60 requests an hour.