Reference playlist tooling #

Rebuilds reference.txt, the tuning target, from the playlist exports listed in playlists.md. Run from this directory, in order:

python3 fetch_soundcloud.py   # writes sc_tracks.json
python3 fetch_youtube.py      # writes yt_tracks.json
python3 build_reference.py    # writes ../reference.txt

Notes on why each script looks the way it does:

• SoundCloud sets must be public. fetch_soundcloud.py scrapes a client_id out of the site's own JS bundles and calls api-v2.soundcloud.com; the resolve endpoint returns nothing for private sets. Only the first handful of tracks arrive with titles, so the rest are fetched by id in batches of 50.
• YouTube playlists must be public too. Contents render client-side, so fetch_youtube.py parses ytInitialData out of the playlist page and reads lockupViewModel entries. The RSS feed at feeds/videos.xml?playlist_id= is useful only as a public/private check — it caps at ~14 entries.
• Artist names need resolving, not just reading. SoundCloud usernames are uploaders rather than artists (theRABBLE uploading Nosaj Thing - Aquarium), and many YouTube titles carry the artist while others leave it to the channel. Both scripts prefer publisher metadata, then an Artist - Title split, then the uploader or channel.
• Seeds are inferred, not recorded. The exports do not preserve the seed artist, so build_reference.py writes the most frequent artist in each playlist as # inferred-seed:. Scoring is per playlist against its own seed; scoring against the union of all playlists is meaningless because the playlists come from different seeds.

Lexicon publishing #

publish-lexicons.js files every schema in ../lexicons/ into an atproto account as a com.atproto.lexicon.schema record, keyed at its own NSID — the mechanism that makes a record type like space.burrito.morelike.playlist discoverable rather than merely written. Defaults to a dry run:

node tools/publish-lexicons.js            # prints what would be published; no credential needed
node --env-file=.env tools/publish-lexicons.js --apply   # actually publishes

--apply needs LEXICON_PDS_URL, LEXICON_IDENTIFIER and LEXICON_APP_PASSWORD in the environment (see .env.example) — none of the three is hardcoded in the script or committed anywhere in this repository, which is public. See lexicons/space.burrito.morelike.playlist.json and the README's Roadmap entry on atproto sign-in for what gets published and where the record type itself is spelled.

Measurement harness #

src/harness.ts scores the fanout/pickArtists/selectTracks pipeline against reference.txt outside the browser, so ranking or weight changes can be checked against a real number instead of by ear. It caches every HTTP response it makes under /tmp/morelike-harness-cache, keyed by request URL, so a sweep over weights/hopCount/hopDecay/ARTIST_POOL pays for one round of fetches instead of one round per config — delete that directory to force a refetch when the upstream Deezer/ListenBrainz/Last.fm data should be considered stale. See the file header for the full list of modes and env vars.

npx esbuild src/harness.ts --bundle --platform=node --format=esm \
  --packages=external --outfile=/tmp/h.mjs
MODE=confirm SEEDS='Four Tet' node /tmp/h.mjs

Baseline to beat, at the committed defaults (ARTIST_POOL=90, mean over 3 shuffles):

Measured findings from sweeping the harness's knobs:

• Raising ARTIST_POOL above 90 lowers both precision and recall. Reference artists are roughly three times denser in the head of the fused ranking (ranks 1-90, ~28%) than in the tail (ranks 90-400, ~9%), and recommender.ts's pickArtists() samples by fraction of list length, so a larger pool pushes the same ~40-artist draw proportionally deeper into the thinner tail. Measured going from pool 90 to 400: The Field recall 13.5% -> 9.4%, Fela Kuti recall 14.1% -> 10.7%, at roughly 700 HTTP requests per generate instead of 150.
• Raising second-hop hopCount from 15 to 30 gains about 0.4pp of average precision but costs recall (Fela Kuti 17.1% -> 14.5%), doubles the Last.fm second-hop queries per generate, and the seeds disagree about whether it helps. hopCount=15, hopDecay=0.15 is the shipped setting.