diff --git a/docs/plans/https-directory-listing.md b/docs/plans/https-directory-listing.md new file mode 100644 index 00000000..e2836258 --- /dev/null +++ b/docs/plans/https-directory-listing.md @@ -0,0 +1,246 @@ +# HTTPS directory listing (`https+json`) + +## Context + +Some HTTPS servers expose a directory listing API: appending `?ls` to a directory URL returns a JSON array describing its contents: + +```json +[ + { "name": "my-directory", "type": "directory", "mtime": "2022-10-07T00:53:50Z" }, + { "name": "my_file.tar.gz", "type": "file", "mtime": "2022-09-27T22:44:34Z", "size": 332 } +] +``` + +Rather than bolting this onto the existing HTTPS input (which is designed for individual file URLs), this feature is implemented as a separate `https+json` input with its own URI scheme. Users add a server URL (e.g. `https+json://music.example.com`) and the input automatically discovers all audio files under it — recursively traversing subdirectories. + +The existing HTTPS input is left untouched. + +## Plan + +### 1. Create `src/components/input/https-json/constants.js` + +```js +export const SCHEME = "https+json"; +``` + +### 2. Create `src/components/input/https-json/common.js` + +Follows the same structure as `https/common.js`. Key additions are `buildURI` and `listDirectory`. + +**`parseURI(uriString)`** — validates the `https+json:` scheme and returns components: + +```js +export function parseURI(uriString) { + try { + const url = new URL(uriString); + if (url.protocol !== "https+json:") return undefined; + + return { + // The real HTTPS URL (for fetch calls) + url: "https:" + uriString.slice("https+json:".length), + domain: url.hostname, + host: url.host, + path: url.pathname, + }; + } catch { + return undefined; + } +} +``` + +**`buildURI(host, path)`** — constructs an `https+json://` URI: + +```js +export function buildURI(host, path = "/") { + return `https+json://${host}${path}`; +} +``` + +**`listDirectory(httpsUrl)`** — fetches `url?ls`, parses the JSON listing, and recursively collects audio file URLs. Returns `https+json://` URIs (not plain `https://`): + +```js +export async function listDirectory(httpsUrl) { + let entries; + + try { + const response = await fetch(httpsUrl + "?ls"); + if (!response.ok) return []; + entries = await response.json(); + } catch { + return []; + } + + if (!Array.isArray(entries)) return []; + + const results = await Promise.all( + entries.map(async (entry) => { + if (entry.type === "file") { + if (!isAudioFile(entry.name)) return []; + const fileHttpsUrl = httpsUrl + entry.name; + // Return as https+json:// URI + return [fileHttpsUrl.replace(/^https:/, "https+json:")]; + } else { + return listDirectory(httpsUrl + entry.name + "/"); + } + }), + ); + + return results.flat(1); +} +``` + +Include `groupTracksByHost`, `groupUrisByHost`, `hostsFromTracks` — same implementations as in `https/common.js` but using `parseURI` from this module. Include `consultHostCached` — same implementation, caches per host. + +### 3. Create `src/components/input/https-json/worker.js` + +Five standard actions. Mirrors the S3 worker pattern for `list()`. + +**`consult`** — same shape as HTTPS: scheme-only returns `undetermined`, full URI checks host reachability via `consultHostCached`. + +**`detach`** — same shape as HTTPS: groups by host, removes the target host's group. + +**`groupConsult`** — same shape as HTTPS: groups URIs by host, checks each host once. + +**`list(cachedTracks)`** — the key new logic: + +```js +export async function list(cachedTracks = []) { + // Build a URI → track cache for preserving metadata across relists. + /** @type {Record} */ + const cacheByUri = {}; + cachedTracks.forEach((t) => { cacheByUri[t.uri] = t; }); + + // Group by host; derive the scan root per host. + // If placeholder tracks exist for a host, use their path as the root. + // Otherwise, find the common path prefix of the audio file tracks. + const groups = groupTracksByHost(cachedTracks); + + const promises = Object.values(groups).map(async ({ host, tracks }) => { + const root = scanRoot(tracks); // "/" or a shared directory prefix + const rootHttpsUrl = `https://${host}${root}`; + const now = new Date().toISOString(); + + const uris = await listDirectory(rootHttpsUrl); + + let discovered = uris.map((uri) => { + const cached = cacheByUri[uri]; + /** @type {Track} */ + return { + $type: "sh.diffuse.output.track", + id: cached?.id ?? TID.now(), + createdAt: cached?.createdAt ?? now, + updatedAt: cached?.updatedAt ?? now, + stats: cached?.stats, + tags: cached?.tags, + uri, + }; + }); + + if (!discovered.length) { + discovered = [{ + $type: "sh.diffuse.output.track", + id: TID.now(), + createdAt: now, + updatedAt: now, + kind: "placeholder", + uri: buildURI(host, root), + }]; + } + + return discovered; + }); + + return (await Promise.all(promises)).flat(1); +} +``` + +`scanRoot(tracks)` is a small helper (defined in `worker.js` or `common.js`) that: +- Returns the path of the first placeholder-kind track if one exists +- Otherwise returns the longest common path prefix of all track paths, trimmed to the last `/` + +**`resolve({ uri })`** — strips the scheme prefix and returns the `https://` URL with a one-year expiry (same as HTTPS input): + +```js +export async function resolve({ uri }) { + const parsed = parseURI(uri); + if (!parsed) return undefined; + + const expiresAt = Math.round(Date.now() / 1000) + 60 * 60 * 24 * 365; + return { url: parsed.url, expiresAt }; +} +``` + +### 4. Create `src/components/input/https-json/element.js` + +Minimal — same shape as `https/element.js`. `sources()` returns unique hosts from tracks. + +```js +class HttpsJsonInput extends DiffuseElement { + static NAME = "diffuse/input/https-json"; + static WORKER_URL = "components/input/https-json/worker.js"; + + SCHEME = SCHEME; + + constructor() { + super(); + this.proxy = this.workerProxy(); + this.consult = this.proxy.consult; + this.detach = this.proxy.detach; + this.groupConsult = this.proxy.groupConsult; + this.list = this.proxy.list; + this.resolve = this.proxy.resolve; + } + + sources(tracks) { + return Object.values(hostsFromTracks(tracks)).map((host) => ({ + label: host, + uri: buildURI(host), + })); + } +} + +export const CLASS = HttpsJsonInput; +export const NAME = "di-https-json"; + +customElements.define(NAME, CLASS); +``` + +### 5. Register in `src/components/orchestrator/input/element.js` + +Add the import and the element to the rendered template: + +```js +import "@components/input/https-json/element.js"; +// ... +render({ html }) { + return html` + + + + + + + `; +} +``` + +## Files changed (5) + +| File | Change | Difficulty | +|------|--------|------------| +| `src/components/input/https-json/constants.js` | New — define `SCHEME = "https+json"` | Trivial | +| `src/components/input/https-json/common.js` | New — `parseURI`, `buildURI`, `listDirectory`, grouping helpers, `consultHostCached` | Low | +| `src/components/input/https-json/worker.js` | New — all 5 actions; `list()` does recursive `?ls` fetching | Low-medium | +| `src/components/input/https-json/element.js` | New — `HttpsJsonInput` element, `di-https-json` | Trivial | +| `src/components/orchestrator/input/element.js` | Add import + `` to template | Trivial | + +**Overall difficulty: Low-medium.** All patterns are direct copies or adaptations of the existing HTTPS and S3 inputs. The only genuinely new logic is `listDirectory()` (recursive `?ls` fetch) and `scanRoot()` (derive listing root from cached tracks). + +## Verification + +- Add `https+json://music.example.com` as a source and trigger a library refresh +- Confirm audio files from the server's directory tree appear as tracks +- Confirm cached metadata (tags, stats) is preserved across relists +- Add a server with no audio files — confirm a placeholder track is produced +- Confirm `resolve()` returns a working `https://` URL for each track +- Confirm `di-https` (plain HTTPS) is unaffected