diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 02ed393..0fc7eb4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,200 +1,6 @@ # Contributing to Malachite -Thanks for wanting to help. This document covers the layout of the project, how to get a dev environment running, and what to keep in mind when opening a PR. +**This repository has been archived.** Development continues in [`ewanc26/pkgs`](https://github.com/ewanc26/pkgs) — please open issues and PRs there. -## Table of Contents - -- [Project layout](#project-layout) -- [Prerequisites](#prerequisites) -- [Getting started](#getting-started) -- [Running the CLI](#running-the-cli) -- [Running the web app](#running-the-web-app) -- [Running tests](#running-tests) -- [Code architecture](#code-architecture) -- [Making changes](#making-changes) -- [Opening a pull request](#opening-a-pull-request) -- [Publishing `@ewanc26/tid`](#publishing-ewanc26tid) - ---- - -## Project layout - -This is a pnpm monorepo with three separate projects: - -``` -malachite/ -├── src/ # CLI — TypeScript, compiled to dist/ by tsc -│ ├── core/ # Environment-agnostic logic (shared with web) -│ ├── lib/ # CLI wrappers around core (Node.js-specific) -│ └── utils/ # Legacy CLI utilities (gradually migrating to core/) -├── web/ # SvelteKit web app -│ └── src/lib/ -│ ├── core/ # Thin re-exports of src/core/ via the $core alias -│ └── ... # Svelte components, routes, config -├── packages/ -│ └── tid/ # @ewanc26/tid — standalone npm package -└── lexicons/ # fm.teal.alpha lexicon definitions -``` - -The key architectural rule is that **`src/core/` is the single source of truth** for all non-UI logic. The CLI wrappers in `src/lib/` adapt it for terminal use (spinners, file I/O, credentials). The web app re-exports it via `$core` path alias — there should be no duplicated logic between the two surfaces. - ---- - -## Prerequisites - -- **Node.js 20+** (required for Web Crypto API via `globalThis.crypto`) -- **pnpm 9+** — install with `npm i -g pnpm` if you don't have it - ---- - -## Getting started - -```sh -# Clone and install everything in one shot — pnpm workspaces handles all three projects -git clone https://github.com/ewanc26/malachite -cd malachite -pnpm install -``` - ---- - -## Running the CLI - -```sh -# Build -pnpm build - -# Dry run against a real Last.fm export -node dist/index.js -i my-export.csv --dry-run - -# Interactive mode -node dist/index.js - -# Rebuild and run in one step -pnpm dev -``` - -For development work it's worth using `--dev` mode, which enables verbose logging and caps batch sizes to 20 records so you're not waiting through thousands of API calls: - -```sh -node dist/index.js -i my-export.csv --dev --dry-run -``` - ---- - -## Running the web app - -```sh -cd web -pnpm dev # starts at http://127.0.0.1:5173 -``` - -The dev server **must** run on `127.0.0.1:5173` exactly — the ATProto OAuth loopback `redirect_uri` is pinned to that origin per RFC 8252. Don't change the host or port without also updating `web/src/lib/core/oauth.ts`. - -For changes to shared `src/core/` files, the web app picks them up immediately via the `$core` alias — no separate build step needed. - ---- - -## Running tests - -Tests use Node's built-in test runner — no Jest, no Vitest. - -```sh -# Build first, then run all tests -pnpm test - -# Run just the TID tests -pnpm test:tid - -# Watch mode -pnpm test:watch -``` - -Tests live in `src/tests/`. If you're adding a new file to `src/core/`, add a corresponding test file there. The existing tests are a good reference for the style — `node:test` + `node:assert`, no extra dependencies. - ---- - -## Code architecture - -### The `src/core/` contract - -Any code that lives in `src/core/` must be: - -- **Zero Node.js dependencies** — no `fs`, `path`, `crypto` module, etc. Web Crypto (`globalThis.crypto`) is fine because Node 20+ and all modern browsers support it. -- **Callback-based for progress/logging** — functions accept optional `onProgress` callbacks rather than calling `console.log` directly. -- **Tested** — new files should have coverage in `src/tests/`. - -If you need Node.js-specific behaviour (file I/O, terminal spinners, credential storage), put it in `src/lib/` as a thin wrapper that calls into `src/core/`. - -### Adding something to the web - -The web's `web/src/lib/core/` files are almost all one-liners: - -```ts -export * from '$core/your-new-file.js'; -``` - -The only exceptions are `csv.ts` and `spotify.ts`, which add browser `File` API loaders on top of the core parsers, and `oauth.ts` / `import.ts`, which are web-only. Follow that same pattern for anything new. - -### Rate limiting - -The rate limiter in `src/core/rate-limiter.ts` learns quota from response headers on the first successful batch and then gates all subsequent batches. If you're changing publish behaviour, keep the 15% headroom buffer intact — exceeding the PDS daily limit affects every user on a shared instance, not just the one importing. - -### TIDs - -Record keys are generated from `playedTime` using the TID clock in `src/core/tid.ts`. The clock is monotonic — even if records arrive out of order, every call produces a strictly increasing TID within the same process/page. Don't make the generation async (it isn't) and don't reset `lastUs` outside of tests. - ---- - -## Making changes - -### Bugfixes - -Open a PR against `main` with a description of what was broken and how you've verified the fix. A failing test that now passes is ideal. - -### New features - -Open an issue first if the change is significant — it's worth a quick discussion before writing a lot of code. For smaller additions (a new CLI flag, a missing field in the data mapping, an extra export from `@ewanc26/tid`) just open the PR directly. - -### Changing shared core logic - -Changes to `src/core/` affect all three surfaces (CLI, web, `@ewanc26/tid` if relevant). Run both `pnpm test` and `cd web && pnpm check` before submitting to make sure nothing is broken on either side. - -### Style - -- TypeScript strict mode is on — no `any` without a comment explaining why. -- No new runtime dependencies in `src/core/` or `packages/tid/`. -- Prefer named exports over default exports. -- Keep comments on the *why*, not the *what*. - ---- - -## Opening a pull request - -1. Fork the repo and create a branch from `main`. -2. Make your changes, including tests where applicable. -3. Verify `pnpm test` passes and `pnpm run type-check` is clean. -4. If you touched the web app, verify `cd web && pnpm check` is clean too. -5. Open a PR with a clear description of what changed and why. - -There's no formal CLA or contributor agreement — AGPL-3.0 covers contributions automatically. - ---- - -## Publishing `@ewanc26/tid` - -> **Note:** `@ewanc26/tid` is now canonically maintained in the [`@ewanc26/pkgs`](https://github.com/ewanc26/pkgs) monorepo. The copy in `packages/tid/` here is kept for historical context. All version bumps, releases, and npm publishes should happen from there. - -To cut a new release, work from the pkgs monorepo: - -```sh -cd /path/to/pkgs -git subtree pull --prefix=packages/tid malachite malachite-split - -cd packages/tid -# Bump the version in package.json, then: -pnpm build -npm publish --access public --otp= -``` - -The package has no runtime dependencies and must stay that way. If a change to `src/core/tid.ts` in this repo affects the public API of the package, update `packages/tid/src/index.ts` in `pkgs` to match and bump the version accordingly (semver — patch for fixes, minor for new exports, major for breaking changes). +- CLI: [`packages/malachite/`](https://github.com/ewanc26/pkgs/tree/main/packages/malachite) +- Web: [`packages/malachite-web/`](https://github.com/ewanc26/pkgs/tree/main/packages/malachite-web) diff --git a/README.md b/README.md index cf588f9..cad96c2 100644 --- a/README.md +++ b/README.md @@ -1,840 +1,14 @@ -# Malachite +# Malachite — Archived -Import your Last.fm and Spotify listening history to the AT Protocol network using the `fm.teal.alpha.feed.play` lexicon. +**This repository has been consolidated into [`ewanc26/pkgs`](https://github.com/ewanc26/pkgs).** -**Repository:** [malachite](https://github.com/ewanc26/malachite) -[Also available on Tangled](https://tangled.org/did:plc:ofrbh253gwicbkc5nktqepol/atproto-lastfm-importer) +The code lives on — it's just no longer maintained here as a standalone repo. -## Table of Contents +| What | Where | +|---|---| +| CLI importer | [`packages/malachite/`](https://github.com/ewanc26/pkgs/tree/main/packages/malachite) | +| Web frontend | [`packages/malachite-web/`](https://github.com/ewanc26/pkgs/tree/main/packages/malachite-web) | +| Issues & PRs | [ewanc26/pkgs](https://github.com/ewanc26/pkgs/issues) | -- [⚠️ Important: Rate Limits](#️-important-rate-limits) - - [How Dynamic Batch Sizing Works](#how-dynamic-batch-sizing-works) -- [What's with the name?](#whats-with-the-name) -- [Web App](#web-app) -- [Quick Start](#quick-start) - - [Interactive Mode (Recommended for First-Time Users)](#interactive-mode-recommended-for-first-time-users) - - [Command Line Mode](#command-line-mode) -- [Features](#features) - - [Import Capabilities](#import-capabilities) - - [Performance & Safety](#performance--safety) - - [User Experience](#user-experience) - - [Technical Features](#technical-features) -- [Usage Examples](#usage-examples) - - [Combined Import (Last.fm + Spotify)](#combined-import-lastfm--spotify) - - [Re-Sync Mode](#re-sync-mode) - - [Remove Duplicates](#remove-duplicates) - - [Import from Spotify](#import-from-spotify) - - [Import from Last.fm](#import-from-lastfm) - - [Advanced Options](#advanced-options) -- [Command Line Options](#command-line-options) - - [Required Options](#required-options) - - [Import Mode](#import-mode) - - [Additional Options](#additional-options) - - [PDS Override](#pds-override) - - [Legacy Flags (Backwards Compatible)](#legacy-flags-backwards-compatible) -- [Getting Your Data](#getting-your-data) - - [Last.fm Export](#lastfm-export) - - [Spotify Export](#spotify-export) -- [Data Format](#data-format) - - [Required Fields](#required-fields) - - [Optional Fields](#optional-fields) - - [Example Records](#example-records) -- [How It Works](#how-it-works) - - [Processing Flow](#processing-flow) - - [Automatic Duplicate Prevention](#automatic-duplicate-prevention) - - [Rate Limiting Algorithm](#rate-limiting-algorithm) - - [Multi-Day Imports](#multi-day-imports) -- [Logging and Output](#logging-and-output) - - [Verbosity Levels](#verbosity-levels) -- [Error Handling](#error-handling) -- [Troubleshooting](#troubleshooting) - - [Authentication Issues](#authentication-issues) - - [Performance Issues](#performance-issues) - - [Connection Issues](#connection-issues) - - [Output Control](#output-control) -- [Development](#development) -- [File Storage](#file-storage) - - [Credential Storage](#credential-storage) -- [Project Structure](#project-structure) -- [Technical Details](#technical-details) - - [Authentication](#authentication) - - [Batch Publishing](#batch-publishing) - - [Data Mapping](#data-mapping) - - [Lexicon Reference](#lexicon-reference) -- [Contributing](#contributing) -- [License](#license) -- [Credits](#credits) - -## ⚠️ Important: Rate Limits - -**CRITICAL**: Bluesky's AppView has rate limits on PDS instances. Exceeding 10K records per day can rate limit your **ENTIRE PDS**, affecting all users on your instance. - -This importer automatically protects your PDS by: - -- **Dynamic batch sizing** (1-200 records) that adapts to available quota in real-time -- **15% headroom buffer** prevents quota exhaustion before hitting the limit -- Limiting imports to **7,500 records per day** (with 75% safety margin) -- Calculating optimal batch sizes and delays -- **Graceful degradation** - scales down smoothly as quota depletes -- **Instant recovery** - immediately returns to maximum speed after quota resets -- Pausing 24 hours between days for large imports -- Providing clear progress tracking and time estimates -- Persisting state across restarts for safe resume - -### How Dynamic Batch Sizing Works - -Malachite continuously monitors your rate limit quota and automatically adjusts batch size: - -``` -Fresh Quota (5000 points) → Batch Size: 200 records (maximum speed) -Half Depleted (2500 points) → Batch Size: 200 records (still optimal) -Approaching Limit (1200) → Batch Size: 150 records (scaling down) -Near Headroom (900) → Batch Size: 50 records (conservative) -Below Headroom (700) → Batch Size: 1 record (minimal progress) -[Quota Resets] → Batch Size: 200 records (instant recovery) -``` - -**Benefits:** - -- ✅ **2x faster** when quota is fresh (200 vs 100 records/batch) -- ✅ **Never hits rate limits** - proactive scaling with 15% buffer -- ✅ **Always makes progress** - even with minimal quota (batch size 1) -- ✅ **Automatic recovery** - no manual intervention needed -- ✅ **Transparent** - logs all batch size changes with reasons - -For more details, see the [Bluesky Rate Limits Documentation](https://docs.bsky.app/blog/rate-limits-pds-v3). - -## What's with the name? - -It used to be called `atproto-lastfm-importer` — generic as fuck. That name told you what it did and nothing about why it mattered, and it sounded like a disposable weekend script. So I renamed it. - -At the moment, the repository is still called `atproto-lastfm-importer` on Tangled, but the GitHub link has been updated to `malachite`. I do not know if this can be resolved. - -**Malachite** is a greenish-blue copper mineral associated with preservation and transformation. That's exactly what this tool does: it preserves your scrobbles and transforms them into proper `fm.teal.alpha.feed.play` records on the AT Protocol. The colour match isn't an accident — malachite sits squarely in the teal/green range, a deliberate nod to the `teal` lexicon it publishes to. - -## Web App - -Malachite also ships a browser-based web app (`web/`) built with SvelteKit. It supports all five import modes and signs in via ATProto OAuth — no app password required. - -**Running the web app in development:** - -```bash -cd web -pnpm install -pnpm dev # starts at http://127.0.0.1:5173 -``` - -> **Note:** The dev server must run on `127.0.0.1:5173` exactly. This is enforced in `vite.config.ts` because the OAuth loopback `redirect_uri` is pinned to that origin (RFC 8252 §7.3). Do not change the host or port without updating the OAuth client metadata. - -The web app fetches existing records using the same CAR-export path as the CLI (`com.atproto.sync.getRepo`) so it costs zero AppView write-quota points to check for duplicates. - -## Quick Start - -**Note:** You must build the project first, then run with arguments. - -### Interactive Mode (Recommended for First-Time Users) - -Just run without any arguments and Malachite will guide you through the process: - -```bash -# Install dependencies and build -pnpm install -pnpm build - -# Run in interactive mode -pnpm start -``` - -The interactive mode will: - -- Present a menu of available actions -- Prompt for all required information (handle, password, files) -- Ask for optional settings (dry run, verbose logging, etc.) -- Provide helpful descriptions for each option - -### Command Line Mode - -For automation or if you prefer command-line arguments: - -```bash -# Show help -pnpm start --help - -# Run with command line arguments -pnpm start -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y - -# Alternative: run directly with node -node dist/index.js -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y -``` - -## Features - -### Import Capabilities - -- ✅ **Last.fm Import**: Full support for Last.fm CSV exports with MusicBrainz IDs -- ✅ **Spotify Import**: Import Extended Streaming History JSON files -- ✅ **Combined Import**: Merge Last.fm and Spotify exports with intelligent deduplication -- ✅ **Re-Sync Mode**: Import only new scrobbles without creating duplicates -- ✅ **Duplicate Removal**: Clean up accidentally imported duplicate records - -### Performance & Safety - -- ✅ **Automatic Duplicate Prevention**: Fetches your existing Teal records via CAR export and skips anything already imported -- ✅ **Input Deduplication**: Removes duplicate entries within the source file before submission -- ✅ **Dynamic Batch Sizing**: Automatically adjusts batch size (1-200 records) based on available rate limit quota -- ✅ **Batch Operations**: Uses `com.atproto.repo.applyWrites` for efficient batch publishing (up to 200 records per call) -- ✅ **Zero-cost sync check**: Existing record fetching uses `com.atproto.sync.getRepo` (CAR export) — a separate, far more generous rate-limit envelope that costs zero AppView write-quota points -- ✅ **Intelligent Rate Limiting**: Real-time quota monitoring with 15% headroom buffer prevents rate limit exhaustion -- ✅ **Adaptive Recovery**: Automatically scales back to maximum speed after quota resets -- ✅ **Multi-Day Imports**: Large imports automatically span multiple days with 24-hour pauses -- ✅ **Resume Support**: Safe to stop (Ctrl+C) and restart - continues from where it left off -- ✅ **Graceful Cancellation**: Press Ctrl+C to stop after the current batch completes - -### User Experience - -- ✅ **Structured Logging**: Color-coded output with debug/verbose modes -- ✅ **Progress Tracking**: Real-time progress with time estimates -- ✅ **Dry Run Mode**: Preview records without publishing -- ✅ **Interactive Mode**: Simple prompts guide you through the process -- ✅ **Command Line Mode**: Full automation support for scripting -- ✅ **Web App**: Browser-based UI with ATProto OAuth sign-in - -### Technical Features - -- ✅ **TID-based Record Keys**: Timestamp-based identifiers for chronological ordering -- ✅ **Identity Resolution**: Resolves ATProto handles/DIDs using Slingshot -- ✅ **PDS Auto-Discovery**: Automatically connects to your personal PDS -- ✅ **MusicBrainz Support**: Preserves MusicBrainz IDs when available (Last.fm) -- ✅ **Chronological Ordering**: Processes oldest first (or newest with `-r` flag) -- ✅ **Error Handling**: Continues on errors with detailed reporting - -## Usage Examples - -### Combined Import (Last.fm + Spotify) - -Merge your Last.fm and Spotify listening history into a single, deduplicated import: - -```bash -# Preview the merged import -pnpm start -i lastfm.csv --spotify-input spotify-export/ -m combined --dry-run - -# Perform the combined import -pnpm start -i lastfm.csv --spotify-input spotify-export/ -m combined -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y -``` - -**What combined mode does:** - -1. Parses both Last.fm CSV and Spotify JSON exports -2. Normalizes track names and artist names for comparison -3. Identifies duplicate plays (same track within 5 minutes) -4. Chooses the best version of each play (prefers Last.fm with MusicBrainz IDs) -5. Merges into a single chronological timeline -6. Shows detailed statistics about the merge - -### Re-Sync Mode - -Sync your Last.fm export with Teal without creating duplicates: - -```bash -# Preview what will be synced -pnpm start -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -m sync --dry-run - -# Perform the sync -pnpm start -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -m sync -y -``` - -**Perfect for:** - -- Re-running imports with updated Last.fm exports -- Recovering from interrupted imports -- Adding recent scrobbles without duplicating old ones - -**Note:** Sync mode requires authentication even in dry-run mode to fetch existing records. - -### Remove Duplicates - -Clean up accidentally imported duplicate records: - -```bash -# Preview duplicates (dry run) -pnpm start -m deduplicate -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx --dry-run - -# Remove duplicates (keeps first occurrence) -pnpm start -m deduplicate -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -``` - -### Import from Spotify - -```bash -# Import single Spotify JSON file -pnpm start -i Streaming_History_Audio_2021-2023_0.json -m spotify -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y - -# Import directory with multiple Spotify files (recommended) -pnpm start -i '/path/to/Spotify Extended Streaming History' -m spotify -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y -``` - -### Import from Last.fm - -```bash -# Standard Last.fm import -pnpm start -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y - -# Preview without publishing -pnpm start -i lastfm.csv --dry-run - -# Process newest tracks first -pnpm start -i lastfm.csv -h alice.bsky.social -r -y - -# Verbose debug output -pnpm start -i lastfm.csv --dry-run -v - -# Quiet mode (only warnings and errors) -pnpm start -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -q -y -``` - -### Advanced Options - -```bash -# Development mode (verbose + file logging + smaller batches for debugging) -pnpm start -i lastfm.csv --dev --dry-run - -# Custom batch settings (advanced users only) -pnpm start -i lastfm.csv -h alice.bsky.social -b 20 -d 3000 - -# Full automation with all flags -pnpm start -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y -q -``` - -## Command Line Options - -**Note:** When importing data (not in deduplicate mode), you must provide `--input`, `--handle`, and `--password`. The `--yes` flag skips confirmation prompts for automation. - -### Required Options - -| Option | Short | Description | Example | -| ------------------- | ----- | -------------------------------------------------- | ------------------------ | -| `--input ` | `-i` | Path to Last.fm CSV or Spotify JSON file/directory | `-i lastfm.csv` | -| `--handle ` | `-h` | ATProto handle or DID | `-h alice.bsky.social` | -| `--password ` | `-p` | ATProto app password | `-p xxxx-xxxx-xxxx-xxxx` | - -### Import Mode - -| Option | Short | Description | Default | -| --------------- | ----- | ----------- | -------- | -| `--mode ` | `-m` | Import mode | `lastfm` | - -**Available modes:** - -- `lastfm` - Import Last.fm export only -- `spotify` - Import Spotify export only -- `combined` - Merge Last.fm + Spotify exports -- `sync` - Skip existing records (sync mode) -- `deduplicate` - Remove duplicate records - -### Additional Options - -| Option | Short | Description | Default | -| ------------------------ | ----- | ----------------------------------------------------------- | --------------- | -| `--spotify-input ` | | Path to Spotify export (for combined mode) | - | -| `--reverse` | `-r` | Process newest first | `false` | -| `--yes` | `-y` | Skip confirmation prompts | `false` | -| `--dry-run` | | Preview without importing | `false` | -| `--verbose` | `-v` | Enable debug logging | `false` | -| `--quiet` | `-q` | Suppress non-essential output | `false` | -| `--dev` | | Development mode (verbose + file logging + smaller batches) | `false` | -| `--batch-size ` | `-b` | Initial batch size (1-200, dynamically adjusted) | Auto-calculated | -| `--batch-delay ` | `-d` | Delay between batches in ms | `500` (min) | -| `--help` | | Show help message | - | - -### PDS Override - -If you already know the base URL of your Personal Data Server (PDS) you can bypass the Slingshot identity resolver and provide it directly with the `--pds` flag. This is useful for private instances, testing, or when the resolver is unreliable. - -| Option | Description | -| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `--pds ` | PDS base URL to use for authentication and API calls (e.g. `https://pds.example.com`). When provided, Malachite will skip Slingshot lookup and use this URL directly. | - -Notes: - -- The `--pds` flag overrides the configured Slingshot resolver for identity lookup. If `--pds` is given, Malachite will attempt to authenticate directly against the supplied PDS using your handle/DID and app password. -- Use the full base URL (including scheme), e.g. `https://pds.example.com`. -- If authentication fails when using `--pds`, try removing the flag so Malachite can resolve your PDS automatically via Slingshot. - -### Legacy Flags (Backwards Compatible) - -These old flags still work but are deprecated: - -- `--file` → Use `--input` -- `--identifier` → Use `--handle` -- `--spotify-file` → Use `--spotify-input` -- `--reverse-chronological` → Use `--reverse` -- `--spotify` → Use `--mode spotify` -- `--combined` → Use `--mode combined` -- `--sync` → Use `--mode sync` -- `--remove-duplicates` → Use `--mode deduplicate` - -## Getting Your Data - -### Last.fm Export - -1. Visit [Last.fm Export Tool](https://lastfm.ghan.nl/export/) -2. Request your data export in CSV format -3. Download the CSV file when ready -4. Use the CSV file path with this importer - -### Spotify Export - -1. Go to [Spotify Privacy Settings](https://www.spotify.com/account/privacy/) -2. Scroll to "Download your data" and request your data -3. Select "Extended streaming history" (can take up to 30 days) -4. When ready, download and extract the ZIP file -5. Use either: - - A single JSON file: `Streaming_History_Audio_2021-2023_0.json` - - The entire extracted directory (recommended) - -**Note:** The importer automatically: - -- Reads all `Streaming_History_Audio_*.json` files in a directory -- Filters out podcasts, audiobooks, and non-music content -- Combines all music tracks into a single import - -## Data Format - -Each scrobble becomes an `fm.teal.alpha.feed.play` record with: - -### Required Fields - -- **trackName**: The name of the track -- **artists**: Array of artist objects (requires `artistName`, optional `artistMbId` for Last.fm) -- **playedTime**: ISO 8601 timestamp of when you listened -- **submissionClientAgent**: Identifies this importer (`malachite/v0.10.0` for CLI, `malachite/v0.3.0 (web)` for the web app) -- **musicServiceBaseDomain**: Set to `last.fm` or `spotify.com` - -### Optional Fields - -- **releaseName**: Album/release name -- **releaseMbId**: MusicBrainz release ID (Last.fm only) -- **recordingMbId**: MusicBrainz recording/track ID (Last.fm only) -- **originUrl**: Link to the track on Last.fm or Spotify - -### Example Records - -**Last.fm Record:** - -```json -{ - "$type": "fm.teal.alpha.feed.play", - "trackName": "Paint My Masterpiece", - "artists": [ - { - "artistName": "Cjbeards", - "artistMbId": "c8d4f4bf-1b82-4d4d-9d73-05909faaff89" - } - ], - "releaseName": "Masquerade", - "releaseMbId": "fdb2397b-78d5-4019-8fad-656d286e4d33", - "recordingMbId": "3a390ad3-fe56-45f2-a073-bebc45d6bde1", - "playedTime": "2025-11-13T23:49:36Z", - "originUrl": "https://www.last.fm/music/Cjbeards/_/Paint+My+Masterpiece", - "submissionClientAgent": "malachite/v0.10.0", - "musicServiceBaseDomain": "last.fm" -} -``` - -**Spotify Record:** - -```json -{ - "$type": "fm.teal.alpha.feed.play", - "trackName": "Don't Give Up", - "artists": [ - { - "artistName": "Chicane" - } - ], - "releaseName": "Twenty", - "playedTime": "2021-09-09T10:34:08Z", - "originUrl": "https://open.spotify.com/track/3gZqDJkMZipOYCRjlHWgOV", - "submissionClientAgent": "malachite/v0.10.0", - "musicServiceBaseDomain": "spotify.com" -} -``` - -## How It Works - -### Processing Flow - -1. **Parses input file(s)**: - - Last.fm: CSV using `csv-parse` library - - Spotify: JSON files (single or multiple in directory) -2. **Filters data**: - - Spotify: Automatically removes podcasts, audiobooks, and non-music content -3. **Converts to schema**: Maps to `fm.teal.alpha.feed.play` format -4. **Deduplicates input**: Removes duplicate entries from the source data (keeps first occurrence) -5. **Checks Teal**: Downloads the entire repo as a CAR file (`com.atproto.sync.getRepo`) and skips any records already imported — costs zero AppView write-quota points -6. **Sorts records**: Chronologically (oldest first) or reverse with `-r` flag -7. **Generates TID-based keys**: From `playedTime` for chronological ordering -8. **Validates fields**: Ensures required fields are present -9. **Publishes in batches**: Uses `com.atproto.repo.applyWrites` (up to 200 records per call) - -### Automatic Duplicate Prevention - -The importer has **two layers of duplicate prevention** to ensure you never import the same record twice: - -#### Step 1: Input File Deduplication - -Removes duplicates within your source file(s): - -**How duplicates are identified:** - -- Same track name (case-insensitive) -- Same artist name (case-insensitive) -- Same timestamp (exact match) - -**What happens:** - -- First occurrence is kept -- Subsequent duplicates are removed -- Shows message: "No duplicates found in input data" or "Removed X duplicate(s)" - -#### Step 2: Teal Comparison via CAR Export - -**Automatically checks your existing Teal records** by downloading your entire repo as a CARv1 file: - -- One HTTP request fetches the whole repo (`com.atproto.sync.getRepo`) -- The CAR file is parsed locally in memory — no AppView quota consumed -- Compares every record against your input and skips anything already imported -- Shows: "Skipped X already-imported record(s)" - -**This means:** - -- ✅ Safe to re-run imports with updated exports -- ✅ Won't create duplicates if you run the import twice -- ✅ Zero AppView write-quota cost for the sync check -- ✅ Works automatically - no special mode needed - -**Note:** - -- Credentials are required even for `--dry-run` to fetch the CAR export -- **Sync mode** (`-m sync`): Shows detailed statistics about what's being skipped -- **Deduplicate mode** (`-m deduplicate`): Removes duplicates from already-imported Teal records (cleanup tool) - -### Rate Limiting Algorithm - -1. Calculates safe daily limit (75% of 10K = 7,500 records/day by default) -2. Determines how many days needed for your import -3. **Monitors rate limit quota in real-time** before each batch -4. **Dynamically adjusts batch size** (1-200 records) based on available points -5. **Preserves 15% headroom buffer** to prevent exhaustion -6. **Automatically waits** when quota is exhausted (with countdown timer) -7. **Instantly scales back up** to maximum batch size after quota resets -8. Enforces minimum delay between batches -9. Shows clear schedule and real-time batch size adjustments - -### Multi-Day Imports - -For imports exceeding the daily limit, the importer automatically: - -1. **Calculates a schedule**: Splits your import across multiple days -2. **Shows the plan**: Displays which records will be imported each day -3. **Processes Day 1**: Imports the first batch of records -4. **Pauses 24 hours**: Waits a full day before continuing -5. **Repeats**: Continues until all records are imported - -**Important notes:** - -- You can safely stop (Ctrl+C) and restart -- Progress is preserved - continues where it left off -- Each day's progress is clearly displayed -- Time estimates account for multi-day duration - -## Logging and Output - -The importer uses color-coded output for clarity: - -- **Green (✓)**: Success messages -- **Cyan (→)**: Progress updates -- **Yellow (⚠️)**: Warnings -- **Red (✗)**: Errors -- **Bold Red (🛑)**: Fatal errors - -### Verbosity Levels - -**Default Mode**: Standard operational messages - -```bash -pnpm start -i lastfm.csv -h alice.bsky.social -p pass -``` - -**Verbose Mode** (`-v`): Detailed debug information including batch timing and API calls - -```bash -pnpm start -i lastfm.csv -h alice.bsky.social -p pass -v -``` - -**Quiet Mode** (`-q`): Only warnings and errors - -```bash -pnpm start -i lastfm.csv -h alice.bsky.social -p pass -q -``` - -**Development Mode** (`--dev`): Verbose logging + file logging to `~/.malachite/logs/` + smaller batch sizes - -```bash -pnpm start -i lastfm.csv --dev --dry-run -``` - -Development mode is perfect for: - -- Debugging import issues with detailed logs -- Testing changes with smaller batches (20 records max) -- Preserving logs for later analysis -- Troubleshooting problems with support - -## Error Handling - -The importer is designed to be resilient: - -- **Network errors**: Failed records are logged but don't stop the import -- **Invalid data**: Skipped with error messages -- **Authentication issues**: Clear error messages with suggested fixes -- **Rate limit hits**: Automatic adjustment and retry logic -- **Ctrl+C handling**: Gracefully stops after current batch - -## Troubleshooting - -### Authentication Issues - -**"Handle not found"** - -- Verify your ATProto handle is correct (e.g., `alice.bsky.social`) -- Ensure you're using a valid DID or handle - -**"Invalid credentials"** - -- Use an **app password**, not your main account password -- Generate app passwords in your account settings - -### Performance Issues - -**"Rate limit exceeded"** - -- The importer should prevent this automatically -- If you see this, wait 24 hours before retrying -- Consider reducing batch size with `-b` flag - -**Import seems stuck** - -- Check progress messages - large imports take time -- Multi-day imports pause for 24 hours between days -- You can safely stop (Ctrl+C) and resume later -- Use `--verbose` flag to see detailed progress - -### Connection Issues - -**"Connection refused"** - -- Check your internet connection -- Verify your PDS is accessible -- Some PDSs may have firewall rules - -### Output Control - -**Too much output** - -- Use `--quiet` flag to suppress non-essential messages -- Only warnings and errors will be shown - -**Need more details** - -- Use `--verbose` flag to see debug-level information -- Shows batch timing, API calls, and detailed progress - -## Development - -```bash -# Type checking -pnpm run type-check - -# Build -pnpm run build - -# Development mode (rebuild + run) -pnpm run dev - -# Run tests -pnpm run test - -# Clean build artifacts -pnpm run clean -``` - -## File Storage - -Malachite stores all its data in `~/.malachite/`: - -``` -~/.malachite/ -├── cache/ # Cached Teal records (24-hour TTL) -├── state/ # Import state for resume functionality -├── logs/ # Import logs (when file logging is enabled) -└── credentials.json # Encrypted credentials (optional, machine-specific) -``` - -This keeps your project directory clean and follows standard Unix conventions. - -### Credential Storage - -Malachite automatically saves your ATProto credentials after a successful login so you don't need to re-enter them on the next run: - -**Security Features:** - -- ✅ **AES-256-GCM encryption** - Military-grade encryption -- ✅ **Machine-specific** - Credentials are bound to your computer and can't be transferred -- ✅ **Secure key derivation** - Uses PBKDF2 with 100,000 iterations -- ✅ **File permissions** - Credentials file is readable only by you (Unix) - -**How It Works:** - -1. Credentials are encrypted using a key derived from your hostname + username and saved to `~/.malachite/credentials.json` after every successful login -2. On the next run, Malachite loads saved credentials automatically -3. In interactive mode, you'll be prompted whether to use the saved credentials or enter new ones - -**Managing Credentials:** - -```bash -# Clear saved credentials -pnpm start --clear-credentials - -# Or through interactive mode (option 7) -pnpm start -``` - -**Important Notes:** - -- Credentials are machine-specific and won't work if you copy the file to another computer -- This is a convenience feature - you can always enter credentials manually -- If you change your password, clear and re-save credentials - -## Project Structure - -``` -malachite/ -├── src/ -│ ├── lib/ -│ │ ├── auth.ts # Authentication & identity resolution -│ │ ├── cli.ts # Command line interface & argument parsing -│ │ ├── csv.ts # CSV parsing & record conversion -│ │ ├── publisher.ts # Batch publishing with rate limiting -│ │ ├── spotify.ts # Spotify JSON parsing -│ │ ├── merge.ts # Combined import deduplication -│ │ └── sync.ts # Re-sync mode & duplicate detection -│ ├── utils/ -│ │ ├── car-fetch.ts # CAR export fetcher (com.atproto.sync.getRepo) -│ │ ├── logger.ts # Structured logging system -│ │ ├── helpers.ts # Utility functions (timing, formatting) -│ │ ├── input.ts # User input handling (prompts, passwords) -│ │ ├── rate-limiter.ts # Rate limiting with server-learned quota -│ │ ├── killswitch.ts # Graceful shutdown handling -│ │ ├── tid.ts # TID generation from timestamps -│ │ └── ui.ts # UI elements (spinners, progress bars) -│ ├── config.ts # Configuration constants & version -│ └── types.ts # TypeScript type definitions -├── web/ # SvelteKit web app -│ └── src/lib/ -│ ├── core/ # Browser-safe equivalents of src/lib & src/utils -│ │ ├── auth.ts # Password-based ATProto login -│ │ ├── car-fetch.ts# CAR export fetcher (browser-safe) -│ │ ├── csv.ts # CSV parser (no csv-parse dep) -│ │ ├── import.ts # Import orchestration -│ │ ├── merge.ts # Combined import deduplication -│ │ ├── oauth.ts # ATProto OAuth client -│ │ ├── publisher.ts# Batch publisher with progress callbacks -│ │ ├── rate-limiter.ts # In-memory rate limiter -│ │ ├── spotify.ts # Spotify JSON parser -│ │ ├── sync.ts # CAR-based sync & dedup -│ │ └── tid.ts # TID generation (Web Crypto API) -│ ├── config.ts # Shared constants (version injected by Vite) -│ ├── modes.ts # Import mode definitions -│ └── types.ts # TypeScript type definitions -├── lexicons/ # fm.teal.alpha lexicon definitions -│ └── fm.teal.alpha/ -│ └── feed/ -│ └── play.json # Play record schema -├── package.json -├── tsconfig.json -└── README.md -``` - -## Technical Details - -### Authentication - -- Uses Slingshot resolver to discover your PDS from your handle/DID -- Requires an ATProto app password (not your main password) for the CLI -- Web app supports ATProto OAuth — no app password needed -- Automatically configures the agent for your personal PDS - -### Batch Publishing - -- Uses `com.atproto.repo.applyWrites` for efficiency (up to 20x faster than individual calls) -- Batches up to 200 records per API call (PDS maximum) -- **Dynamic batch sizing** (1-200 records) based on real-time rate limit quota -- **Intelligent quota monitoring** with 15% headroom buffer -- **Automatic adjustment** - scales down as quota depletes, scales up after reset -- Enforces minimum delays between batches for rate limit safety - -### CAR Export Sync - -All read paths (duplicate checks, sync, deduplicate) use `com.atproto.sync.getRepo` to download the user's entire repo as a CARv1 file. The CAR is parsed locally using `@ipld/car` and `@ipld/dag-cbor` — no AppView XRPC calls are made for reads, so the sync check costs zero write-quota points. - -### Data Mapping - -**Last.fm:** - -- Direct mapping from CSV columns -- Converts Unix timestamps to ISO 8601 -- Preserves MusicBrainz IDs when present -- Generates URLs from artist/track names -- Wraps artists in array format with optional MBID - -**Spotify:** - -- Extracts data from JSON fields -- Already in ISO 8601 format (`ts` field) -- Generates URLs from `spotify_track_uri` -- Automatically filters non-music content -- Extracts artist and album from metadata fields - -### Lexicon Reference - -This importer follows the official `fm.teal.alpha` lexicon defined in `/lexicons/fm.teal.alpha/feed/play.json`. - -The lexicon defines required and optional field types, string length constraints, array formats, timestamp formatting, and URL validation. - -## Contributing - -Contributions are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for setup instructions, architecture notes, and PR guidelines. - -## License - -AGPL-3.0-only - See LICENCE file for details - -## ☕ Support - -If you found this useful, consider [buying me a ko-fi](https://ko-fi.com/ewancroft)! - -## Credits - -- Uses [@atproto/api](https://www.npmjs.com/package/@atproto/api) for ATProto interactions -- CSV parsing via [csv-parse](https://www.npmjs.com/package/csv-parse) -- Identity resolution via [Slingshot](https://slingshot.danner.cloud) -- Follows the `fm.teal.alpha` lexicon standard -- Colored output via [chalk](https://www.npmjs.com/package/chalk) -- Progress indicators via [ora](https://www.npmjs.com/package/ora) and [cli-progress](https://www.npmjs.com/package/cli-progress) -- Web app built with [SvelteKit](https://kit.svelte.dev) and [Tailwind CSS](https://tailwindcss.com) -- ATProto OAuth via [@atproto/oauth-client-browser](https://www.npmjs.com/package/@atproto/oauth-client-browser) - ---- - -**Note**: This tool is for personal use. Respect the terms of service and rate limits when importing your data. +Git history has been fully preserved in the monorepo via `git filter-repo`. +This repository is now archived and will not receive further updates.