diff --git a/README.md b/README.md index 35a990a..9c42765 100644 --- a/README.md +++ b/README.md @@ -1,532 +1,37 @@ # kobo-shelf -A self-hosted Kobo eReader sync server and web library manager. Written in Rust -with [axum](https://github.com/tokio-rs/axum) and a -[Leptos](https://leptos.dev/) web UI. +A self-hosted Kobo eReader sync server and web library manager, written in Rust +([axum](https://github.com/tokio-rs/axum)) with a [Leptos](https://leptos.dev/) web UI. -kobo-shelf implements the Kobo Store API so a Kobo device can sync books directly -from your own server, with no dependency on Kobo's cloud. Books enter the -catalog two ways: - -- **From a [Calibre](https://calibre-ebook.com/) library** — kobo-shelf reads - `metadata.db` (read-only) and ingests it into its own catalog. -- **By upload** — a browser upload portal and a bundled **Calibre plugin** push - EPUBs straight to the server. Calibre is entirely optional: with no library - configured, the catalog is served from uploads alone. - -Everything else — sync state, reading progress, shelves, per-user sync -preferences, uploaded files, and accounts — lives in kobo-shelf' own SQLite -database. A web UI provides library management, per-book cover and metadata -editing, device setup, account/token management, and login via local -password or OIDC single sign-on. - -Inspired by the Kobo sync integration in -[calibre-web](https://github.com/janeczku/calibre-web). - -## Features - -- **Kobo device sync** — library, covers, reading state, and shelves sync to a - Kobo with no Kobo account (drive a factory-reset device; see below). -- **Two catalog sources** — Calibre ingest and/or direct EPUB uploads, unified - in one catalog that is the single source of truth. -- **Web library manager** — browse the catalog, upload books, edit metadata - (title, authors, series, description), change or search covers, toggle - per-book sync, and trash/restore books. -- **Calibre plugin** — send selected books from Calibre to kobo-shelf; it - pre-checks by book UUID and skips already-synced books without re-uploading. - See [releases](https://git.darksailor.dev/servius/kobo-shelf/releases). -- **Accounts** — local username/password and/or OIDC SSO; optional - self-service registration; per-user auth tokens for devices and the plugin. -- **Optional niceties** — `kepubify` conversion (EPUB→KEPUB for native per-page - progress) and online metadata/cover enrichment (OpenLibrary / Google Books). +kobo-shelf implements the Kobo Store API, so a Kobo device syncs books, covers, +reading progress, and shelves directly from your own server with no Kobo account. +Books come from a [Calibre](https://calibre-ebook.com/) library (read-only ingest of +`metadata.db`) and/or direct EPUB uploads via the web portal or the bundled Calibre +plugin. Calibre is optional. ## Quick start -```sh -# Build (server binary with the web UI bundle embedded) +```bash cargo build --release -# Run against a Calibre library... -kobo-shelf -l /path/to/calibre/library - -# ...or with no library at all (uploads-only catalog) -kobo-shelf - -# Bootstrap the first admin account on startup, then open the web UI -KOBO_SHELF_ADMIN_ACCOUNT=admin KOBO_SHELF_ADMIN_PASSWORD=change-me kobo-shelf -# → browse to http://127.0.0.1:8080, log in, and start uploading / setting up a device -``` - -With [Nix](https://nixos.org/): `nix run .# -- -l /path/to/calibre/library`, or -`nix build` for the package. A NixOS module is exported at -`.#nixosModules.kobo-shelf`. - -## Configuration - -Every option can be set via CLI flag, environment variable, or a `config.toml`. -Precedence is: **CLI args > environment / `.env` > `config.toml` > defaults**. -(`config.toml` path is `$KOBO_SHELF_CONFIG`, else `./config.toml`; it is optional.) - -### Core - -| Flag | Env var | Default | Description | -|------|---------|---------|-------------| -| `-p, --socket` | `KOBO_SHELF_HOST` | `127.0.0.1:8080` | Listen address (TCP `host:port` or `unix:/path`) | -| `-u, --external-url` | `KOBO_SHELF_EXTERNAL_URL` | `http://` | Public URL base for download/image links | -| `-l, --calibre-library-path` | `KOBO_SHELF_CALIBRE_LIBRARY` | *(none)* | Path to a Calibre library directory (optional) | -| `--calibre-ingest` | `KOBO_SHELF_CALIBRE_INGEST` | `true` | Ingest the Calibre library into the catalog on startup | -| `-d, --app-db-path` | `KOBO_SHELF_APP_DB` | `kobo-shelf.db` | Path to the application SQLite database | -| `--upload-dir` | `KOBO_SHELF_UPLOAD_DIR` | `uploads` | Directory for uploaded books and their covers | -| `--max-upload-size` | `KOBO_SHELF_MAX_UPLOAD_SIZE` | `209715200` | Maximum accepted upload size, in bytes (200 MiB) | -| `-s, --sync-item-limit` | `KOBO_SHELF_SYNC_ITEM_LIMIT` | `100` | Max items per sync response | - -### Uploads & metadata - -| Flag | Env var | Default | Description | -|------|---------|---------|-------------| -| `--kepubify-path` | `KOBO_SHELF_KEPUBIFY_PATH` | *(none)* | Path to `kepubify`; when set, uploads are converted EPUB→KEPUB | -| `--enrichment` | `KOBO_SHELF_ENRICHMENT` | `none` | Online metadata/cover source: `none` \| `openlibrary` \| `google` | - -### Accounts & login - -| Flag | Env var | Default | Description | -|------|---------|---------|-------------| -| — | `KOBO_SHELF_ADMIN_ACCOUNT` | *(none)* | Username of an admin account created/ensured on startup | -| — | `KOBO_SHELF_ADMIN_PASSWORD` | *(none)* | Password for that admin account (pair with the above) | -| `--allow-registration` | `KOBO_SHELF_ALLOW_REGISTRATION` | `false` | Allow self-service signup at `/register` | -| `--local-login` | `KOBO_SHELF_LOCAL_LOGIN` | `true` | Enable the username/password login form | -| `--oidc-issuer-url` | `KOBO_SHELF_OIDC_ISSUER` | *(none)* | OIDC issuer base URL (enables SSO when set with the two below) | -| `--oidc-client-id` | `KOBO_SHELF_OIDC_CLIENT_ID` | *(none)* | OIDC client id for this instance | -| — | `KOBO_SHELF_OIDC_CLIENT_SECRET` / `_FILE` | *(none)* | OIDC client secret (kept out of `AppConfig`, which is logged) | -| `--oidc-scopes` | `KOBO_SHELF_OIDC_SCOPES` | `openid profile email` | Requested OIDC scopes | -| `--oidc-username-claim` | `KOBO_SHELF_OIDC_USERNAME_CLAIM` | `preferred_username` | ID-token claim used as the username | - -OIDC is enabled when issuer + client id + secret are all set. The server -**refuses to start** if `--local-login` is off and OIDC is not configured — -there would be no way to log in. - -### Store profile - -`KOBO_SHELF_STORE_USER_ID`, `KOBO_SHELF_STORE_EMAIL`, and `KOBO_SHELF_STORE_COUNTRY` -(default country `US`) back the `/v1/user/profile` response; the UserId must -match what the device recorded at sign-in. Usually only relevant when adopting a -device that was previously signed into a real Kobo account. - -### Examples - -```sh -# Listen on all interfaces, behind a reverse proxy -kobo-shelf -p 0.0.0.0:9090 -l /srv/calibre -u https://books.example.com - -# Uploads-only, with kepubify conversion and OpenLibrary enrichment -kobo-shelf --kepubify-path kepubify --enrichment openlibrary - -# SSO-only login (no local password form) -KOBO_SHELF_OIDC_ISSUER=https://auth.example.com \ -KOBO_SHELF_OIDC_CLIENT_ID=kobo-shelf \ -KOBO_SHELF_OIDC_CLIENT_SECRET_FILE=/run/secrets/kobo-shelf-oidc \ -kobo-shelf --local-login false -l /srv/calibre -u https://books.example.com -``` - -### Logging - -kobo-shelf uses the `RUST_LOG` environment variable for log level control -(via `tracing-subscriber`). Defaults to `info`. - -```sh -RUST_LOG=debug kobo-shelf -l /srv/calibre -``` - -## The web UI - -kobo-shelf serves a browser UI (a Leptos client-side app, embedded in the binary) -at the root URL. Log in with a local account or via SSO, then: - -- **Library** — browse the catalog, upload EPUBs, edit metadata and covers, - toggle per-book sync, and trash/restore books. -- **Set up a device** — writes the two device files for you on Chromium-based - browsers (see the next section for the manual equivalent). -- **Settings** — runtime toggles (e.g. whether unknown store paths proxy to - Kobo, and the default sync flag for new uploads) and API-token management. - -**First account.** Set `KOBO_SHELF_ADMIN_ACCOUNT` + `KOBO_SHELF_ADMIN_PASSWORD` to -create/ensure an admin on startup, or enable `--allow-registration` and sign up -at `/register`. - -**Device & plugin tokens.** Devices and the Calibre plugin authenticate with a -per-user token minted in the web UI (Settings → tokens). The same token type is -used both for a Kobo device's `Authorization: Bearer` header and for the plugin. - -## Adding books - -Books reach the catalog in two ways, and both end up in the same unified -catalog that the Kobo sync reads from: - -1. **Calibre ingest.** Point kobo-shelf at a Calibre library - (`-l /path/to/library`); on startup (and via `POST /api/ingest/calibre`) it - reads `metadata.db` read-only and ingests the books. The Calibre library and - its `metadata.db` are **never modified**. -2. **Uploads.** Upload EPUBs from the web UI, or use the **Calibre plugin** to - push selected books from Calibre. Uploaded files live under `--upload-dir`. - On upload kobo-shelf extracts OPF metadata + cover, optionally enriches missing - fields online, and optionally converts EPUB→KEPUB with `kepubify`. - -### Calibre plugin - -The `calibre-plugin/` directory contains **Kobo Shelf Upload**, a Calibre plugin -that sends selected books to your server. It embeds Calibre's curated metadata -(title, authors, series, cover) into an in-memory copy of each EPUB before -sending — your library files on disk are untouched — and pre-checks by book -UUID so already-synced books are skipped without re-uploading. - -Install the packaged zip from the -[releases page](https://git.darksailor.dev/servius/kobo-shelf/releases) (Calibre → -Preferences → Plugins → Load plugin from file → `kobo-shelf-upload.zip`), then set -the server URL and an API token in the plugin's config. To build it yourself: -`cd calibre-plugin && uv run build.py`. - -## Setting up a freshly-reset Kobo (no Kobo account) - -kobo-shelf can drive a factory-reset device with **no Kobo account at all**. The -device authenticates to kobo-shelf with an `Authorization: Bearer` token: the auth -token is carried inside the `KoboAccessToken` (a JWT claim), and a fake local -`user` row convinces the firmware it is signed in. Store, auth, annotations, and -downloads are all served by kobo-shelf — nothing reaches Kobo. - -> **Easiest path:** open the web UI, log in, and click **Set up a device** — the -> browser writes both files below for you (Chromium-based browsers only). The -> manual steps follow for everyone else. - -**Prerequisites** - -- kobo-shelf running and reachable at your external URL (e.g. `https://books.example.com`). -- An auth token: a row in the `auth_tokens` table with a `token` string and - `user_id` (generate one in the web UI). Note the token value. -- `sqlite3` and `python3` on the machine you plug the device into. - -**On the device.** Factory-reset the Kobo and connect it via USB (it mounts at -e.g. `/run/media/$USER/KOBOeReader`). Set `DEV=/run/media/$USER/KOBOeReader` and -`TOKEN=`. - -1. **Point the device at kobo-shelf.** Append to `$DEV/.kobo/Kobo/Kobo eReader.conf` - (remove any existing `api_endpoint=` line first — the last one wins). The - endpoint is **token-free**; the token travels in the header: - - ```ini - [OneStoreServices] - api_endpoint=https://books.example.com/kobo - ``` - -2. **Fake a signed-in account** in `$DEV/.kobo/KoboReader.sqlite` so the firmware - skips the sign-in wizard while staying in sync mode. Do **not** use - `SideloadedMode=true` — that disables sync. The `KoboAccessToken` is an - unsigned JWT (never cryptographically verified) whose `kobo_shelf_token` claim - carries your auth token — that claim is what authenticates every request. - - ```sh - JWT="$(TOKEN="$TOKEN" python3 -c 'import base64,json,os - b=lambda o: base64.urlsafe_b64encode(json.dumps(o).encode()).rstrip(b"=").decode() - print(b({"alg":"none","typ":"JWT"})+"."+b({"sub":"kobo-shelf","exp":4102444800,"kobo_shelf_token":os.environ["TOKEN"]})+".")')" - - sqlite3 "$DEV/.kobo/KoboReader.sqlite" " - INSERT OR REPLACE INTO user - (UserID, UserKey, UserEmail, UserDisplayName, Storefront, - IsOneStoreAccount, KoboAccessToken, KoboAccessTokenExpiry) - VALUES - ('-', 'kobo-shelf', 'reader@kobo-shelf.local', 'Reader', 'US', - 'true', '$JWT', '2099-12-31T23:59:59.0000000Z');" - ``` - -3. **Arm the device registration** so the device's first `device_auth` (which - carries the device serial but no auth header) binds the serial to your token. - In the kobo-shelf **app** database (e.g. `/var/lib/kobo-shelf/kobo-shelf.db`), with - `USER_ID` the owner of the token: - - ```sh - sqlite3 /path/to/kobo-shelf.db " - INSERT OR REPLACE INTO pending_registrations (auth_token, user_id) - VALUES ('$TOKEN', USER_ID);" - ``` - - (The **Set up a device** page does this for you automatically.) - -4. **Flush and eject**, then unplug: - - ```sh - sync && udisksctl unmount -b /dev/sdX # your Kobo's block device - ``` - -5. **Reboot the Kobo.** It boots straight to the home screen. Trigger a sync - (Menu → Sync). Library, covers, reading state, and annotations all sync from - your Calibre library — the UI shows sync completing, no "Sync failed". - -## Architecture - -Two routers are mounted: the Kobo API under `/kobo/{auth_token}` and the web UI -+ `/api/...` routes at the root. The `books` table is the **single source of -truth** — handlers and the sync engine query it, never Calibre live. - -``` -src/ - main.rs Entry point: config, DB setup, admin bootstrap, server start - config.rs AppConfig (clap) + optional config.toml - errors.rs Error types (unit variants + error-stack) - web.rs Web API + static SPA serving (/api/*, /covers, login, OIDC) - oidc.rs OIDC relying party: discovery, PKCE, token exchange - db/ - calibre_db.rs Read-only reader for Calibre's metadata.db (ingest only) - app_db.rs Read-write app DB: sync state, accounts, shelves, settings - book_store.rs BookStore: the unified `books` catalog - ingest/ - calibre.rs metadata.db -> `books` table (source='calibre') - upload/ - handler.rs POST /api/upload, cover/metadata edit endpoints - epub.rs EPUB OPF metadata + cover extraction - enrich.rs Optional online metadata enrichment - cover.rs Cover transcode to JPEG - convert.rs Optional kepubify EPUB->KEPUB conversion - kobo/ - router.rs axum router for the Kobo API - handlers.rs Kobo request handlers - models.rs Kobo API JSON types (PascalCase serde) - resources.rs 131-entry resources dictionary (six overridden) - sync.rs SyncEngine: incremental sync algorithm - sync_token.rs Sync token (base64-encoded JSON in HTTP header) - auth.rs AppState and KoboAuth extractor -web-ui/ Leptos client-side web UI (built with Trunk, embedded) -calibre-plugin/ Kobo Shelf Upload Calibre plugin (Python, stdlib only) -``` - -### Databases - -- **Calibre `metadata.db`** (read-only) -- only ever *read*, by the ingest. - Never written. -- **App database** (`kobo-shelf.db`) -- the `books` catalog, sync state, reading - progress, shelves, accounts + sessions + auth tokens, per-user sync - preferences, device registrations, and runtime settings. Schema is managed by - an inline, append-only migration list (no migration framework) and created - automatically on first run. - -## Kobo eReader sync API reference - -The following documents the Kobo Store API as implemented by kobo-shelf. Endpoints -are served under `/kobo/` and authenticated with the device's -`Authorization: Bearer` token (the auth token, wrapped in a `KoboAccessToken` -JWT claim). Self-authorizing image and download links keep the token in the -path (`/kobo/{auth_token}/...`) since the device fetches them without a header. - -### Initialization - -#### `GET /v1/initialization` - -Returns a resource dictionary containing URLs for all Kobo API services. The -response includes 131 resource entries. kobo-shelf overrides six keys to point at -itself; the rest are left pointing at Kobo: - -- `image_host` -- base URL for cover images -- `image_url_template` -- URL template for cover image requests -- `image_url_quality_template` -- URL template with quality parameter -- `library_sync` -- URL for the sync endpoint -- `device_auth` -- device authentication endpoint -- `device_refresh` -- token refresh endpoint - -Response header: `x-kobo-apitoken: e30=` (base64-encoded empty JSON object). - -### Library sync - -#### `GET /v1/library/sync` - -Main incremental sync endpoint. Returns new, changed, and deleted entitlements, -reading states, and shelf (tag) changes since the last sync. - -**Request header:** `x-kobo-synctoken` -- base64-encoded JSON envelope: +# Against a Calibre library (directory containing metadata.db) +KOBO_SHELF_ADMIN_ACCOUNT=admin KOBO_SHELF_ADMIN_PASSWORD=change-me \ + ./target/release/kobo-shelf -l /path/to/calibre-library -```json -{ - "version": "1-1-0", - "data": { - "raw_kobo_store_token": "", - "books_last_modified": 0.0, - "books_last_created": 0.0, - "archive_last_modified": 0.0, - "reading_state_last_modified": 0.0, - "tags_last_modified": 0.0 - } -} +# Or uploads-only, no library +./target/release/kobo-shelf ``` -Timestamp fields are Unix epoch seconds (float). On first sync the header is -absent and all timestamps default to minimum values, triggering a full sync. - -**Response headers:** - -- `x-kobo-synctoken` -- updated token for the next request -- `x-kobo-sync: continue` -- present when there are more items to fetch - (pagination, limit controlled by `--sync-item-limit`) - -**Response body:** JSON array of sync items. Each item is one of: - -- **`NewEntitlement`** -- a book not previously synced to this device -- **`ChangedEntitlement`** -- a book whose metadata changed since last sync -- **`DeletedEntitlement`** -- UUID of a book that was archived/removed -- **`NewTag`** / **`ChangedTag`** / **`DeletedTag`** -- shelf changes -- **`ChangedReadingState`** -- reading progress updated since last sync - -**Sync algorithm:** - -1. Query the catalog (`books` table) for the user's syncable books with a KEPUB - or EPUB format not yet reflected in this device's sync point. A book is - syncable when its per-user preference (or the book default) is on, so a book - toggled off in the web UI drops out of that user's next sync. -2. Build entitlement + metadata for each book. Classify as `NewEntitlement` or - `ChangedEntitlement` based on whether the book's timestamp is newer than the - sync token's `books_last_created`. -3. Advance the device's sync point. -4. Fetch reading states modified since `reading_state_last_modified` (excluding - books already included above). -5. Sync shelves: deleted shelves first, then new, then changed. -6. KEPUB format is preferred over EPUB when both are available. - -### Book metadata - -#### `GET /v1/library/{book_uuid}/metadata` - -Returns metadata for a single book as a JSON array containing one metadata -object. - -Metadata includes: title, authors, description, publisher, language, publish -date, series info (name + index), and download URLs. - -### Reading state - -#### `GET /v1/library/{book_uuid}/state` - -Returns the current reading state for a book (bookmark position, statistics, -read status). - -#### `PUT /v1/library/{book_uuid}/state` - -Updates reading state. Request body contains an array of state updates, each -optionally including: +Then open , log in, and add books or pair a device. -- **`CurrentBookmark`** -- progress percent, content source progress, location - (source, type, value) -- **`Statistics`** -- spent reading minutes, remaining time minutes -- **`StatusInfo`** -- reading status string (`ReadyToRead`, `Reading`, - `Finished`) - -### Book deletion - -#### `DELETE /v1/library/{book_uuid}` - -Archives a book. Sets `is_archived = true` in the app database and removes it -from `synced_books` so it will not appear in future syncs. Does not delete the -file from the Calibre library. - -### Shelves (tags) - -#### `POST /v1/library/tags` - -Creates a new shelf. Request body: `{ "Name": "...", "Items": [...] }`. - -#### `DELETE /v1/library/tags/{tag_id}` - -Deletes a shelf. Archives it first (for sync propagation), then removes it. - -#### `PUT /v1/library/tags/{tag_id}` - -Renames a shelf. Request body: `{ "Name": "..." }`. - -#### `POST /v1/library/tags/{tag_id}/items` - -Adds books to a shelf. Request body: `{ "Items": [{ "RevisionId": "book-uuid" }] }`. - -#### `POST /v1/library/tags/{tag_id}/items/delete` - -Removes books from a shelf. Same request body format as adding items. - -### Authentication - -#### `POST /v1/auth/device` -#### `POST /v1/auth/refresh` -#### `POST /v1/user/add-device` - -Called token-free (no `Authorization` header) during the device's auth flow. -`device_auth` resolves the device's serial number to a real auth token via the -`device_registrations` table — bound on first sync by claiming a -`pending_registrations` row that device setup inserts — and returns an -`AccessToken` (a JWT carrying that token in its `kobo_shelf_token` claim). The -device then presents that token as `Authorization: Bearer` on every subsequent -request; the `KoboAuth` extractor validates it against the app database. -Self-authorizing image and download links instead carry the token in the path -(`/kobo/{auth_token}/...`), since the device fetches them without a header. - -### Book download - -#### `GET /download/{book_id}/{format}` - -Downloads a book file, resolved through the catalog (the Calibre library for -ingested books, or the upload directory for uploaded ones). The format is -typically `kepub` or `epub`. Returns the file with appropriate `Content-Type` -and `Content-Disposition` headers. - -### Cover images - -#### `GET /{book_uuid}/{width}/{height}/{is_greyscale}/image.jpg` -#### `GET /{book_uuid}/{width}/{height}/{quality}/{is_greyscale}/image.jpg` - -Serves a book's cover image. The file is resolved through the catalog: a -cover override set in the web UI wins, else the upload directory (uploaded -books) or the Calibre library's `cover.jpg` (ingested books). Width, height, -quality, and greyscale parameters are accepted but the original cover is served -unmodified. - -### Stub endpoints - -The following endpoints return empty or minimal responses to prevent errors on -the Kobo device: - -| Endpoint | Response | -|----------|----------| -| `GET /` | `{}` | -| `GET /v1/user/loyalty/benefits` | `{"Benefits": {}}` | -| `GET,POST /v1/analytics/gettests` | `{"Result":"Success","TestKey":"A","Tests":{}}` | -| `/v1/user/loyalty/*` | `200 OK` | -| `/v1/user/profile` | `200 OK` | -| `/v1/user/wishlist` | `200 OK` | -| `/v1/user/recommendations` | `200 OK` | -| `/v1/analytics/*` | `200 OK` | -| `/v1/assets` | `200 OK` | -| `/v1/products/*` | `200 OK` | -| `/v1/affiliate` | `200 OK` | -| `/v1/deals` | `200 OK` | - -## Sync token format - -The sync token is passed between client and server via the `x-kobo-synctoken` -HTTP header. It is a base64-encoded JSON envelope: - -```json -{ - "version": "1-1-0", - "data": { - "raw_kobo_store_token": "", - "books_last_modified": 1700000000.0, - "books_last_created": 1700000000.0, - "archive_last_modified": 1700000000.0, - "reading_state_last_modified": 1700000000.0, - "tags_last_modified": 1700000000.0 - } -} -``` +## Documentation -- **version** -- must be `>= 1-0-0` (validated on parse) -- **Timestamp fields** -- Unix epoch seconds; each is advanced as the - corresponding data type is synced -- If the header contains a `.` character, it is treated as a raw Kobo store - token (forwarded from Kobo's servers) and all timestamps reset to minimum -- If the header is empty or absent, a default token with minimum timestamps is - used, triggering a full sync +- [docs/manual.md](docs/manual.md) — full guide: configuration reference, web UI, + adding books, pairing a factory-reset Kobo, architecture, API reference +- [docs/README.md](docs/README.md) — knowledge base: device internals, auth research, + sync algorithm, wipe traps +- [calibre-plugin/README.md](calibre-plugin/README.md) — Calibre upload plugin +- [AGENTS.md](AGENTS.md) — build, test, and code-style conventions ## License diff --git a/calibre-plugin/README.md b/calibre-plugin/README.md index 75d3fa7..9ac1a21 100644 --- a/calibre-plugin/README.md +++ b/calibre-plugin/README.md @@ -25,7 +25,7 @@ dependencies are installed. The zip is just the `.py` files plus the ## Configure -1. In kobo-shelf' web UI, create an API token (the **Tokens** section — the same +1. In kobo-shelf's web UI, create an API token (the **Tokens** section — the same tokens used for Kobo devices). 2. Calibre → **Preferences → Plugins → Kobo Shelf Upload → Customize plugin**. 3. Enter your server URL (e.g. `https://books.example.com`) and paste the token. diff --git a/calibre-plugin/main.py b/calibre-plugin/main.py index 7741b86..7741021 100644 --- a/calibre-plugin/main.py +++ b/calibre-plugin/main.py @@ -143,7 +143,7 @@ def _prepare_book(db, book_id): def _embed_metadata(db, book_id, raw, fmt): """Embed Calibre's curated metadata (incl. series + cover) into the EPUB so - kobo-shelf' OPF parser preserves it. Falls back to the raw bytes on any error.""" + kobo-shelf's OPF parser preserves it. Falls back to the raw bytes on any error.""" from calibre.ebooks.metadata.meta import set_metadata stream_type = "kepub" if fmt == "KEPUB" else "epub" diff --git a/docs/manual.md b/docs/manual.md new file mode 100644 index 0000000..2391511 --- /dev/null +++ b/docs/manual.md @@ -0,0 +1,533 @@ +# kobo-shelf + +A self-hosted Kobo eReader sync server and web library manager. Written in Rust +with [axum](https://github.com/tokio-rs/axum) and a +[Leptos](https://leptos.dev/) web UI. + +kobo-shelf implements the Kobo Store API so a Kobo device can sync books directly +from your own server, with no dependency on Kobo's cloud. Books enter the +catalog two ways: + +- **From a [Calibre](https://calibre-ebook.com/) library** — kobo-shelf reads + `metadata.db` (read-only) and ingests it into its own catalog. +- **By upload** — a browser upload portal and a bundled **Calibre plugin** push + EPUBs straight to the server. Calibre is entirely optional: with no library + configured, the catalog is served from uploads alone. + +Everything else — sync state, reading progress, shelves, per-user sync +preferences, uploaded files, and accounts — lives in kobo-shelf's own SQLite +database. A web UI provides library management, per-book cover and metadata +editing, device setup, account/token management, and login via local +password or OIDC single sign-on. + +Inspired by the Kobo sync integration in +[calibre-web](https://github.com/janeczku/calibre-web). + +## Features + +- **Kobo device sync** — library, covers, reading state, and shelves sync to a + Kobo with no Kobo account (drive a factory-reset device; see below). +- **Two catalog sources** — Calibre ingest and/or direct EPUB uploads, unified + in one catalog that is the single source of truth. +- **Web library manager** — browse the catalog, upload books, edit metadata + (title, authors, series, description), change or search covers, toggle + per-book sync, and trash/restore books. +- **Calibre plugin** — send selected books from Calibre to kobo-shelf; it + pre-checks by book UUID and skips already-synced books without re-uploading. + See [releases](https://git.darksailor.dev/servius/kobo-shelf/releases). +- **Accounts** — local username/password and/or OIDC SSO; optional + self-service registration; per-user auth tokens for devices and the plugin. +- **Optional niceties** — `kepubify` conversion (EPUB→KEPUB for native per-page + progress) and online metadata/cover enrichment (OpenLibrary / Google Books). + +## Quick start + +```sh +# Build (server binary with the web UI bundle embedded) +cargo build --release + +# Run against a Calibre library... +kobo-shelf -l /path/to/calibre/library + +# ...or with no library at all (uploads-only catalog) +kobo-shelf + +# Bootstrap the first admin account on startup, then open the web UI +KOBO_SHELF_ADMIN_ACCOUNT=admin KOBO_SHELF_ADMIN_PASSWORD=change-me kobo-shelf +# → browse to http://127.0.0.1:8080, log in, and start uploading / setting up a device +``` + +With [Nix](https://nixos.org/): `nix run .# -- -l /path/to/calibre/library`, or +`nix build` for the package. A NixOS module is exported at +`.#nixosModules.kobo-shelf`. + +## Configuration + +Every option can be set via CLI flag, environment variable, or a `config.toml`. +Precedence is: **CLI args > environment / `.env` > `config.toml` > defaults**. +(`config.toml` path is `$KOBO_SHELF_CONFIG`, else `./config.toml`; it is optional.) + +### Core + +| Flag | Env var | Default | Description | +|------|---------|---------|-------------| +| `-p, --socket` | `KOBO_SHELF_HOST` | `127.0.0.1:8080` | Listen address (TCP `host:port` or `unix:/path`) | +| `-u, --external-url` | `KOBO_SHELF_EXTERNAL_URL` | `http://` | Public URL base for download/image links | +| `-l, --calibre-library-path` | `KOBO_SHELF_CALIBRE_LIBRARY` | *(none)* | Path to a Calibre library directory (optional) | +| `--calibre-ingest` | `KOBO_SHELF_CALIBRE_INGEST` | `true` | Ingest the Calibre library into the catalog on startup | +| `-d, --app-db-path` | `KOBO_SHELF_APP_DB` | `kobo-shelf.db` | Path to the application SQLite database | +| `--upload-dir` | `KOBO_SHELF_UPLOAD_DIR` | `uploads` | Directory for uploaded books and their covers | +| `--max-upload-size` | `KOBO_SHELF_MAX_UPLOAD_SIZE` | `209715200` | Maximum accepted upload size, in bytes (200 MiB) | +| `-s, --sync-item-limit` | `KOBO_SHELF_SYNC_ITEM_LIMIT` | `100` | Max items per sync response | + +### Uploads & metadata + +| Flag | Env var | Default | Description | +|------|---------|---------|-------------| +| `--kepubify-path` | `KOBO_SHELF_KEPUBIFY_PATH` | *(none)* | Path to `kepubify`; when set, uploads are converted EPUB→KEPUB | +| `--enrichment` | `KOBO_SHELF_ENRICHMENT` | `none` | Online metadata/cover source: `none` \| `openlibrary` \| `google` | + +### Accounts & login + +| Flag | Env var | Default | Description | +|------|---------|---------|-------------| +| — | `KOBO_SHELF_ADMIN_ACCOUNT` | *(none)* | Username of an admin account created/ensured on startup | +| — | `KOBO_SHELF_ADMIN_PASSWORD` | *(none)* | Password for that admin account (pair with the above) | +| `--allow-registration` | `KOBO_SHELF_ALLOW_REGISTRATION` | `false` | Allow self-service signup at `/register` | +| `--local-login` | `KOBO_SHELF_LOCAL_LOGIN` | `true` | Enable the username/password login form | +| `--oidc-issuer-url` | `KOBO_SHELF_OIDC_ISSUER` | *(none)* | OIDC issuer base URL (enables SSO when set with the two below) | +| `--oidc-client-id` | `KOBO_SHELF_OIDC_CLIENT_ID` | *(none)* | OIDC client id for this instance | +| — | `KOBO_SHELF_OIDC_CLIENT_SECRET` / `_FILE` | *(none)* | OIDC client secret (kept out of `AppConfig`, which is logged) | +| `--oidc-scopes` | `KOBO_SHELF_OIDC_SCOPES` | `openid profile email` | Requested OIDC scopes | +| `--oidc-username-claim` | `KOBO_SHELF_OIDC_USERNAME_CLAIM` | `preferred_username` | ID-token claim used as the username | + +OIDC is enabled when issuer + client id + secret are all set. The server +**refuses to start** if `--local-login` is off and OIDC is not configured — +there would be no way to log in. + +### Store profile + +`KOBO_SHELF_STORE_USER_ID`, `KOBO_SHELF_STORE_EMAIL`, and `KOBO_SHELF_STORE_COUNTRY` +(default country `US`) back the `/v1/user/profile` response; the UserId must +match what the device recorded at sign-in. Usually only relevant when adopting a +device that was previously signed into a real Kobo account. + +### Examples + +```sh +# Listen on all interfaces, behind a reverse proxy +kobo-shelf -p 0.0.0.0:9090 -l /srv/calibre -u https://books.example.com + +# Uploads-only, with kepubify conversion and OpenLibrary enrichment +kobo-shelf --kepubify-path kepubify --enrichment openlibrary + +# SSO-only login (no local password form) +KOBO_SHELF_OIDC_ISSUER=https://auth.example.com \ +KOBO_SHELF_OIDC_CLIENT_ID=kobo-shelf \ +KOBO_SHELF_OIDC_CLIENT_SECRET_FILE=/run/secrets/kobo-shelf-oidc \ +kobo-shelf --local-login false -l /srv/calibre -u https://books.example.com +``` + +### Logging + +kobo-shelf uses the `RUST_LOG` environment variable for log level control +(via `tracing-subscriber`). Defaults to `info`. + +```sh +RUST_LOG=debug kobo-shelf -l /srv/calibre +``` + +## The web UI + +kobo-shelf serves a browser UI (a Leptos client-side app, embedded in the binary) +at the root URL. Log in with a local account or via SSO, then: + +- **Library** — browse the catalog, upload EPUBs, edit metadata and covers, + toggle per-book sync, and trash/restore books. +- **Set up a device** — writes the two device files for you on Chromium-based + browsers (see the next section for the manual equivalent). +- **Settings** — runtime toggles (e.g. whether unknown store paths proxy to + Kobo, and the default sync flag for new uploads) and API-token management. + +**First account.** Set `KOBO_SHELF_ADMIN_ACCOUNT` + `KOBO_SHELF_ADMIN_PASSWORD` to +create/ensure an admin on startup, or enable `--allow-registration` and sign up +at `/register`. + +**Device & plugin tokens.** Devices and the Calibre plugin authenticate with a +per-user token minted in the web UI (Settings → tokens). The same token type is +used both for a Kobo device's `Authorization: Bearer` header and for the plugin. + +## Adding books + +Books reach the catalog in two ways, and both end up in the same unified +catalog that the Kobo sync reads from: + +1. **Calibre ingest.** Point kobo-shelf at a Calibre library + (`-l /path/to/library`); on startup (and via `POST /api/ingest/calibre`) it + reads `metadata.db` read-only and ingests the books. The Calibre library and + its `metadata.db` are **never modified**. +2. **Uploads.** Upload EPUBs from the web UI, or use the **Calibre plugin** to + push selected books from Calibre. Uploaded files live under `--upload-dir`. + On upload kobo-shelf extracts OPF metadata + cover, optionally enriches missing + fields online, and optionally converts EPUB→KEPUB with `kepubify`. + +### Calibre plugin + +The `calibre-plugin/` directory contains **Kobo Shelf Upload**, a Calibre plugin +that sends selected books to your server. It embeds Calibre's curated metadata +(title, authors, series, cover) into an in-memory copy of each EPUB before +sending — your library files on disk are untouched — and pre-checks by book +UUID so already-synced books are skipped without re-uploading. + +Install the packaged zip from the +[releases page](https://git.darksailor.dev/servius/kobo-shelf/releases) (Calibre → +Preferences → Plugins → Load plugin from file → `kobo-shelf-upload.zip`), then set +the server URL and an API token in the plugin's config. To build it yourself: +`cd calibre-plugin && uv run build.py`. + +## Setting up a freshly-reset Kobo (no Kobo account) + +kobo-shelf can drive a factory-reset device with **no Kobo account at all**. The +device authenticates to kobo-shelf with an `Authorization: Bearer` token: the auth +token is carried inside the `KoboAccessToken` (a JWT claim), and a fake local +`user` row convinces the firmware it is signed in. Store, auth, annotations, and +downloads are all served by kobo-shelf — nothing reaches Kobo. + +> **Easiest path:** open the web UI, log in, and click **Set up a device** — the +> browser writes both files below for you (Chromium-based browsers only). The +> manual steps follow for everyone else. + +**Prerequisites** + +- kobo-shelf running and reachable at your external URL (e.g. `https://books.example.com`). +- An auth token: a row in the `auth_tokens` table with a `token` string and + `user_id` (generate one in the web UI). Note the token value. +- `sqlite3` and `python3` on the machine you plug the device into. + +**On the device.** Factory-reset the Kobo and connect it via USB (it mounts at +e.g. `/run/media/$USER/KOBOeReader`). Set `DEV=/run/media/$USER/KOBOeReader` and +`TOKEN=`. + +1. **Point the device at kobo-shelf.** Append to `$DEV/.kobo/Kobo/Kobo eReader.conf` + (remove any existing `api_endpoint=` line first — the last one wins). The + endpoint is **token-free**; the token travels in the header: + + ```ini + [OneStoreServices] + api_endpoint=https://books.example.com/kobo + ``` + +2. **Fake a signed-in account** in `$DEV/.kobo/KoboReader.sqlite` so the firmware + skips the sign-in wizard while staying in sync mode. Do **not** use + `SideloadedMode=true` — that disables sync. The `KoboAccessToken` is an + unsigned JWT (never cryptographically verified) whose `kobo_shelf_token` claim + carries your auth token — that claim is what authenticates every request. + + ```sh + JWT="$(TOKEN="$TOKEN" python3 -c 'import base64,json,os + b=lambda o: base64.urlsafe_b64encode(json.dumps(o).encode()).rstrip(b"=").decode() + print(b({"alg":"none","typ":"JWT"})+"."+b({"sub":"kobo-shelf","exp":4102444800,"kobo_shelf_token":os.environ["TOKEN"]})+".")')" + + sqlite3 "$DEV/.kobo/KoboReader.sqlite" " + INSERT OR REPLACE INTO user + (UserID, UserKey, UserEmail, UserDisplayName, Storefront, + IsOneStoreAccount, KoboAccessToken, KoboAccessTokenExpiry) + VALUES + ('-', 'kobo-shelf', 'reader@kobo-shelf.local', 'Reader', 'US', + 'true', '$JWT', '2099-12-31T23:59:59.0000000Z');" + ``` + +3. **Arm the device registration** so the device's first `device_auth` (which + carries the device serial but no auth header) binds the serial to your token. + In the kobo-shelf **app** database (e.g. `/var/lib/kobo-shelf/kobo-shelf.db`), with + `USER_ID` the owner of the token: + + ```sh + sqlite3 /path/to/kobo-shelf.db " + INSERT OR REPLACE INTO pending_registrations (auth_token, user_id) + VALUES ('$TOKEN', USER_ID);" + ``` + + (The **Set up a device** page does this for you automatically.) + +4. **Flush and eject**, then unplug: + + ```sh + sync && udisksctl unmount -b /dev/sdX # your Kobo's block device + ``` + +5. **Reboot the Kobo.** It boots straight to the home screen. Trigger a sync + (Menu → Sync). Library, covers, reading state, and annotations all sync from + your Calibre library — the UI shows sync completing, no "Sync failed". + +## Architecture + +Two routers are mounted: the Kobo API under `/kobo/{auth_token}` and the web UI ++ `/api/...` routes at the root. The `books` table is the **single source of +truth** — handlers and the sync engine query it, never Calibre live. + +``` +src/ + main.rs Entry point: config, DB setup, admin bootstrap, server start + config.rs AppConfig (clap) + optional config.toml + errors.rs Error types (unit variants + error-stack) + web.rs Web API + static SPA serving (/api/*, /covers, login, OIDC) + oidc.rs OIDC relying party: discovery, PKCE, token exchange + db/ + calibre_db.rs Read-only reader for Calibre's metadata.db (ingest only) + app_db.rs Read-write app DB: sync state, accounts, shelves, settings + book_store.rs BookStore: the unified `books` catalog + ingest/ + calibre.rs metadata.db -> `books` table (source='calibre') + upload/ + handler.rs POST /api/upload, cover/metadata edit endpoints + epub.rs EPUB OPF metadata + cover extraction + enrich.rs Optional online metadata enrichment + cover.rs Cover transcode to JPEG + convert.rs Optional kepubify EPUB->KEPUB conversion + kobo/ + router.rs axum router for the Kobo API + handlers.rs Kobo request handlers + models.rs Kobo API JSON types (PascalCase serde) + resources.rs 131-entry resources dictionary (six overridden) + sync.rs SyncEngine: incremental sync algorithm + sync_token.rs Sync token (base64-encoded JSON in HTTP header) + auth.rs AppState and KoboAuth extractor +web-ui/ Leptos client-side web UI (built with Trunk, embedded) +calibre-plugin/ Kobo Shelf Upload Calibre plugin (Python, stdlib only) +``` + +### Databases + +- **Calibre `metadata.db`** (read-only) -- only ever *read*, by the ingest. + Never written. +- **App database** (`kobo-shelf.db`) -- the `books` catalog, sync state, reading + progress, shelves, accounts + sessions + auth tokens, per-user sync + preferences, device registrations, and runtime settings. Schema is managed by + an inline, append-only migration list (no migration framework) and created + automatically on first run. + +## Kobo eReader sync API reference + +The following documents the Kobo Store API as implemented by kobo-shelf. Endpoints +are served under `/kobo/` and authenticated with the device's +`Authorization: Bearer` token (the auth token, wrapped in a `KoboAccessToken` +JWT claim). Self-authorizing image and download links keep the token in the +path (`/kobo/{auth_token}/...`) since the device fetches them without a header. + +### Initialization + +#### `GET /v1/initialization` + +Returns a resource dictionary containing URLs for all Kobo API services. The +response includes 131 resource entries. kobo-shelf overrides six keys to point at +itself; the rest are left pointing at Kobo: + +- `image_host` -- base URL for cover images +- `image_url_template` -- URL template for cover image requests +- `image_url_quality_template` -- URL template with quality parameter +- `library_sync` -- URL for the sync endpoint +- `device_auth` -- device authentication endpoint +- `device_refresh` -- token refresh endpoint + +Response header: `x-kobo-apitoken: e30=` (base64-encoded empty JSON object). + +### Library sync + +#### `GET /v1/library/sync` + +Main incremental sync endpoint. Returns new, changed, and deleted entitlements, +reading states, and shelf (tag) changes since the last sync. + +**Request header:** `x-kobo-synctoken` -- base64-encoded JSON envelope: + +```json +{ + "version": "1-1-0", + "data": { + "raw_kobo_store_token": "", + "books_last_modified": 0.0, + "books_last_created": 0.0, + "archive_last_modified": 0.0, + "reading_state_last_modified": 0.0, + "tags_last_modified": 0.0 + } +} +``` + +Timestamp fields are Unix epoch seconds (float). On first sync the header is +absent and all timestamps default to minimum values, triggering a full sync. + +**Response headers:** + +- `x-kobo-synctoken` -- updated token for the next request +- `x-kobo-sync: continue` -- present when there are more items to fetch + (pagination, limit controlled by `--sync-item-limit`) + +**Response body:** JSON array of sync items. Each item is one of: + +- **`NewEntitlement`** -- a book not previously synced to this device +- **`ChangedEntitlement`** -- a book whose metadata changed since last sync +- **`DeletedEntitlement`** -- UUID of a book that was archived/removed +- **`NewTag`** / **`ChangedTag`** / **`DeletedTag`** -- shelf changes +- **`ChangedReadingState`** -- reading progress updated since last sync + +**Sync algorithm:** + +1. Query the catalog (`books` table) for the user's syncable books with a KEPUB + or EPUB format not yet reflected in this device's sync point. A book is + syncable when its per-user preference (or the book default) is on, so a book + toggled off in the web UI drops out of that user's next sync. +2. Build entitlement + metadata for each book. Classify as `NewEntitlement` or + `ChangedEntitlement` based on whether the book's timestamp is newer than the + sync token's `books_last_created`. +3. Advance the device's sync point. +4. Fetch reading states modified since `reading_state_last_modified` (excluding + books already included above). +5. Sync shelves: deleted shelves first, then new, then changed. +6. KEPUB format is preferred over EPUB when both are available. + +### Book metadata + +#### `GET /v1/library/{book_uuid}/metadata` + +Returns metadata for a single book as a JSON array containing one metadata +object. + +Metadata includes: title, authors, description, publisher, language, publish +date, series info (name + index), and download URLs. + +### Reading state + +#### `GET /v1/library/{book_uuid}/state` + +Returns the current reading state for a book (bookmark position, statistics, +read status). + +#### `PUT /v1/library/{book_uuid}/state` + +Updates reading state. Request body contains an array of state updates, each +optionally including: + +- **`CurrentBookmark`** -- progress percent, content source progress, location + (source, type, value) +- **`Statistics`** -- spent reading minutes, remaining time minutes +- **`StatusInfo`** -- reading status string (`ReadyToRead`, `Reading`, + `Finished`) + +### Book deletion + +#### `DELETE /v1/library/{book_uuid}` + +Archives a book. Sets `is_archived = true` in the app database and removes it +from `synced_books` so it will not appear in future syncs. Does not delete the +file from the Calibre library. + +### Shelves (tags) + +#### `POST /v1/library/tags` + +Creates a new shelf. Request body: `{ "Name": "...", "Items": [...] }`. + +#### `DELETE /v1/library/tags/{tag_id}` + +Deletes a shelf. Archives it first (for sync propagation), then removes it. + +#### `PUT /v1/library/tags/{tag_id}` + +Renames a shelf. Request body: `{ "Name": "..." }`. + +#### `POST /v1/library/tags/{tag_id}/items` + +Adds books to a shelf. Request body: `{ "Items": [{ "RevisionId": "book-uuid" }] }`. + +#### `POST /v1/library/tags/{tag_id}/items/delete` + +Removes books from a shelf. Same request body format as adding items. + +### Authentication + +#### `POST /v1/auth/device` +#### `POST /v1/auth/refresh` +#### `POST /v1/user/add-device` + +Called token-free (no `Authorization` header) during the device's auth flow. +`device_auth` resolves the device's serial number to a real auth token via the +`device_registrations` table — bound on first sync by claiming a +`pending_registrations` row that device setup inserts — and returns an +`AccessToken` (a JWT carrying that token in its `kobo_shelf_token` claim). The +device then presents that token as `Authorization: Bearer` on every subsequent +request; the `KoboAuth` extractor validates it against the app database. +Self-authorizing image and download links instead carry the token in the path +(`/kobo/{auth_token}/...`), since the device fetches them without a header. + +### Book download + +#### `GET /download/{book_id}/{format}` + +Downloads a book file, resolved through the catalog (the Calibre library for +ingested books, or the upload directory for uploaded ones). The format is +typically `kepub` or `epub`. Returns the file with appropriate `Content-Type` +and `Content-Disposition` headers. + +### Cover images + +#### `GET /{book_uuid}/{width}/{height}/{is_greyscale}/image.jpg` +#### `GET /{book_uuid}/{width}/{height}/{quality}/{is_greyscale}/image.jpg` + +Serves a book's cover image. The file is resolved through the catalog: a +cover override set in the web UI wins, else the upload directory (uploaded +books) or the Calibre library's `cover.jpg` (ingested books). Width, height, +quality, and greyscale parameters are accepted but the original cover is served +unmodified. + +### Stub endpoints + +The following endpoints return empty or minimal responses to prevent errors on +the Kobo device: + +| Endpoint | Response | +|----------|----------| +| `GET /` | `{}` | +| `GET /v1/user/loyalty/benefits` | `{"Benefits": {}}` | +| `GET,POST /v1/analytics/gettests` | `{"Result":"Success","TestKey":"A","Tests":{}}` | +| `/v1/user/loyalty/*` | `200 OK` | +| `/v1/user/profile` | `200 OK` | +| `/v1/user/wishlist` | `200 OK` | +| `/v1/user/recommendations` | `200 OK` | +| `/v1/analytics/*` | `200 OK` | +| `/v1/assets` | `200 OK` | +| `/v1/products/*` | `200 OK` | +| `/v1/affiliate` | `200 OK` | +| `/v1/deals` | `200 OK` | + +## Sync token format + +The sync token is passed between client and server via the `x-kobo-synctoken` +HTTP header. It is a base64-encoded JSON envelope: + +```json +{ + "version": "1-1-0", + "data": { + "raw_kobo_store_token": "", + "books_last_modified": 1700000000.0, + "books_last_created": 1700000000.0, + "archive_last_modified": 1700000000.0, + "reading_state_last_modified": 1700000000.0, + "tags_last_modified": 1700000000.0 + } +} +``` + +- **version** -- must be `>= 1-0-0` (validated on parse) +- **Timestamp fields** -- Unix epoch seconds; each is advanced as the + corresponding data type is synced +- If the header contains a `.` character, it is treated as a raw Kobo store + token (forwarded from Kobo's servers) and all timestamps reset to minimum +- If the header is empty or absent, a default token with minimum timestamps is + used, triggering a full sync + +## License + +MIT