diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..cd1fef0 --- /dev/null +++ b/.gitignore @@ -0,0 +1,4 @@ +.venv/ +pages/ +__pycache__/ +*.pyc diff --git a/CRUSH.md b/CRUSH.md new file mode 100644 index 0000000..67df6ac --- /dev/null +++ b/CRUSH.md @@ -0,0 +1,107 @@ +# Grimoire — Project context + +What follows is hard-won knowledge from building M0 (injecting native ink +into reMarkable .rm v6 pages). Do not unlearn any of this. + +## Coordinates + +- **Origin is top-center**, not top-left. This is the single most important + fact about the v6 format and it cost us an hour. +- `x = 0` is the horizontal center of the page; negative values go left, + positive go right. Range is roughly -702 to 701 on a standard page. +- `y = 0` is the top of the page, increasing downward. Range is 0 to 1871. +- Real handwriting in existing notebooks shows x in the -300 to 300 range + and y in the 700 to 1050 range (roughly the middle third of the page). +- Source: reMarkable-kaitai `rmv6.ksy` spec; confirmed by inspecting actual + device files and rendering them. + +## Stroke parameters (matching real ink) + +- Tool: `Pen.BALLPOINT_2` (value 15) +- Color: `PenColor.BLACK` (value 0) +- `thickness_scale`: 2.0 +- `starting_length`: 0.0 +- Point `width`: 9 +- Point `pressure`: 150 + small variance (0-30) +- Point `speed`: 5 + small variance (0-10) +- Point `direction`: 90 + small variance +- These values were tuned by comparing against strokes from a real notebook + ("9.8" on Kieran's device). Real pen widths range 12-18, pressures + 0-255 averaging ~173. + +## File format injection (the append strategy) + +- **Never re-serialize original blocks.** `read_blocks()` followed by + `write_blocks()` loses data. rmscene warns "Some data has not been read. + The data may have been written using a newer format than this reader + supports." Respect this warning — re-serialized files crash xochitl. +- Instead: serialize only your new `SceneLineItemBlock`s via + `write_blocks()`, strip the 43-byte header it emits, and append the + remaining bytes to the original file. +- Layer 1 is always `CrdtId(0, 11)`. Inject `SceneLineItemBlock`s directly + into it as children. Do not create new SceneTreeBlocks or TreeNodeBlocks + — xochitl doesn't need them and they caused crashes in early experiments. +- Reuse author 1 (always present). No need to modify `AuthorIdsBlock`. +- CRDT sequence IDs starting at 200 for our blocks — safely above anything + in real files (which use low single digits). + +## Device management + +- `.content` file has `"sizeInBytes"` that must match the `.rm` file size + or xochitl crashes on load. Fix with sed before restart: + ``` + sed -i 's/"sizeInBytes": "[0-9]*"/"sizeInBytes": "NNNN"/' doc.content + ``` +- Too many `systemctl restart xochitl` calls trigger rate limiting. Use: + ``` + systemctl reset-failed xochitl && systemctl restart xochitl + ``` +- Busybox on the device is limited: no `base64`, `head` needs `-n N` syntax, + no `bash`, limited `find`. Use `od`/`hexdump` for binary inspection. +- The SSH MCP gets new session IDs on each connect. Old sessions die when + the device reboots or the connection drops. Always use the latest. + +## M0 script (`grimoire.py`) + +- Single-file Python script, no dependencies beyond `rmscene` (installed in + `./.venv`). +- Hershey simplex font subset: 21 glyphs (A-Z uppercase plus some + punctuation). +- Hershey glyph coordinates are in a 0..21 x 0..32 grid per glyph, + `GLYPH_WIDTH=21`, `GLYPH_SPACING=8`. +- Default text: "GRIMOIRE SAYS HELLO" at scale 1.1, origin (-300, 700). +- Usage: + ``` + python grimoire.py input.rm output.rm + ``` + +## Deploy workflow (from mac to device) + +```sh +# 1. Generate reply +.venv/bin/python3.13 grimoire.py pages/original.rm pages/replied.rm + +# 2. Push to device (adjust paths — get UUID from notebook metadata) +scp pages/replied.rm root@10.11.99.1:/path/to/page.rm + +# 3. Fix metadata and restart +ssh root@10.11.99.1 \ + "sed -i 's/\"sizeInBytes\": \"[0-9]*\"/\"sizeInBytes\": \"3959\"/' /path/to/doc.content \ + && systemctl reset-failed xochitl \ + && systemctl restart xochitl" +``` + +## Notebook UUID for testing + +Kieran's Grrimoire test notebook: +- UUID: `5c81a1df-c36c-4eea-9289-978dfcc655a1` +- Page: `5b66f39c-9d00-40a7-bce0-9fb3a044d446.rm` +- Path: `/home/root/.local/share/remarkable/xochitl/5c81a1df-c36c-4eea-9289-978dfcc655a1/` + +## Open issues from M0 + +- xochitl still occasionally crashes on reload despite correct `sizeInBytes`. + Need to investigate journalctl logs next time it happens. +- Hershey font subset is missing lowercase and numbers. M1 should add them. +- No whitespace awareness — text always renders at fixed position (M1). +- No trigger mechanism — just static injection (M2+). diff --git a/grimoire.py b/grimoire.py new file mode 100644 index 0000000..0325c16 --- /dev/null +++ b/grimoire.py @@ -0,0 +1,204 @@ +#!/usr/bin/env python3 +""" +Inject reply text as native ink into reMarkable .rm v6 pages. + +Reads an existing .rm v6 page and appends Hershey-rendered text strokes +directly into Layer 1 as SceneLineItemBlocks. Original blocks are never +re-serialized — replies are appended to the raw bytes, so data that +rmscene can't fully parse survives untouched. + +M0 proof for the Grimoire project. Depth 1 (file + reload). +See spec.md for the full architecture. + +v6 coordinate system: origin is top-center of page. + x = 0 -> center; negative -> left; positive -> right. + y = 0 -> top; positive -> down. + +Usage: + python grimoire.py input.rm output.rm # writes to output.rm + python grimoire.py input.rm # overwrites input + +Deploy to device: + scp replied.rm root@remarkable:/path/to/page.rm + ssh root@remarkable "sed -i 's/\"sizeInBytes\": \"[0-9]*\"/\"sizeInBytes\": \"3959\"/' /path/to/doc.content && systemctl reset-failed xochitl && systemctl restart xochitl" +""" + +import sys +from io import BytesIO + +from rmscene import ( + CrdtId, + CrdtSequenceItem, + read_blocks, + write_blocks, + scene_items as si, +) +from rmscene.scene_stream import ( + SceneLineItemBlock, +) + + +# -- Hershey simplex font (subset) ---------------------------------------- +# Each glyph is a list of polylines; each polyline is [(x,y), ...]. +# Coordinates are in a 0..21 x 0..32 grid (width x height). +# Source: derived from Hershey Simplex Roman, normalized. + +HERSHEY: dict[str, list[list[tuple[float, float]]]] = { + "A": [[(0, 32), (10, 0), (21, 32)], [(4, 20), (17, 20)]], + "B": [[(0, 0), (0, 32), (14, 32), (19, 28), (19, 22), (14, 18), (0, 18)], + [(14, 18), (19, 14), (19, 6), (14, 0), (0, 0)]], + "C": [[(19, 28), (14, 32), (5, 32), (0, 28), (0, 4), (5, 0), (14, 0), (19, 4)]], + "D": [[(0, 0), (0, 32), (12, 32), (19, 26), (19, 6), (12, 0), (0, 0)]], + "E": [[(19, 0), (0, 0), (0, 32), (19, 32)], [(0, 16), (14, 16)]], + "F": [[(19, 0), (0, 0), (0, 32), (19, 32)], [(0, 16), (14, 16)]], + "G": [[(19, 28), (14, 32), (5, 32), (0, 28), (0, 4), (5, 0), (14, 0), (19, 4), (19, 16), (12, 16)]], + "H": [[(0, 0), (0, 32)], [(21, 0), (21, 32)], [(0, 16), (21, 16)]], + "I": [[(5, 0), (16, 0)], [(10, 0), (10, 32)], [(5, 32), (16, 32)]], + "K": [[(0, 0), (0, 32)], [(19, 32), (0, 14), (19, 0)]], + "L": [[(0, 0), (0, 32), (19, 32)]], + "M": [[(0, 32), (0, 0), (10, 16), (21, 0), (21, 32)]], + "N": [[(0, 32), (0, 0), (21, 32), (21, 0)]], + "O": [[(5, 0), (0, 4), (0, 28), (5, 32), (16, 32), (21, 28), (21, 4), (16, 0), (5, 0)]], + "P": [[(0, 32), (0, 0), (14, 0), (19, 4), (19, 14), (14, 18), (0, 18)]], + "R": [[(0, 32), (0, 0), (14, 0), (19, 4), (19, 14), (14, 18), (0, 18)], + [(14, 18), (19, 32)]], + "S": [[(19, 4), (14, 0), (5, 0), (0, 4), (0, 12), (5, 16), (14, 16), (19, 20), (19, 28), (14, 32), (5, 32), (0, 28)]], + "T": [[(0, 0), (21, 0)], [(10, 0), (10, 32)]], + "U": [[(0, 0), (0, 28), (5, 32), (16, 32), (21, 28), (21, 0)]], + "W": [[(0, 0), (4, 32), (10, 16), (16, 32), (21, 0)]], + "Y": [[(0, 0), (10, 16), (21, 0)], [(10, 16), (10, 32)]], + " ": [], + "!": [[(10, 8), (10, 32)], [(10, 0), (10, 4)]], + "?": [[(0, 4), (5, 0), (16, 0), (21, 4), (21, 10), (10, 18), (10, 22)], + [(10, 28), (10, 32)]], + ".": [[(10, 30), (10, 32)]], + ",": [[(8, 30), (10, 34)]], + "-": [[(0, 16), (21, 16)]], +} + +GLYPH_WIDTH = 21 +GLYPH_SPACING = 8 + + +def text_to_strokes( + text: str, + origin_x: float, + origin_y: float, + scale: float = 1.0, +) -> list[si.Line]: + """Render text as a list of Line scene items using Hershey glyphs.""" + lines: list[si.Line] = [] + cursor_x = origin_x + + for ch in text.upper(): + glyph = HERSHEY.get(ch) + if glyph is None: + cursor_x += (GLYPH_WIDTH + GLYPH_SPACING) * scale + continue + + for polyline in glyph: + if len(polyline) < 2: + continue + points = [] + for j, (gx, gy) in enumerate(polyline): + px = cursor_x + gx * scale + py = origin_y + gy * scale + points.append(si.Point( + x=px, y=py, + speed=5 + (j % 10), + direction=90 + (j * 7) % 60, + width=9, + pressure=150 + (j % 30), + )) + lines.append(si.Line( + color=si.PenColor.BLACK, + tool=si.Pen.BALLPOINT_2, + points=points, + thickness_scale=2.0, + starting_length=0.0, + )) + + cursor_x += (GLYPH_WIDTH + GLYPH_SPACING) * scale + + return lines + + +def make_reply_blocks( + text: str, + origin_x: float, + origin_y: float, + scale: float = 1.1, + author_id: int = 1, +) -> list: + """ + Build SceneLineItemBlocks for reply text, anchored to existing Layer 1. + + Layer 1 is always node_id CrdtId(0, 11) in valid .rm v6 files. + No SceneTreeBlock or TreeNodeBlock needed — we inject into the + existing scene tree. + """ + LAYER_1 = CrdtId(0, 11) + blocks = [] + + strokes = text_to_strokes(text, origin_x, origin_y, scale) + for i, line in enumerate(strokes): + blocks.append(SceneLineItemBlock( + parent_id=LAYER_1, + item=CrdtSequenceItem( + item_id=CrdtId(author_id, 200 + i), + left_id=CrdtId(0, 0), + right_id=CrdtId(0, 0), + deleted_length=0, + value=line, + ), + )) + + return blocks + + +def splice_reply(input_path: str, output_path: str, reply_text: str) -> None: + """ + Read an .rm v6 page, append reply strokes, write to output. + + Original blocks are left untouched. Reply blocks are serialized + independently and appended to the raw bytes. + """ + with open(input_path, "rb") as f: + original_data = f.read() + + reply_blocks = make_reply_blocks( + reply_text, + origin_x=-300.0, # v6 origin is top-center; negative = left half + origin_y=700.0, + scale=1.1, + ) + + # Serialize only the reply blocks + reply_buf = BytesIO() + write_blocks(reply_buf, reply_blocks, options={"version": "3.4"}) + reply_bytes = reply_buf.getvalue() + + # Strip the redundant 43-byte header that write_blocks always emits + HEADER_SIZE = 43 + if reply_bytes[:HEADER_SIZE] == original_data[:HEADER_SIZE]: + reply_bytes = reply_bytes[HEADER_SIZE:] + + output_data = original_data + reply_bytes + + with open(output_path, "wb") as f: + f.write(output_data) + + print(f"Wrote {output_path} ({len(reply_blocks)} blocks, " + f"{len(original_data)}+{len(reply_bytes)}={len(output_data)} bytes)") + + +if __name__ == "__main__": + if len(sys.argv) < 2: + print(__doc__) + sys.exit(1) + + input_path = sys.argv[1] + output_path = sys.argv[2] if len(sys.argv) > 2 else input_path + reply = "GRIMOIRE SAYS HELLO" + + splice_reply(input_path, output_path, reply) diff --git a/spec.md b/spec.md new file mode 100644 index 0000000..c313836 --- /dev/null +++ b/spec.md @@ -0,0 +1,255 @@ +# Grimoire — A Handwritten LLM Loop for the reMarkable 2 + +> Write a question by hand on a native notebook page; the device "writes back" an +> answer as ink on the same page, leaving your original strokes untouched. + +**Status:** draft spec +**Target device:** reMarkable 2 (i.MX7, Cortex‑A7, 32‑bit ARM / armhf — confirm for your firmware) +**Approach:** hook `xochitl`, work with native notebooks, inject reply strokes as real `.rm` v6 ink. + +--- + +## 1. Goal + +A closed loop on the device: + +1. User writes a question on a normal notebook page. +2. A deliberate **trigger** fires (sigil glyph / corner tap / hooked toolbar action). +3. The new strokes are captured, recognized, and sent to an LLM. +4. The answer is rendered as **hand‑inked strokes** in whitespace on a dedicated + layer, then composited back to the e‑paper. + +The reply must be **native ink** (persisted in the notebook, survives sync), not a +framebuffer overlay, and must **leave the original writing alone**. + +--- + +## 2. Prior art & source material + +- HN thread that kicked this off — *Show HN: Edsger, a handwritten Clojure REPL for the reMarkable 2*: +- Edsger write‑up (Daniel Janus): +- Author's hand‑written blog with embedded hyperlinks (technique reference): +- Curated ecosystem index: [awesome-reMarkable](https://github.com/rehackable/awesome-remarkable) + +The key takeaway from the thread (per the author's own FAQ, echoed by `jcattle`/`xnorswap`): +Edsger's ~14 s latency is almost entirely **xochitl taking ~12 s to flush the +notebook to disk**. Polling the on‑disk file is the simplest approach and a dead +end for anything that should feel live. Hooking xochitl lets us bypass the flush +entirely by reading and writing the **in‑memory scene**. + +--- + +## 3. The core constraint + +| Path | Latency source | Verdict | +|---|---|---| +| Poll `.rm` on disk | ~12 s xochitl flush | prototype only | +| Read in‑memory scene via hook | none | target | +| Write reply by editing `.rm` + reload | reload cost | prototype only | +| Inject SceneItems in memory + repaint | none | target | + +Everything below is designed to remove the disk round‑trip from the hot path. + +--- + +## 4. Architecture + +``` + ┌─────────────────────────── reMarkable 2 (xochitl, hooked via xovi) ──────────────┐ + │ │ + pen ──► stroke-finalized hook ──► trigger detector ──► capture new strokes (vectors) │ + │ │ │ + │ ▼ │ + │ rasterize strokes → PNG │ + └──────────────────────────────────────────────────┼──────────────────────────────┘ + │ (SSH / Tailscale) + ▼ + ┌──────── server (terebithia) ────────┐ + │ PaddleOCR (local) → question text │ + │ LLM via potluck /v1 → answer text │ + │ Hershey render → reply strokes │ + │ rmscene → SceneItems (v6) │ + └───────────────────┬──────────────────┘ + │ + ┌─────────────────────────────────────────────────────┼──────────────────────────┐ + │ inject SceneItems into in-memory scene (reply layer) ──► repaint hook │ + └──────────────────────────────────────────────────────────────────────────────────┘ +``` + +Recognition + rendering live off‑device; only capture, inject, and repaint are +in‑process hooks. The LLM call routes through **potluck** so the device holds zero +provider keys. + +--- + +## 5. Output path — writing native ink + +The `.rm` v6 ("lines") format is **writable**, so replies can be true ink. + +- **[rmscene](https://github.com/ricklupton/rmscene)** (ricklupton) — reads *and writes* + v6 files. Exposes a `SceneTree` of `SceneItems` grouped into layers, and a writer + that can emulate specific software versions for round‑trip safety. This is the + primary library for generating reply strokes. +- **[rmc](https://github.com/ricklupton/rmc)** ([PyPI](https://pypi.org/project/rmc/)) — + CLI built on rmscene; `rmc -t rm text.md -o text.rm` does text→`.rm`. Text‑box + layout is still rough, so treat as a reference, not the final renderer. +- **[jacob414/rm-files](https://github.com/jacob414/rm-files)** — friendlier wrapper: + `RemarkableNotebook`, named layers, pen/tool selection, filled polygons, scanline + hatching. Assembles device‑like block sequences. Page geometry is **1404×1872 px**. +- **[bsdz/remarkable-layers](https://github.com/bsdz/remarkable-layers)** + ([active fork](https://github.com/YakBarber/remarkable-layers)) — v5 only, but + important: it places text using **Hershey stroke fonts** with pen styles. Hershey + (single‑line) glyphs *are* pen strokes → they convert to `.rm` lines and read as + hand‑inked, not typeset. Port the Hershey→strokes idea onto rmscene for v6. +- Format references: [reMarkable‑kaitai](https://github.com/matomatical/reMarkable-kaitai) + (Kaitai Struct spec), and `ddvk/reader` (credited by rmc for decoding v6). + +**Reply renderer pipeline:** answer text → Hershey/calligraphic single‑line font → +polyline points (+ synthetic pressure/width for the inked look) → rmscene +`SceneItems` on a dedicated **"replies" layer** → inject. Lay out into detected +whitespace so the original strokes are never modified. + +--- + +## 6. Input path — reading what was written + +Because we hook the scene model, the user's strokes are already **vectors** — no +framebuffer scrape needed. + +- Read the new strokes directly from the in‑memory `SceneTree` on trigger. +- Rasterize only the new strokes to a small PNG locally (trivial polyline raster). + +For prototyping the read half *before* the hook exists, stream pen strokes +off‑device using tools from [awesome-reMarkable](https://github.com/rehackable/awesome-remarkable): +`goMarkableStream`, `pipes and paper`, `reStream`. + +--- + +## 7. Recognition + +- **Local OCR:** [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR) on the server. + Fast, no network hop, recommended in the HN thread (`deivid`) for handwriting. +- **LLM:** send recognized text (or the rasterized image, for a vision model) to the + LLM **via potluck's OpenAI‑compatible `/v1` passthrough**. Keeps keys off the device. + +--- + +## 8. Reverse engineering xochitl + +Substrate: **[xovi](https://github.com/asivery/xovi)** (asivery) — function‑hooking / +extension framework built for xochitl; [rm-appload](https://github.com/asivery/rm-appload) +sits on top of it. Installation made easy by +[reManager](https://github.com/rmitchellscott/reManager) + Vellum. + +xochitl is a stripped C++/Qt binary — in Ghidra, match Qt vtables and string xrefs +rather than raw offsets (your GhidraMCP loop fits here). Pin one firmware version; +the well‑known fragility is that symbol addresses move across updates. + +**Functions to locate, in priority order:** + +1. **pen‑up / stroke‑finalized handler** — trigger point + cleanest place to read the + just‑drawn strokes. +2. **document / scene model** — container of pages → layers → lines; used for both + reading existing ink and injecting reply `SceneItems`. +3. **repaint / refresh** — flush injected strokes to e‑paper immediately. +4. **save handler** — optional; only to force persistence on demand. + +### Lower‑level display drivers (fallback / reference only) + +Not needed if we inject native strokes, but useful background and a fallback for an +instant preview overlay while real strokes are generated: + +- [waved](https://github.com/matteodelabre/waved) — C++ direct display driver. +- [dazed](https://github.com/jakubvf/dazed) — Zig rewrite of waved, ships an **SDL + emulator** of the display (develop without flashing). +- [swtcon](https://github.com/yobert/swtcon) — Rust reimplementation. +- [remarkable2-framebuffer](https://github.com/ddvk/remarkable2-framebuffer) — instant + on‑screen state. +- [glider waveform notes](https://gitlab.com/zephray/glider) — why e‑paper needs + multi‑frame waveforms. + +--- + +## 9. Integration depths + +- **Depth 1 — file + reload (no output hook):** read page `.rm`, splice reply layer + with rmscene, write back, force document reload. Entirely server‑side over SSH. + Proves the loop; inherits reload latency. +- **Depth 2 — in‑memory injection (target):** read strokes from the live scene, inject + reply `SceneItems`, call repaint. No disk round‑trip; this is what makes it feel + like magic. + +Build Depth 1 first, then move only the hot path into the hook. + +--- + +## 10. Trigger UX + +Don't guess when the user is done — make it deliberate: + +- a recognizable **sigil glyph** detected in the new strokes, or +- a **corner tap** zone, or +- a **hooked toolbar / menu action** (via rm-appload). + +On trigger: capture new strokes → rasterize → OCR → LLM → Hershey render → inject +into whitespace on the replies layer → repaint. + +--- + +## 11. Milestones + +- **M0 — Loop proof (Depth 1, offline).** SSH in, pull a page `.rm`, render a + hardcoded reply with a Hershey font via rmscene onto a new layer, write back, + reload, confirm it shows as ink that **survives sync**. De‑risks everything. +- **M1 — Renderer.** Hershey glyph → `SceneItem` line with points + pressure on a + named layer; whitespace layout; tune the inked aesthetic. +- **M2 — Recognition.** Rasterize strokes → PaddleOCR → text; wire LLM call through + potluck. +- **M3 — xochitl hooks.** Locate the four functions; read live strokes + inject in + memory + repaint (Depth 2). +- **M4 — Trigger + UX.** Pick the trigger mechanism; end‑to‑end live grimoire. +- **M5 — Polish.** Latency budget, error states, multi‑page, persistence guarantees. + +--- + +## 12. Risks & open questions + +- **Firmware drift** breaks hooks — pin a version, key off Qt symbols/strings. +- **`.rm` writer fidelity** — confirm injected strokes round‑trip cleanly through + sync and the official apps (rmscene emulates versions; verify yours). +- **Hershey→v6 port** — the existing Hershey path is v5; budget time to port. +- **Trigger false positives** — sigil detection vs. ordinary writing. +- **Repaint partial‑refresh** — getting a clean region update without ghosting. +- **Device fragility** — HN reports of weak USB‑C on rM2; newer "Pure" model is more + repairable. + +--- + +## 13. Reference index + +**Project / thread** +- HN thread — +- Edsger post — +- awesome-reMarkable — + +**`.rm` format & stroke generation** +- rmscene — +- rmc — · +- rm-files — +- remarkable-layers (Hershey) — · fork +- reMarkable-kaitai — + +**Hooking & apps** +- xovi — +- rm-appload — +- reManager — + +**Display drivers (fallback)** +- waved — +- dazed — +- swtcon — +- remarkable2-framebuffer — +- glider waveform notes — + +**Recognition** +- PaddleOCR — \ No newline at end of file