diff --git a/.env b/.env deleted file mode 100644 index 527f11d..0000000 --- a/.env +++ /dev/null @@ -1,2 +0,0 @@ -POTLUCK_API_KEY=pot_slate_OSZqyYeJ1363l6ZKdS_1641K -GEMINI_API_KEY=AIzaSyC9WVz9kAEDWozFrk3OBgWbYplZyIglG-c diff --git a/CRUSH.md b/CRUSH.md index b3fb662..596012a 100644 --- a/CRUSH.md +++ b/CRUSH.md @@ -97,8 +97,11 @@ Native C++ extension that runs inside xochitl's process via LD_PRELOAD. ### Build & deploy workflow +The unified build script at `xovi-ext/build.sh` compiles **both** the +extension `.so` and the `uinject` binary, then deploys both to the device. + ```sh -cd xovi-ext/grimoire-injector +cd xovi-ext # First time: creates persistent Docker builder container # Subsequent runs: incremental rebuild (~30s vs ~2min full build) @@ -110,9 +113,62 @@ ssh remarkable 'systemctl reset-failed xochitl && systemctl restart xochitl' The `build.sh` script: 1. Creates/reuses a persistent `grimoire-builder` Docker container with the - `eeems/remarkable-toolchain:latest-rm2` image -2. Runs `qmake6 && make` inside the container (incremental compilation) -3. Copies the `.so` out and SCPs it to the device + `eeems/remarkable-toolchain:latest-rm2` image. Mounts the **whole** + `xovi-ext` tree at `/xovi-ext` (not just `grimoire-injector`) so any + sub-project can compile. +2. Builds the extension: `qmake6 && make` inside the container + (incremental compilation). +3. Builds `uinject`: `$CC -o uinject uinject.c -O2 $CFLAGS $LDFLAGS` after + sourcing the SDK environment-setup. **Must use `$CC` + `$CFLAGS`**, not a + bare `arm-remarkable-linux-gnueabi-gcc`, or the sysroot include paths + (stdio.h etc.) won't resolve. +4. Copies both artifacts out and SCPs them to the device. + +**Deploy gotcha**: SCP to `/home/root/uinject` fails with "dest open ... +Failure" if a uinject process is still running or the file is busy. Always +`ssh remarkable 'killall uinject 2>/dev/null; rm -f /home/root/uinject'` +before the SCP. The build script does this, but if deploying manually, +remember the kill+rm dance. + +**The .so is loaded once at xochitl startup.** Updating it on disk requires +restarting xochitl. The `uinject` binary, by contrast, is exec'd fresh each +call, so a new uinject takes effect immediately — no restart needed. + +### xovi activation survives restart but not reboot + +The xovi tmpfs systemd drop-in (LD_PRELOAD injection) is lost on reboot. +After a reboot, re-run `bash /home/root/xovi/start` (or triple-tap) to +remount it. A plain `systemctl restart xochitl` preserves the tmpfs if it's +already mounted. + +### uinject CLI + +```sh +# Inject pen strokes (speed = ms delay between points, lower = faster) +uinject --speed 3 strokes.json + +# Erase by retracing stroke paths with the eraser tool (6 offset passes) +uinject --speed 2 --erase-strokes strokes.json + +# Erase a rectangular region (rm v6 coords) +uinject --erase x,y,w,h +``` + +Key uinject learnings: +- **Pen input arrives as mouse + touch events, NOT Qt tablet events.** The + reMarkable digitizer surfaces through `/dev/input/event1` as a Wacom + device. uinject writes raw evdev events there directly. +- **Stroke injection is real-time replay** — uinject sleeps `--speed` ms + between every point, so a long reply with thousands of points takes many + seconds. Keep replies short and densify sparingly (see `_densify` + `min_seg_len=300`). Render is instant; the bottleneck is event replay. +- **There is no erase primitive.** Erasing means retracing the ink path with + `BTN_TOOL_RUBBER`. The eraser tip is narrower than expected, so + `--erase-strokes` makes 6 offset passes (center + 4 cardinal ±10px + 1 + diagonal) to fully cover stroke width. +- **JSON parser must handle nested brackets.** The points array is + `[[x,y,...],[x,y,...]]`. A naive "stop at first `]`" only erases the first + point of each stroke. Track bracket depth. ### JSON stroke format (matches clipboard-injector) @@ -130,12 +186,95 @@ The `build.sh` script: Generate with: `python grimoire.py --json output.json "text"` -### Remaining xovi work -- Fix vtable acquisition (hook QImage constructor like framebuffer-spy, or - find existing SceneLineItems in memory) -- Find QQmlEngine to access Clipboard context property -- Trigger paste programmatically after setting Clipboard.items -- Alternative: direct scene graph manipulation (deeper RE needed) +## Closed-loop daemon (`grimoire_loop.py`) + +The full pen-to-reply loop. Watches for pen idle, captures the screen, +sends it to a vision model, and writes the reply back as ink. + +```sh +python grimoire_loop.py [--model MODEL] [--once] +``` + +### Pipeline + +1. **Idle detection** (xovi extension, `PenIdleWatcher`): an event filter on + QGuiApplication watches mouse/touch press+release. A debounce thread + writes `/tmp/grimoire_idle` after the pen has been up for `DEBOUNCE_MS` + (currently 2500ms). The Python daemon polls that file over persistent SSH. +2. **Capture**: trigger `/tmp/grimoire_screenshot`, the extension dumps the + framebuffer to `/tmp/grimoire_fb.raw`, pull it via SCP. +3. **Crop**: drop top 80px (toolbar) and bottom 200px (swirl zone). +4. **Blank check**: skip the API entirely if no real content (see grid + artifact note below). +5. **Vision model**: send the full page image; the model does OCR + + relevance filtering + reply in one call. Returns `[NO_NEW_TEXT]` when + there's nothing new for it. +6. **Position + inject**: find content bottom, render reply below it, SCP the + JSON, run uinject. + +### Persistent SSH + +`SSHSession` uses ControlMaster multiplexing (one connection, reused for all +commands) to kill the ~1s handshake per call. Cleans up stale sockets on +open and uses `ConnectTimeout=10` so it fails fast instead of hanging. + +### The reMarkable display grid artifact (IMPORTANT) + +The captured framebuffer contains a **uniform grid pattern**: isolated +2px-tall rows every ~42px, each with **exactly 68 dark pixels**, pure black +(value 0), spanning the entire screen height. This is a display/refresh +artifact, NOT content. It is indistinguishable from ink by pixel value. + +To find real handwriting, filter it out by: +- Counting dark pixels per row (`< 128` threshold). +- Ignoring rows with **exactly 68** dark pixels (the grid signature). +- Real text rows have variable counts (anything but 68) and form contiguous + clusters many rows tall; the grid is always isolated 2-row pairs. + +This filter is used both for the blank-page check and for finding the +content bottom to position replies. + +### Vision model API + +Currently uses Charm's Hyper gateway: +- Endpoint: `https://hyper.charmcli.dev/v1/chat/completions` +- Auth: `HYPER_API_KEY` from `.env` +- Model: `kimi-k2.6-vercel` +- OpenAI-compatible format. **Roles must be `user`/`assistant`**, not + `model` (that's Gemini's format and Hyper rejects it with HTTP 400). +- Images sent as `image_url` with `data:image/png;base64,...` URLs. + +Conversation history is kept as `(role, text)` tuples and replayed each +turn for context. It resets on a real page change but NOT when our own +injection changes the framebuffer hash (detected via `last_inject_y`). + +### Thinking indicator + +While capture + inference run, a background thread draws a small curly +swirl in the bottom-left corner (rm coords ~x=-580, y=1750), then erases and +redraws it in a loop (1s hold after draw, 3s pause after erase). On +completion it does a final erase. The animation thread is stopped on every +exit path (success, skip, blank, error) via an `Event`, and the main thread +joins with a generous 15s timeout so the final erase finishes cleanly — +**a short join timeout abandons the thread mid-erase and leaves ink behind.** + +## Hard-won iteration loop + +When working on grimoire, this is the tight loop that works: + +1. **Python changes** take effect on the next `grimoire_loop.py` run. No + deploy needed. +2. **uinject changes**: `cd xovi-ext && bash build.sh` then the kill+rm+scp + dance. Effective immediately (exec'd per call). +3. **Extension (.so) changes**: `bash build.sh`, then restart xochitl. + Slowest loop — avoid unless touching idle detection or framebuffer hooks. +4. **Debugging the extension**: stderr is swallowed by xochitl. Write to a + file instead (`/tmp/grimoire_ext.log`) via a `grimoire_log()` helper. + Read it with `ssh remarkable 'cat /tmp/grimoire_ext.log'`. +5. **Inspecting captures**: pull `/tmp/grimoire_fb.raw`, load as + `Image.frombytes("RGBA", (1404, 1872), raw)`, annotate with PIL, save a + PNG, and View it. This is how the grid artifact was found — always look + at the actual pixels before trusting a heuristic. ## Script usage (`grimoire.py`) @@ -143,9 +282,14 @@ Generate with: `python grimoire.py --json output.json "text"` # File-based injection (creates new layer in .rm file) python grimoire.py input.rm output.rm "reply text" -# JSON export for xovi injector -python grimoire.py --json /tmp/grimoire_strokes.json "reply text" +# JSON export for xovi injector (--y sets start Y in device coords) +python grimoire.py --json /tmp/grimoire_strokes.json --y 400 "reply text" # Preview (renders SVG for visual tuning) python grimoire.py --preview "text" ``` + +`grimoire.py` is also importable as a library: +`text_to_strokes(text, x, y)` returns `si.Line` objects; +`strokes_to_json(strokes, path)` writes the uinject JSON. The loop uses it +this way for the thinking indicator instead of shelling out. diff --git a/grimoire.py b/grimoire.py index f020e10..66a2a70 100644 --- a/grimoire.py +++ b/grimoire.py @@ -171,7 +171,7 @@ def text_to_strokes( # points for the device renderer, which collapses or drops # 2-point strokes. Insert midpoints until every stroke has # at least a few samples per segment. - dense = _densify(polyline, min_seg_len=120.0) + dense = _densify(polyline, min_seg_len=300.0) points = [] for j, (gx, gy) in enumerate(dense): sx = gx * scale diff --git a/grimoire_loop.py b/grimoire_loop.py index f78a8a7..71c4213 100644 --- a/grimoire_loop.py +++ b/grimoire_loop.py @@ -18,6 +18,7 @@ import argparse import base64 import hashlib import io +import json import os import re import subprocess @@ -32,6 +33,32 @@ from dotenv import load_dotenv load_dotenv() +# Only one uinject job may touch /dev/input/event1 at a time. Two +# concurrent pen-event streams interleave into garbage strokes, so every +# uinject invocation must hold this lock (see run_inject). +import threading as _threading + +_inject_lock = _threading.Lock() +_inject_verbose = False # set from args.verbose in main() + + +def run_inject(ssh, args, timeout): + """Run a uinject command, serialized against all other pen jobs. + + `args` is the part after the binary, e.g. "--speed 2 --erase-strokes + /tmp/x.json". Blocks until any in-flight injection finishes so we + never drive the digitizer from two processes at once. + Passes -v to uinject when the daemon was started with -v/--verbose. + """ + v_flag = " -v" if _inject_verbose else "" + with _inject_lock: + result = ssh.run(f"/home/root/uinject{v_flag} {args}", timeout=timeout) + if result.stderr: + for line in result.stderr.strip().split('\n'): + print(f" {line}") + return result + + # ─── Persistent SSH session ──────────────────────────────────────── class SSHSession: @@ -48,6 +75,17 @@ class SSHSession: def _open(self): """Establish master connection.""" + # Clean up any stale socket from a previous run + if os.path.exists(self.socket): + subprocess.run( + ["ssh", "-o", f"ControlPath={self.socket}", "-O", "exit", self.host], + capture_output=True, timeout=5, + ) + try: + os.unlink(self.socket) + except OSError: + pass + cmd = [ "ssh", "-fN", "-o", "ControlMaster=yes", @@ -55,9 +93,10 @@ class SSHSession: "-o", "ControlPersist=600", "-o", "ServerAliveInterval=30", "-o", "ServerAliveCountMax=3", + "-o", "ConnectTimeout=10", self.host, ] - result = subprocess.run(cmd, capture_output=True, timeout=15) + result = subprocess.run(cmd, capture_output=True, timeout=20) if result.returncode != 0: raise RuntimeError( f"SSH master failed: {result.stderr.decode().strip()}" @@ -136,8 +175,8 @@ def capture_framebuffer(ssh): img = Image.frombytes("RGBA", (1404, 1872), raw) img_rgb = img.convert("RGB") - # Crop toolbar (top 80px) - img_rgb = img_rgb.crop((0, 80, img_rgb.width, img_rgb.height)) + # Crop toolbar (top 80px) and swirl zone (bottom 200px) + img_rgb = img_rgb.crop((0, 80, img_rgb.width, img_rgb.height - 200)) return img_rgb @@ -171,61 +210,63 @@ def _image_to_base64(image): return base64.b64encode(buf.getvalue()).decode("utf-8") -def ask_gemini(image, conversation_history=None, model="gemini-2.5-flash-lite"): - """Send cropped handwriting image to Gemini, return response text. +def ask_gemini(image, conversation_history=None, model="kimi-k2.6-vercel"): + """Send page image to Hyper API, return response text. - Combines OCR and LLM into a single API call — Gemini reads the + Combines OCR and LLM into a single API call — the model reads the handwriting directly from the image. Includes conversation history for context across multiple exchanges. """ - api_key = os.environ.get("GEMINI_API_KEY") + api_key = os.environ.get("HYPER_API_KEY") if not api_key: - raise RuntimeError("GEMINI_API_KEY not set in .env") + raise RuntimeError("HYPER_API_KEY not set in .env") img_b64 = _image_to_base64(image) - # Build contents array with history + current turn - contents = [] + # Build messages array with history + current turn + messages = [ + {"role": "system", "content": SYSTEM_PROMPT}, + ] if conversation_history: for role, text in conversation_history: - contents.append({ + messages.append({ "role": role, - "parts": [{"text": text}], + "content": text, }) - # Current turn: full page image - contents.append({ + # Current turn: image + prompt + messages.append({ "role": "user", - "parts": [ - {"text": "Here is the current page:"}, + "content": [ + {"type": "text", "text": "Read this handwritten text and respond:"}, { - "inline_data": { - "mime_type": "image/png", - "data": img_b64, - } + "type": "image_url", + "image_url": { + "url": f"data:image/png;base64,{img_b64}", + }, }, ], }) r = requests.post( - f"https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent", - headers={"Content-Type": "application/json"}, - params={"key": api_key}, + "https://hyper.charmcli.dev/v1/chat/completions", + headers={ + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + }, json={ - "system_instruction": { - "parts": [{"text": SYSTEM_PROMPT}], - }, - "contents": contents, + "model": model, + "messages": messages, }, timeout=60, ) if r.status_code != 200: - raise RuntimeError(f"Gemini API error {r.status_code}: {r.text[:200]}") + raise RuntimeError(f"Hyper API error {r.status_code}: {r.text[:200]}") data = r.json() try: - content = data["candidates"][0]["content"]["parts"][0]["text"].strip() + content = data["choices"][0]["message"]["content"].strip() except (KeyError, IndexError): content = "..." @@ -264,8 +305,12 @@ def _strip_formatting(text): # ─── Render + Inject ────────────────────────────────────────────── -def render_and_inject(ssh, text, reply_y=None): - """Render text as strokes and inject via uinject.""" +def render_and_inject(ssh, text, reply_y=None, speed_ms=3): + """Render text as strokes and inject via uinject. + + reply_y: pixel Y coordinate to start rendering at (in device coords). + speed_ms: delay between points in milliseconds (lower = faster). + """ json_path = "/tmp/grimoire_reply.json" cmd = [".venv/bin/python3", "grimoire.py", "--json", json_path] @@ -282,7 +327,11 @@ def render_and_inject(ssh, text, reply_y=None): if r.returncode != 0: raise RuntimeError(f"SCP failed: {r.stderr.decode().strip()}") - result = ssh.run("/home/root/uinject /tmp/grimoire_strokes.json", timeout=60) + result = run_inject( + ssh, + f"--speed {speed_ms} /tmp/grimoire_strokes.json", + timeout=120, + ) out = result.stdout.strip() if result.stdout else "" err = result.stderr.strip() if result.stderr else "" if result.returncode != 0: @@ -293,6 +342,33 @@ def render_and_inject(ssh, text, reply_y=None): # ─── Idle watching ──────────────────────────────────────────────── + +def stop_animation(anim_ssh, anim_stop, anim_thread): + """Halt the thinking animation and guarantee the swirl is erased. + + Uses the animation thread's own SSH session (not the main one) so + the final erase doesn't contend with capture/SCP on a shared + ControlMaster socket. The _inject_lock still serializes against + any in-flight pen job from the thread. + """ + anim_stop.set() + print(f" [anim] Waiting for thread to finish...") + anim_thread.join(timeout=20) + alive = anim_thread.is_alive() + print(f" [anim] Thread {'still alive (timeout!)' if alive else 'exited'}") + try: + print(f" [anim] Running final erase...") + result = run_inject( + anim_ssh, + "--speed 20 --erase-strokes /tmp/grimoire_thinking_live.json", + timeout=30, + ) + print(f" [anim] Final erase rc={result.returncode}") + except Exception as e: + print(f" [anim] final erase failed: {e}") + finally: + anim_ssh.close() + def watch_for_idle(ssh, last_hash): """Block until /tmp/grimoire_idle changes on device. @@ -392,17 +468,26 @@ def _find_new_region(current_img, last_img, ignore_below_y=None): def main(): parser = argparse.ArgumentParser(description="Grimoire continuous loop") parser.add_argument( - "--model", type=str, default="gemini-2.5-flash-lite", - help="Gemini model", + "--model", type=str, default="kimi-k2.6-vercel", + help="Hyper API model", ) parser.add_argument( "--once", action="store_true", help="Run one cycle then exit (for testing)", ) + parser.add_argument( + "-v", "--verbose", action="store_true", + help="Pass -v to uinject for progress logging", + ) args = parser.parse_args() + global _inject_verbose + _inject_verbose = args.verbose + print("=== Grimoire Loop ===") print(f"Model: {args.model}") + if _inject_verbose: + print("Verbose: uinject -v enabled") print("Waiting for pen idle signal...") print() @@ -424,6 +509,87 @@ def main(): t0 = time.time() print(f"[{_ts()}] Pen idle detected!") + # Animate thinking indicator: draw → erase → draw → erase → ... + # Runs in background while we do capture + API work + import threading + + def _thinking_animation(ssh_session, stop_event): + """Draw and erase the thinking curl repeatedly.""" + import math + + def _make_thinking_json(): + """Generate swirl as strokes JSON.""" + # Swirl centered at y=1750 + swirl_pts = [] + for i in range(40): + t = i / 39.0 + angle = t * math.pi * 3 + r = 15 + t * 25 + x = -580 + r * math.cos(angle) + y = 1750 + r * math.sin(angle) * 0.6 + swirl_pts.append([x, y, 6, 11, 90, 170]) + swirl = { + 'points': swirl_pts, + 'rgba': 4278190080, 'color': 0, + 'bounds': [min(p[0] for p in swirl_pts), min(p[1] for p in swirl_pts), + max(p[0] for p in swirl_pts) - min(p[0] for p in swirl_pts), + max(p[1] for p in swirl_pts) - min(p[1] for p in swirl_pts)], + 'tool': 15, 'maskScale': 2.0, 'thickness': 2.0, + } + out_path = "/tmp/grimoire_thinking_live.json" + with open(out_path, 'w') as f: + json.dump([swirl], f) + return out_path + + drawing = False + device_json = "/tmp/grimoire_thinking_live.json" + thinking_json = _make_thinking_json() + ssh_session.scp_to(thinking_json, device_json) + print(f" [anim] Animation thread started") + + while not stop_event.is_set(): + try: + action = "erase" if drawing else "draw" + print(f" [anim] Starting {action}...") + if drawing: + run_inject( + ssh_session, + f"--speed 20 --erase-strokes {device_json}", + timeout=30, + ) + else: + run_inject( + ssh_session, + f"--speed 32 {device_json}", + timeout=30, + ) + print(f" [anim] {action} done") + drawing = not drawing + except Exception as e: + print(f" [anim] ERROR: {e}") + # 3s pause after erase; 1s hold after draw + wait_time = 3.0 if drawing else 1.0 + print(f" [anim] Waiting {wait_time}s...") + if stop_event.wait(wait_time): + print(f" [anim] Stop signaled during wait, exiting loop") + break + print(f" [anim] Thread loop exited (drawing={drawing})") + # The caller performs the final, guaranteed erase in the + # main thread (see stop_animation) so it can't be cut short + # by a join timeout while a draw/erase is still in flight. + + # The animation thread gets its own SSH connection so pen + # injection never shares a ControlMaster socket with the + # main thread's capture/SCP traffic. Sharing caused the + # erase to get starved mid-stream when both threads hit the + # multiplexed socket at once. + anim_ssh = SSHSession("remarkable") + anim_stop = threading.Event() + anim_thread = threading.Thread( + target=_thinking_animation, args=(anim_ssh, anim_stop), daemon=True + ) + anim_thread.start() + try: # Capture print(f" [{_ts()}] Capturing...") @@ -433,6 +599,7 @@ def main(): # Quick hash check — skip if framebuffer unchanged fb_hash = _framebuffer_hash(img) if fb_hash == last_fb_hash: + stop_animation(anim_ssh, anim_stop, anim_thread) print(f" [{_ts()}] Framebuffer unchanged, skipping.") print() continue @@ -452,21 +619,33 @@ def main(): last_inject_y = None last_page_hash = fb_hash - # Send full page to Gemini — it handles OCR + relevance filtering - print(f" [{_ts()}] Asking Gemini ({len(conversation)} history turns)...") + # Quick content check — skip blank pages before hitting API + import numpy as np + gray = np.array(img.convert('L')) + dark_per_row = np.sum(gray < 128, axis=1) + non_grid_rows = np.where((dark_per_row > 5) & (dark_per_row != 68))[0] + if len(non_grid_rows) == 0: + stop_animation(anim_ssh, anim_stop, anim_thread) + print(f" [{_ts()}] Blank page, skipping API call.") + print() + continue + + # Send full page to model — it handles OCR + relevance filtering + print(f" [{_ts()}] Asking {args.model} ({len(conversation)} history turns)...") answer = ask_gemini(img, conversation, args.model) print(f" [{_ts()}] Reply ({_elapsed(t0)}): {answer[:80]}{'...' if len(answer) > 80 else ''}") # Skip if Gemini says no new text if "[NO_NEW_TEXT]" in answer: + stop_animation(anim_ssh, anim_stop, anim_thread) print(f" [{_ts()}] No new text detected by Gemini, skipping.") last_img = img print() continue # Add to conversation history - conversation.append(("user", f"[handwritten question]")) - conversation.append(("model", answer)) + conversation.append(("user", "[handwritten question]")) + conversation.append(("assistant", answer)) # Position reply below existing content. # The reMarkable display has a uniform grid artifact: @@ -509,6 +688,9 @@ def main(): # Render + Inject print(f" [{_ts()}] Rendering + injecting (reply_y={reply_y})...") + # Stop thinking animation and fully erase the swirl before + # we draw the reply, so the two never overlap on screen. + stop_animation(anim_ssh, anim_stop, anim_thread) render_and_inject(ssh, answer, reply_y=reply_y) last_inject_y = reply_y print(f" [{_ts()}] Injected ({_elapsed(t0)})") @@ -517,6 +699,7 @@ def main(): print(f" [{_ts()}] Done in {elapsed:.1f}s") except Exception as e: + stop_animation(anim_ssh, anim_stop, anim_thread) print(f" ERROR: {e}") print() diff --git a/xovi-ext/build.sh b/xovi-ext/build.sh new file mode 100755 index 0000000..0ae93db --- /dev/null +++ b/xovi-ext/build.sh @@ -0,0 +1,52 @@ +#!/bin/bash +# Build all grimoire xovi components: extension .so + uinject binary +set -e +cd "$(dirname "$0")" + +CONTAINER="grimoire-builder" +EXT_DIR="$(pwd)/grimoire-injector" +UINJECT_DIR="$(pwd)/uinject" + +# Create persistent builder container if needed +if ! docker inspect $CONTAINER &>/dev/null; then + echo "Creating builder container..." + docker create --platform linux/amd64 --name $CONTAINER \ + -v "$(pwd):/xovi-ext" \ + -w /xovi-ext/grimoire-injector \ + eeems/remarkable-toolchain:latest-rm2 \ + bash -c "git clone https://github.com/asivery/xovi /tmp/xovi 2>/dev/null; tail -f /dev/null" + docker start $CONTAINER + sleep 3 +fi + +# Build extension .so +echo "=== Building grimoire-injector.so ===" +docker exec $CONTAINER bash -c ' +export XOVI_REPO=/tmp/xovi +. /opt/codex/*/*/environment-setup-* +cd /xovi-ext/grimoire-injector +qmake6 2>/dev/null +make 2>&1 +' + +echo "Copying .so..." +docker cp $CONTAINER:/xovi-ext/grimoire-injector/grimoire-injector.so "$EXT_DIR/" + +# Build uinject +echo "=== Building uinject ===" +docker exec $CONTAINER bash -c ' +. /opt/codex/*/*/environment-setup-* +cd /xovi-ext/uinject +$CC -o uinject uinject.c -O2 $CFLAGS $LDFLAGS +echo "uinject built OK" +' + +echo "Copying uinject..." +docker cp $CONTAINER:/xovi-ext/uinject/uinject "$UINJECT_DIR/" + +# Deploy both to device +echo "=== Deploying to device ===" +scp "$EXT_DIR/grimoire-injector.so" remarkable:/home/root/xovi/extensions.d/ +scp "$UINJECT_DIR/uinject" remarkable:/home/root/uinject + +echo "Done! Restart xochitl to load new extension." diff --git a/xovi-ext/grimoire-injector/GrimoireInjector.hpp b/xovi-ext/grimoire-injector/GrimoireInjector.hpp index dafd705..2228ef4 100644 --- a/xovi-ext/grimoire-injector/GrimoireInjector.hpp +++ b/xovi-ext/grimoire-injector/GrimoireInjector.hpp @@ -50,7 +50,7 @@ private: std::atomic m_running{false}; std::atomic m_lastLiftMs{0}; - static constexpr int DEBOUNCE_MS = 4000; + static constexpr int DEBOUNCE_MS = 2500; static constexpr const char *IDLE_PATH = "/tmp/grimoire_idle"; static void *debounceThreadFunc(void *arg); diff --git a/xovi-ext/uinject/uinject.c b/xovi-ext/uinject/uinject.c index d1ae119..92bc418 100644 --- a/xovi-ext/uinject/uinject.c +++ b/xovi-ext/uinject/uinject.c @@ -1,4 +1,4 @@ -/* uinject.c — Inject pen strokes via evdev for reMarkable 2 */ +/* uinject.c — Inject pen strokes or erase along paths via evdev for reMarkable 2 */ #include #include #include @@ -8,6 +8,11 @@ #include #include +static int g_delay_us = 5000; /* default 5ms between points */ +static int g_verbose = 0; /* -v enables info logging to stderr */ + +#define vlog(...) do { if (g_verbose) fprintf(stderr, __VA_ARGS__); } while (0) + static void emit_event(int fd, int type, int code, int value) { struct input_event ev; memset(&ev, 0, sizeof(ev)); @@ -22,38 +27,202 @@ static void emit_syn(int fd) { } /* Convert .rm v6 coordinates to Wacom digitizer coordinates */ -/* RM v6: x[-702,701] y[0,1871], screen portrait 1404x1872 */ -/* Wacom digitizer: x[0,20966] y[0,15725], landscape orientation */ -/* 90° CCW rotation: wacom_x = (screen_height - screen_y) * scale, wacom_y = screen_x * scale */ static void rm_to_wacom(float rm_x, float rm_y, int *wx, int *wy) { - float screen_x = rm_x + 702.0f; /* 0..1403 */ - float screen_y = rm_y; /* 0..1871 */ - /* Rotate 90° CCW and scale to Wacom range */ + float screen_x = rm_x + 702.0f; + float screen_y = rm_y; *wx = (int)((1872.0f - screen_y) * 20966.0f / 1872.0f); *wy = (int)(screen_x * 15725.0f / 1404.0f); } static int rm_pressure_to_wacom(int rm_pressure) { - /* RM pressure: 0-255, Wacom: 0-4095 */ return rm_pressure * 4095 / 255; } +/* Erase by replaying stroke paths with the eraser tool. + * Makes multiple passes with slight offsets for full coverage. */ +static int do_erase_strokes(int fd, const char *json, long fsize) { + vlog("[uinject] Erasing along stroke paths\n"); + + /* Offset patterns for multi-pass erase (x,y in rm coords). + * The eraser footprint is much wider than the swirl's pen width, + * so three tight passes give full coverage without the extra time + * (and cutoff risk) of a six-pass sweep. */ + static const float offsets[][2] = { + {0, 0}, {-6, 0}, {6, 0}, + }; + int num_passes = sizeof(offsets) / sizeof(offsets[0]); + + int total_emitted = 0; + + for (int pass = 0; pass < num_passes; pass++) { + float ox = offsets[pass][0]; + float oy = offsets[pass][1]; + char *p = json; /* Reset to start of JSON for each pass */ + vlog("[uinject] Pass %d/%d\n", pass+1, num_passes); + int pass_emitted = 0; + + while ((p = strstr(p, "\"points\"")) != NULL) { + p = strchr(p, '['); + if (!p) break; + p++; + + emit_event(fd, EV_KEY, BTN_TOOL_RUBBER, 1); + emit_event(fd, EV_KEY, BTN_TOUCH, 1); + + int bracket_depth = 1; /* we're inside the outer [ */ + float last_x = 1e9f, last_y = 1e9f; + const float min_dist_sq = 15.0f * 15.0f; + + while (*p && bracket_depth > 0) { + if (*p == '[') { + /* Start of a point: [x,y,speed,width,direction,pressure] */ + float x, y; + int speed, width, direction, pressure; + if (sscanf(p, "[%f,%f,%d,%d,%d,%d]", &x, &y, &speed, &width, &direction, &pressure) == 6) { + float dx = (x + ox) - last_x; + float dy = (y + oy) - last_y; + if (dx*dx + dy*dy >= min_dist_sq || pass_emitted == 0) { + int wx, wy; + rm_to_wacom(x + ox, y + oy, &wx, &wy); + + emit_event(fd, EV_ABS, ABS_X, wx); + emit_event(fd, EV_ABS, ABS_Y, wy); + emit_event(fd, EV_ABS, ABS_PRESSURE, 4095); + emit_event(fd, EV_ABS, ABS_DISTANCE, 0); + emit_syn(fd); + + last_x = x + ox; + last_y = y + oy; + pass_emitted++; + total_emitted++; + usleep(5000); /* 5ms between points */ + } + } + /* Skip past this inner [...] */ + char *close = strchr(p + 1, ']'); + if (close) p = close + 1; + else break; + } else if (*p == ']') { + bracket_depth--; + if (bracket_depth <= 0) break; + p++; + } else { + p++; + } + } + + emit_event(fd, EV_ABS, ABS_PRESSURE, 0); + emit_event(fd, EV_ABS, ABS_DISTANCE, 86); + emit_syn(fd); + emit_event(fd, EV_KEY, BTN_TOUCH, 0); + emit_event(fd, EV_KEY, BTN_TOOL_RUBBER, 0); + emit_syn(fd); + + usleep(10000); /* 10ms between strokes */ + } + vlog("[uinject] Pass %d emitted %d points\n", pass+1, pass_emitted); + } + + vlog("[uinject] Erased %d passes (total emitted=%d)\n", num_passes, total_emitted); + return 0; +} + int main(int argc, char **argv) { if (argc < 2) { - fprintf(stderr, "Usage: %s \n", argv[0]); + fprintf(stderr, "Usage: %s [--speed MS] \n", argv[0]); + fprintf(stderr, " %s [--speed MS] --erase-strokes \n", argv[0]); + fprintf(stderr, " %s --erase x,y,w,h\n", argv[0]); + fprintf(stderr, " --speed MS Delay between points in ms (default: 5)\n"); + fprintf(stderr, " -v Verbose: print progress info to stderr\n"); return 1; } - /* Write directly to the real Wacom digitizer device */ + /* Parse flags anywhere in args */ + for (int i = 1; i < argc; i++) { + if (strcmp(argv[i], "--speed") == 0 && i + 1 < argc) { + g_delay_us = atoi(argv[i + 1]) * 1000; + if (g_delay_us < 0) g_delay_us = 0; + for (int j = i; j + 2 < argc; j++) argv[j] = argv[j + 2]; + argc -= 2; i--; + } else if (strcmp(argv[i], "-v") == 0 || strcmp(argv[i], "--verbose") == 0) { + g_verbose = 1; + for (int j = i; j + 1 < argc; j++) argv[j] = argv[j + 1]; + argc -= 1; i--; + } + } + const char *dev_path = "/dev/input/event1"; int fd = open(dev_path, O_WRONLY); if (fd < 0) { perror("open /dev/input/event1"); return 1; } - fprintf(stderr, "[uinject] Writing to %s\n", dev_path); + vlog("[uinject] Writing to %s (delay=%dus)\n", dev_path, g_delay_us); + + /* Region erase mode */ + if (strcmp(argv[1], "--erase") == 0 && argc >= 3) { + float ex, ey, ew, eh; + if (sscanf(argv[2], "%f,%f,%f,%f", &ex, &ey, &ew, &eh) != 4) { + fprintf(stderr, "[uinject] Bad erase region format. Use: x,y,w,h\n"); + close(fd); + return 1; + } + vlog("[uinject] Erasing region: x=%.0f y=%.0f w=%.0f h=%.0f\n", ex, ey, ew, eh); + + float step = 8.0f; + int wx, wy; + emit_event(fd, EV_KEY, BTN_TOOL_RUBBER, 1); + emit_event(fd, EV_KEY, BTN_TOUCH, 1); + + for (float row = ey; row <= ey + eh; row += step) { + rm_to_wacom(ex, row, &wx, &wy); + emit_event(fd, EV_ABS, ABS_X, wx); + emit_event(fd, EV_ABS, ABS_Y, wy); + emit_event(fd, EV_ABS, ABS_PRESSURE, 4095); + emit_event(fd, EV_ABS, ABS_DISTANCE, 0); + emit_syn(fd); + + rm_to_wacom(ex + ew, row, &wx, &wy); + emit_event(fd, EV_ABS, ABS_X, wx); + emit_event(fd, EV_ABS, ABS_Y, wy); + emit_event(fd, EV_ABS, ABS_PRESSURE, 4095); + emit_event(fd, EV_ABS, ABS_DISTANCE, 0); + emit_syn(fd); + + usleep(g_delay_us); + } + + emit_event(fd, EV_ABS, ABS_PRESSURE, 0); + emit_event(fd, EV_ABS, ABS_DISTANCE, 86); + emit_syn(fd); + emit_event(fd, EV_KEY, BTN_TOUCH, 0); + emit_event(fd, EV_KEY, BTN_TOOL_RUBBER, 0); + emit_syn(fd); + + vlog("[uinject] Region erase done\n"); + close(fd); + return 0; + } + + /* Stroke-based erase mode */ + if (strcmp(argv[1], "--erase-strokes") == 0 && argc >= 3) { + FILE *fp = fopen(argv[2], "r"); + if (!fp) { perror("fopen"); close(fd); return 1; } + fseek(fp, 0, SEEK_END); + long fsize = ftell(fp); + fseek(fp, 0, SEEK_SET); + char *json = malloc(fsize + 1); + fread(json, 1, fsize, fp); + json[fsize] = '\0'; + fclose(fp); + + int ret = do_erase_strokes(fd, json, fsize); + free(json); + close(fd); + return ret; + } - /* Read JSON file - simple parser for our known format */ + /* Normal pen injection mode */ FILE *fp = fopen(argv[1], "r"); if (!fp) { perror("fopen"); @@ -61,7 +230,6 @@ int main(int argc, char **argv) { return 1; } - /* Read entire file */ fseek(fp, 0, SEEK_END); long fsize = ftell(fp); fseek(fp, 0, SEEK_SET); @@ -70,25 +238,21 @@ int main(int argc, char **argv) { json[fsize] = '\0'; fclose(fp); - fprintf(stderr, "[uinject] Loaded %ld bytes from %s\n", fsize, argv[1]); + vlog("[uinject] Loaded %ld bytes from %s\n", fsize, argv[1]); - /* Simple JSON parsing - find each stroke's points array */ - /* Format: [{"points":[[x,y,speed,width,direction,pressure],...], ...}, ...] */ char *p = json; int stroke_count = 0; while ((p = strstr(p, "\"points\"")) != NULL) { - p = strchr(p, '['); /* Find start of points array */ + p = strchr(p, '['); if (!p) break; - p++; /* Skip [ */ + p++; - /* Pen down */ emit_event(fd, EV_KEY, BTN_TOOL_PEN, 1); emit_event(fd, EV_KEY, BTN_TOUCH, 1); int point_count = 0; while (*p && *p != ']') { - /* Parse [x, y, speed, width, direction, pressure] */ float x, y; int speed, width, direction, pressure; if (sscanf(p, "[%f,%f,%d,%d,%d,%d]", &x, &y, &speed, &width, &direction, &pressure) == 6) { @@ -103,18 +267,14 @@ int main(int argc, char **argv) { emit_syn(fd); point_count++; - /* Small delay between points (~5ms) */ - usleep(5000); + usleep(g_delay_us); } - /* Advance to next point */ p = strchr(p, ']'); if (p) p++; - /* Skip comma and whitespace */ while (*p && (*p == ',' || *p == ' ' || *p == '\n' || *p == '\r')) p++; } - /* Pen up */ emit_event(fd, EV_ABS, ABS_PRESSURE, 0); emit_event(fd, EV_ABS, ABS_DISTANCE, 86); emit_syn(fd); @@ -123,13 +283,12 @@ int main(int argc, char **argv) { emit_syn(fd); stroke_count++; - fprintf(stderr, "[uinject] Stroke %d: %d points\n", stroke_count, point_count); + vlog("[uinject] Stroke %d: %d points\n", stroke_count, point_count); - /* Small delay between strokes */ usleep(50000); } - fprintf(stderr, "[uinject] Done: %d strokes injected\n", stroke_count); + vlog("[uinject] Done: %d strokes injected\n", stroke_count); free(json); close(fd);