From b6f94766bf38bb32af3fdf1e6235d744247f8b32 Mon Sep 17 00:00:00 2001 From: Kieran Klukas Date: Fri, 19 Jun 2026 21:11:09 -0400 Subject: [PATCH] feat: update crush.md --- CRUSH.md | 103 ++++++++++++++++++++++++++++++------- xovi-ext/glossad/glossad.c | 2 +- 2 files changed, 84 insertions(+), 21 deletions(-) diff --git a/CRUSH.md b/CRUSH.md index 06aa4a4..2429d23 100644 --- a/CRUSH.md +++ b/CRUSH.md @@ -186,10 +186,90 @@ Key uinject learnings: Generate with: `python glossa.py --json output.json "text"` +## On-device daemon (`glossad.c`) + +Single C binary that runs on the reMarkable and replaces the Python loop +for capture + inference + injection. Lives at `xovi-ext/glossad/glossad.c`. + +### What it does +- Watches `/tmp/glossa_armed` for toggle state. +- Accepts commands via `/tmp/glossa_cmd` (JSON: draw, erase, swipe). +- Captures framebuffer, crops, despeckles, masks known content, sends to + vision API, renders reply with built-in font renderer, injects via evdev. +- All in one process. No socket, no separate daemon. + +### Image processing pipeline (in `glossad.c`) + +1. **Capture**: trigger `/tmp/glossa_screenshot`, read raw RGBA framebuffer. +2. **Crop**: drop top 80px (toolbar) and bottom 200px (swirl/safezone). +3. **Despeckle** (`despeckle_crop`): two passes on the grayscale crop. + - Pass 1: isolated pixel removal. Dark pixels with fewer than 2 dark + 8-neighbors are e-ink noise; blank them. + - Pass 2: connected-component analysis via flood fill. Components with + fewer than 4 pixels are killed. Additionally, components of exactly + 4 pixels fitting in a 2x2 bounding box are killed (this is the grid + dot signature: paired dots on consecutive rows). +4. **Influence mask** (`apply_influence_mask`): pixels in known content + regions (ornaments, previous replies) are blanked to white before the + PNG is saved for the AI. Prevents the model from re-reading its own + output or getting confused by decorative elements. +5. **New content detection** (`detect_new_content`): scans the crop for + dark pixels outside the influence mask. Returns whether new handwriting + exists and the lowest row with new content (for reply positioning). + Replaces the old separate `page_is_empty` + `find_content_bottom`. + +### Influence map + +A `CROP_HEIGHT x FB_WIDTH` uint8 array (`g_influence`) tracking known +content regions. Zeroed at startup, cleared on page turn. + +- **`mark_stroke_region()`**: stamps a Gaussian blob (radius 30px, sigma + 15.0) around each stroke point into the map. Points are converted from + rm coordinates (origin top-center) to crop-space. +- **`mark_strokes_from_json()`**: parses uinject-format JSON and marks all + stroke points. Called after every injection so the next capture knows + where our ink lives. +- **`apply_influence_mask()`**: blanks pixels above threshold (128) to + white in the grayscale crop before saving the AI PNG. +- **Constants**: `INFLUENCE_RADIUS=30`, `INFLUENCE_SIGMA=15.0`, + `INFLUENCE_THRESH=128`, `DARK_THRESHOLD=128`. + +### Grid artifact filtering (IMPORTANT) + +The reMarkable framebuffer contains a uniform grid pattern: isolated +2px-tall rows every ~42px, each with exactly 68 dark pixels. This is a +display/refresh artifact, NOT content. It is indistinguishable from ink +by pixel value alone. + +The old approach (counting dark pixels per row, filtering rows with +exactly 68) has been replaced by connected-component analysis in +`despeckle_crop()`. The grid dots survive as tiny 4-pixel components in +2x2 bounding boxes and are killed by the specific filter: +`count == 4 && comp_w <= 2 && comp_h <= 2`. + +This works because real handwriting forms large connected blobs while +grid artifacts are always tiny isolated clusters. Verified: 1286 grid +dots killed, 22 real handwriting components preserved on a test page. + +### Build & deploy + +`glossad` is built by `xovi-ext/build.sh` alongside the extension and +uinject. It links against libpng and OpenSSL (for HTTPS API calls). + +```sh +cd xovi-ext && bash build.sh +# Deploys glossad binary to device. Restart to pick up changes: +ssh remarkable 'systemctl restart glossad' +``` + +Local clangd diagnostics show errors for `linux/input.h`, `png.h`, etc. +These are expected: the code cross-compiles against the reMarkable +toolchain sysroot, not local macOS headers. Ignore them. + ## Closed-loop daemon (`glossa_loop.py`) -The full pen-to-reply loop. Watches for pen idle, captures the screen, -sends it to a vision model, and writes the reply back as ink. +Python version of the closed loop. Largely superseded by `glossad.c` but +still useful for development and testing. ```sh python glossa_loop.py [--model MODEL] [--once] @@ -204,8 +284,7 @@ python glossa_loop.py [--model MODEL] [--once] 2. **Capture**: trigger `/tmp/glossa_screenshot`, the extension dumps the framebuffer to `/tmp/glossa_fb.raw`, pull it via SCP. 3. **Crop**: drop top 80px (toolbar) and bottom 200px (swirl zone). -4. **Blank check**: skip the API entirely if no real content (see grid - artifact note below). +4. **Blank check**: skip the API entirely if no real content. 5. **Vision model**: send the full page image; the model does OCR + relevance filtering + reply in one call. Returns `[NO_NEW_TEXT]` when there's nothing new for it. @@ -218,22 +297,6 @@ python glossa_loop.py [--model MODEL] [--once] commands) to kill the ~1s handshake per call. Cleans up stale sockets on open and uses `ConnectTimeout=10` so it fails fast instead of hanging. -### The reMarkable display grid artifact (IMPORTANT) - -The captured framebuffer contains a **uniform grid pattern**: isolated -2px-tall rows every ~42px, each with **exactly 68 dark pixels**, pure black -(value 0), spanning the entire screen height. This is a display/refresh -artifact, NOT content. It is indistinguishable from ink by pixel value. - -To find real handwriting, filter it out by: -- Counting dark pixels per row (`< 128` threshold). -- Ignoring rows with **exactly 68** dark pixels (the grid signature). -- Real text rows have variable counts (anything but 68) and form contiguous - clusters many rows tall; the grid is always isolated 2-row pairs. - -This filter is used both for the blank-page check and for finding the -content bottom to position replies. - ### Vision model API Currently uses Charm's Hyper gateway: diff --git a/xovi-ext/glossad/glossad.c b/xovi-ext/glossad/glossad.c index feac8c8..41a0f1f 100644 --- a/xovi-ext/glossad/glossad.c +++ b/xovi-ext/glossad/glossad.c @@ -51,7 +51,7 @@ #define API_KEY_PATH "/home/root/.glossa_key" #define API_HOST "potluck.dunkirk.sh" #define API_PATH "/v1/chat/completions" -#define API_MODEL "gpt-4.1-nano" +#define API_MODEL "pioneer/gpt-5-nano" #define FONT_PATH "/home/root/font_data.json" #define RENDER_OUT_PATH "/tmp/glossa_render.json" #define THINKING_PATH "/tmp/glossa_thinking" -- 2.51.2