From b974a3a2e08ed6037bfc6b2b34655982010a53fd Mon Sep 17 00:00:00 2001 From: Roscoe Rubin-Rottenberg Date: Tue, 16 Jun 2026 16:47:41 -0400 Subject: [PATCH] chore: add serve-sim skill --- .agents/skills/serve-sim/README.md | 111 +++++++++ .agents/skills/serve-sim/SKILL.md | 178 ++++++++++++++ .agents/skills/serve-sim/evals/evals.json | 80 +++++++ .../serve-sim/references/buttons-rotation.md | 61 +++++ .../skills/serve-sim/references/ca-debug.md | 39 ++++ .agents/skills/serve-sim/references/camera.md | 151 ++++++++++++ .../skills/serve-sim/references/endpoints.md | 108 +++++++++ .../skills/serve-sim/references/gestures.md | 131 +++++++++++ .../serve-sim/references/permissions.md | 99 ++++++++ .../skills/serve-sim/references/workflows.md | 220 ++++++++++++++++++ .../skills/serve-sim/scripts/check-prereqs.sh | 47 ++++ .../serve-sim/scripts/ensure-running.sh | 40 ++++ 12 files changed, 1265 insertions(+) create mode 100644 .agents/skills/serve-sim/README.md create mode 100644 .agents/skills/serve-sim/SKILL.md create mode 100644 .agents/skills/serve-sim/evals/evals.json create mode 100644 .agents/skills/serve-sim/references/buttons-rotation.md create mode 100644 .agents/skills/serve-sim/references/ca-debug.md create mode 100644 .agents/skills/serve-sim/references/camera.md create mode 100644 .agents/skills/serve-sim/references/endpoints.md create mode 100644 .agents/skills/serve-sim/references/gestures.md create mode 100644 .agents/skills/serve-sim/references/permissions.md create mode 100644 .agents/skills/serve-sim/references/workflows.md create mode 100755 .agents/skills/serve-sim/scripts/check-prereqs.sh create mode 100755 .agents/skills/serve-sim/scripts/ensure-running.sh diff --git a/.agents/skills/serve-sim/README.md b/.agents/skills/serve-sim/README.md new file mode 100644 index 00000000..59af49fc --- /dev/null +++ b/.agents/skills/serve-sim/README.md @@ -0,0 +1,111 @@ +# serve-sim agent skill + +A portable [Agent Skill](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview) that teaches AI coding agents to drive an Apple Simulator via the [serve-sim](https://github.com/EvanBacon/serve-sim) CLI. + +Works in Claude Code, Cursor, Codex CLI, Gemini CLI, GitHub Copilot, and any other tool that implements the open Agent Skills standard. The same `SKILL.md` works across all of them without modification. + +## What it does + +Once installed, your agent knows how to: + +- Tap at normalized coordinates (`serve-sim tap`). +- Send multi-touch / drag / swipe gestures with the correct JSON shape and edge flags. +- Press the six valid hardware buttons (`home`, `swipe_home`, `app_switcher`, `lock`, `siri`, `side_button`). +- Rotate the simulator (`portrait`, `portrait_upside_down`, `landscape_left`, `landscape_right`). +- Inject a synthetic camera feed (placeholder, image, video, or live webcam) with mirror-mode control. +- Toggle CoreAnimation debug overlays (blended layers, off-screen rendering, slow animations, …). +- Simulate a memory warning. +- Discover the running stream's URL and read the simulator's accessibility tree to find UI elements. +- Hand the stream URL off to the host agent's preview pane (`preview_start` in Claude Code, equivalents elsewhere) so the user sees the simulator inline. + +It also teaches the agent the **gotchas** (use `tap`, not `gesture`, for plain taps), the **prerequisites** (macOS, Xcode CLI tools, Node 18+, macOS 14+ for camera), and **anti-patterns** to avoid. + +## Install + +The skill lives in this repo under `skills/serve-sim/`, so it is discoverable by the Agent Skills tooling directly from the serve-sim repository. + +### Claude Code + +```sh +/plugin marketplace add EvanBacon/serve-sim +/plugin install serve-sim +``` + +### Any agent that supports the Agent Skills standard (Cursor, Codex CLI, Gemini CLI, …) + +```sh +bunx add-skill EvanBacon/serve-sim +# or +npx skills add EvanBacon/serve-sim +``` + +### Manual install + +Copy this folder into your agent's skills directory: + +```sh +# from a clone of this repo +cp -r skills/serve-sim ~/.claude/skills/serve-sim +# or for other agents: ~/.agents/skills/serve-sim, ~/.cursor/skills/serve-sim, etc. +``` + +The skill is a folder with a `SKILL.md` file plus reference documents. No build step. + +## Prerequisites on the user's machine + +The agent checks these for you, but for reference: + +- macOS host (any recent version). +- Xcode command line tools (`xcode-select --install`). +- Node.js 18+. +- macOS 14+ if you want camera injection. +- At least one booted iOS, iPad, or Apple Watch simulator. + +`serve-sim` itself is invoked via `npx serve-sim` — no global install required. + +## How it's structured + +``` +serve-sim/ +├── SKILL.md (loaded when the skill triggers) +├── references/ +│ ├── gestures.md (gesture JSON, edges, multi-touch, recipes) +│ ├── buttons-rotation.md (the six buttons, the four orientations) +│ ├── camera.md (camera injection: sources, mirroring, hot-swap) +│ ├── ca-debug.md (CoreAnimation debug flags) +│ ├── endpoints.md (HTTP + WebSocket surface) +│ └── workflows.md (end-to-end recipes incl. preview handoff) +├── scripts/ +│ ├── check-prereqs.sh (verify host satisfies requirements) +│ └── ensure-running.sh (idempotent start of the helper) +└── evals/ + └── evals.json (6 test prompts for agent quality) +``` + +Following Anthropic's recommended structure: short `SKILL.md`, references one level deep, executable scripts that the agent can run without loading their source into context. + +## Designed around progressive disclosure + +- **Discovery**: only the `name` and `description` from the frontmatter cost tokens at startup. +- **Activation**: when the agent decides the task matches, it reads `SKILL.md`. +- **Execution**: it reads only the reference files relevant to the current task. + +This keeps context usage low across hundreds of installed skills. + +## Source of truth + +Every claim in this skill — the six button names, the four orientations, the gesture JSON shape, the edge values, the HTTP endpoints — was verified against the serve-sim source at the time of authoring. The skill does not invent behavior the CLI does not expose. When the CLI changes, update the skill and the `evals/` alongside it. + +## Evals + +`evals/evals.json` contains six representative prompts with expected behaviors, suitable for running through Anthropic's `skill-creator` eval framework. When changing the skill, re-run the evals to catch regressions. + +## Contributing + +Found a divergence between this skill and serve-sim's actual behavior? Open an issue or PR on this repo. + +Want to add a workflow recipe? Add it to `references/workflows.md` with an explanation of when an agent would use it, and add a matching eval to `evals/evals.json`. + +## License + +Apache-2.0, same as the rest of the serve-sim repository. diff --git a/.agents/skills/serve-sim/SKILL.md b/.agents/skills/serve-sim/SKILL.md new file mode 100644 index 00000000..aba8f519 --- /dev/null +++ b/.agents/skills/serve-sim/SKILL.md @@ -0,0 +1,178 @@ +--- +name: serve-sim +description: Control and stream a running iOS, iPad, or Apple Watch Simulator with npx serve-sim. Use for simulator preview, taps, gestures, hardware buttons, rotation, camera injection, permissions, accessibility, and CoreAnimation debug. +license: Apache-2.0 +--- + +# serve-sim + +Drive an Apple Simulator (iOS, iPad, Apple Watch) from an agent using the [serve-sim](https://github.com/EvanBacon/serve-sim) CLI. serve-sim spawns a Swift helper that captures the simulator framebuffer via `simctl io`, exposes it as an MJPEG stream plus a binary WebSocket input channel, and serves a React preview UI on top. This skill teaches an agent the exact CLI surface, the gesture JSON shape, the gotchas, and the recommended workflows. + +## When to use + +- The user wants an agent to **tap, swipe, drag, pinch, or send hardware buttons** to a running Apple Simulator. +- The user wants to **stream a simulator** to a browser (local, LAN, or tunneled) for review or remote control. +- The user wants to **inject a synthetic camera feed** (file, webcam, or animated placeholder) into a specific app on the simulator. +- The user wants to **toggle CoreAnimation debug overlays** (off-screen rendering, blended layers, slow animations) for performance work. +- The user wants to **simulate a memory warning** or **rotate the device** programmatically. +- The user wants to **read the simulator's accessibility tree** to find UI elements without pixel hunting. +- The user wants to **grant, revoke, or reset an app's privacy permissions** — camera, photos, location, contacts, or **push notifications**. + +## When NOT to use + +- Android emulators → use `adb shell` tooling. +- Building or installing an iOS app → use `xcodebuild` or `xcrun simctl install`. +- React Native in-app runtime debugging (Redux state, network inspection, component tree) → use rn-debugger tooling. +- Real iOS hardware devices → use `xcrun devicectl` or Xcode. + +## Prerequisites + +Before any other action, verify the host satisfies these. If something is missing, tell the user exactly what to install — do not proceed. + +| Requirement | Check command | Why | +|---|---|---| +| macOS host | `uname -s` returns `Darwin` | serve-sim only runs on macOS | +| Xcode CLI tools | `xcrun --version` exits 0 | `simctl` is the underlying simulator driver | +| Node.js ≥18 | `node --version` ≥18 | serve-sim is an npm package run via `npx` | +| macOS 14+ (optional) | `sw_vers -productVersion` ≥14 | Required ONLY for `camera` subcommand | + +A bundled helper script is available: `scripts/check-prereqs.sh`. Run it; if it exits non-zero, surface the message to the user. + +A booted simulator is required for most subcommands. Check with `xcrun simctl list devices booted`. If none are booted, tell the user to open Xcode → Simulator or to run `xcrun simctl boot `. + +## Mental model + +```text +┌──────────────┐ simctl io ┌─────────────────┐ MJPEG / WS ┌─────────┐ +│ iOS Simulator│ ──────────► │ serve-sim-bin │ ───────────► │ Browser │ +└──────────────┘ (Swift) │ (per-device) │ └─────────┘ + └─────────────────┘ + ▲ + state file in + $TMPDIR/serve-sim/ + ▲ + ┌──────────────────┐ + │ serve-sim CLI │ + └──────────────────┘ +``` + +Key invariants the agent must respect: + +- **All coordinates are normalized 0..1**, with `(0, 0)` at top-left and `(1, 1)` at bottom-right of the display. Never pass pixel coordinates. +- **One helper per device**. Multiple booted simulators are supported by passing several device names or by attaching to all. +- **State lives in `$TMPDIR/serve-sim/server-{udid}.json`**. Use `serve-sim --list` to query it; do not read the JSON directly unless you know what you are doing. +- **The orientation set via `rotate` is remembered by the helper**, and subsequent gestures are rotated client-side. An agent that sends raw coords after a rotation does not need to compensate manually. + +## Common operations + +| Goal | Command | Notes | +|---|---|---| +| Start preview server | `npx serve-sim [device]` | Default preview at `http://localhost:3200`, stream at `:3100`. Foreground process. | +| Start headless / daemon | `npx serve-sim --detach [device]` | Returns JSON with `pid`, `port`, `url`. Use for agent loops. | +| Show stream in host's preview | `npx serve-sim --detach -q` → hand off `url` to host preview tool | See "Showing the stream in your agent's preview" section. | +| List running streams | `npx serve-sim --list` | Add `-q` for JSON-only output. | +| Stop all helpers | `npx serve-sim --kill` | Pass `[device]` to stop a specific one. | +| Single tap | `npx serve-sim tap ` | ` ` in `0..1`. **Use this, not `gesture`, for plain taps.** See "Critical gotcha" below. | +| Multi-step gesture | `npx serve-sim gesture ''` | See [references/gestures.md](references/gestures.md). | +| Hardware button | `npx serve-sim button ` | Names: `home`, `swipe_home`, `app_switcher`, `lock`, `siri`, `side_button`. See [references/buttons-rotation.md](references/buttons-rotation.md). | +| Rotate device | `npx serve-sim rotate ` | `portrait`, `portrait_upside_down`, `landscape_left`, `landscape_right`. | +| Simulate memory warning | `npx serve-sim memory-warning` | Equivalent to Debug → Simulate Memory Warning. | +| CoreAnimation debug | `npx serve-sim ca-debug