diff --git a/captutor/README.md b/captutor/README.md --- a/captutor/README.md +++ b/captutor/README.md @@ -115,10 +115,12 @@ ### Captutor Stage Mode CDP clicks do not move the macOS cursor, and `reel` films the real window — so without coordination the video would show a dead pointer parked in a corner while -buttons depressed by themselves. `bin/stage.mjs` is the production path: it films -the real 1.5× macOS pointer and moves it smoothly to the exact same coordinate as -each trusted CDP click. The older shadow-DOM tutorial pointer remains available -outside Stage Mode. +buttons depressed by themselves. `bin/stage.mjs` is the production path: it +compiles a transparent, click-through Swift cursor overlay and moves its exact +arrow-tip hotspot to the same coordinate as each trusted CDP click. The native +overlay eases at 120 Hz, leaves a restrained particle trail, and emits a small +click burst. It is capture-visible but never participates in browser hit-testing; +`CAPTUTOR_REAL_CURSOR=1` remains available for explicitly human-driven takes. Stage Mode is a reversible transaction around any Captutor command. It saves the current desk, closes stale QuickTime previews, switches macOS to Light appearance @@ -126,16 +128,37 @@ and the display to 2× HiDPI (1280×720 logical), centers the browser, raises encoding quality, uses a branded light wallpaper, and temporarily hides desktop icons, Dock, menu bar, Stats, Macpal's desktop badge, and Slab prompt sigils. The recorder captures the complete physical desktop, preserving the real rounded -window, shadow, and equal margin. A Swift desktop-level renderer adds a subtle, -deterministic field of rising icon-only production Fuser SVG marks. The -wallpaper contains no wordmark or decorative color field: it renders only the -mark, black-on-white or white-on-black from the active macOS appearance. It is -click-through, runs behind every normal window, and exits inside the same Stage -transaction. Delivery changes only the tiny +window, shadow, and equal margin. A Swift desktop-level renderer supplies the +selected client backdrop. It is click-through, runs behind every normal window, +and exits inside the same Stage transaction. Delivery changes only the tiny ScreenCaptureKit status dot in the extreme top-right, using live pixels sampled from the adjacent desktop; it never masks or crops the browser window. -Its `finally` handler restores the saved display mode, pointer size, wallpaper, +Its `finally` handler restores the saved display mode, optional real-pointer size, wallpaper, processes, and desktop preferences on success, failure, or interruption. + +Stage backdrops are selected per brand. `fuser` is currently the default and +uses the twelve node positions from the production `fuser-mark.svg` as one +connected glossy metaball sculpture over a quiet black-and-white field. Obsidian, +pearl, and graphite variants stay inside Fuser's monochrome brand system. One +instanced Metal pass raymarches the live smooth-union field at the display's +native backing resolution; each logo tumbles independently through yaw, pitch, +and roll. Smaller background marks use a softer, lower-step depth-of-field +treatment to preserve the power budget. There is no sprite +stepping, opacity pulse, wordmark, or generated approximation. The former +monochrome rising-mark field remains available as `classic`. +The ambient drift reverses gently inside a per-logo safe inset, so even the +largest rotating metaball volume remains fully on-screen without edge fades. + +Regenerate the three palette stills after changing the implicit surface: + +```bash +swift captutor/bin/render-fuser-metaballs.swift captutor/assets +``` + +```bash +node bin/stage.mjs --brand fuser render +node bin/stage.mjs --brand classic render +``` ## Staying signed in, and not spending the client's money @@ -191,6 +214,36 @@ `/api/v1/account/getQuotas` looks nicer and is CORS-blocked from `app.` — don't.) ## Running it +### Pathfinding preflight (invisible CDP frame) + +Before authoring selectors, take one structured internal frame of the real +Fuser tab: + +```sh +CDP_PORT=9333 node bin/cdp-frame.mjs --match fuser.studio +CDP_PORT=9333 node bin/cdp-frame.mjs --match fuser.studio \ + --screenshot /tmp/fuser-preflight.png +``` + +The JSON reports the viewport, focus, visible controls with locator candidates, +and React Flow nodes, handles, and edges. The optional screenshot comes from +`Page.captureScreenshot`: it is read directly from Chrome's compositor and +never draws Frame, OCR, target, cursor, or Puppet UI on the filming display. + +Use this command for exploratory preflight instead of one-off `node -e` scripts +that call `attach()` and forget to close the socket. Programmatic probes should +use `withSession()` from `lib/cdp.mjs`, or close a directly owned session in a +`finally` block. The CLI has a 15-second hard ceiling and always closes CDP, so +pathfinding cannot silently turn into repeated 120-second tool timeouts. + +Captutor also enables Chrome's renderer-crash event before setup and runs a +bounded page heartbeat immediately before recording and throughout every take. +Chrome can leave an "Aw, Snap!" target in `/json` with its old Fuser title and +URL, so target metadata alone is never treated as proof of health. A crash stops +the reel, preserves an `aborted-browser-*.mp4`, and records +`browser-renderer-crash` in `out/failures.ndjson` instead of leaving the mission +apparently in progress. + `reel` needs SlabMenubar running with the Screen Recording grant (`node slab/bin/frame.mjs doctor`). @@ -277,6 +330,23 @@ Deriving the video from the page is what keeps the two from drifting apart. `apps/` has 13 pages ready to draft. **Recipes has no docs page at all** — so that one is authored by hand (`screenplays/smoke.mjs` is its skeleton), and the page and the video get written together. + +### Fuser onboarding contract + +Tutorial capture must not depend on translated button copy, generated classes, +or the portal structure of a tooltip. Fuser exposes active tutorial state through +semantic DOM attributes: + +- `data-onboarding-step` and `data-onboarding-active` mark the product target; +- `data-onboarding-overlay` and `data-onboarding-content-index` identify the card; +- `data-onboarding-requirement` reports a gated interaction; +- `data-onboarding-action` names Previous, Next, Finish, Skip, and Replay controls. + +`lib/onboarding.mjs` reads this contract, waits for an exact step, and advances +the walkthrough. On localhost it can satisfy manual tutorial requirements through +Fuser's narrow `window.__fuserOnboardingAudit` bridge, allowing a non-billable +replay audit without reaching into Zustand or executing a model. The bridge is +absent in production. ## Known gaps diff --git a/captutor/assets/fuser-metaballs-0.png b/captutor/assets/fuser-metaballs-0.png new file mode 100644 --- /dev/null +++ b/captutor/assets/fuser-metaballs-0.png diff --git a/captutor/assets/fuser-metaballs-1.png b/captutor/assets/fuser-metaballs-1.png new file mode 100644 --- /dev/null +++ b/captutor/assets/fuser-metaballs-1.png diff --git a/captutor/assets/fuser-metaballs-2.png b/captutor/assets/fuser-metaballs-2.png new file mode 100644 --- /dev/null +++ b/captutor/assets/fuser-metaballs-2.png diff --git a/captutor/bin/captutor-cursor.swift b/captutor/bin/captutor-cursor.swift new file mode 100644 --- /dev/null +++ b/captutor/bin/captutor-cursor.swift @@ -0,0 +1,328 @@ +import AppKit +import CoreGraphics +import Foundation + +// Captutor's filmed pointer. Browser input remains in CDP; this process only +// paints a native, click-through overlay at the same global screen coordinate. + +private struct Command: Decodable { + let op: String + let x: CGFloat? + let y: CGFloat? + let durationMs: Double? +} + +private struct Particle { + var position: CGPoint + var velocity: CGVector + var bornAt: TimeInterval + var lifetime: TimeInterval + var radius: CGFloat + var color: NSColor +} + +private final class CursorView: NSView { + var pointer = CGPoint.zero + var isPointerVisible = false + var isPressed = false + var particles: [Particle] = [] + + override var isFlipped: Bool { true } + override var isOpaque: Bool { false } + + override func draw(_ dirtyRect: NSRect) { + guard let context = NSGraphicsContext.current?.cgContext else { return } + context.clear(bounds) + + let now = ProcessInfo.processInfo.systemUptime + for particle in particles { + let age = max(0, now - particle.bornAt) + let progress = min(1, age / particle.lifetime) + let alpha = CGFloat(pow(1 - progress, 1.65)) + let center = CGPoint( + x: particle.position.x + particle.velocity.dx * age, + y: particle.position.y + particle.velocity.dy * age + CGFloat(age * age) * 22 + ) + context.setFillColor(particle.color.withAlphaComponent(alpha * 0.82).cgColor) + context.addEllipse(in: CGRect( + x: center.x - particle.radius, + y: center.y - particle.radius, + width: particle.radius * 2, + height: particle.radius * 2 + )) + context.fillPath() + } + + guard isPointerVisible else { return } + + // The path begins at (0, 0): pointer is the actual click hotspot, not + // the centre of an oversized cursor box. Its proportions follow the + // familiar macOS arrow while the dark face and light rim read cleanly + // on both Fuser's pale canvas and its darker controls. + context.saveGState() + context.translateBy(x: pointer.x, y: pointer.y) + let pressedScale: CGFloat = isPressed ? 0.92 : 1 + context.scaleBy(x: pressedScale, y: pressedScale) + + let path = CGMutablePath() + path.move(to: CGPoint(x: 0, y: 0)) + path.addLine(to: CGPoint(x: 1.6, y: 22.8)) + path.addLine(to: CGPoint(x: 7.3, y: 17.1)) + path.addLine(to: CGPoint(x: 12.0, y: 27.0)) + path.addLine(to: CGPoint(x: 17.0, y: 24.6)) + path.addLine(to: CGPoint(x: 12.3, y: 15.0)) + path.addLine(to: CGPoint(x: 20.4, y: 14.2)) + path.closeSubpath() + + context.setShadow(offset: CGSize(width: 0.7, height: 1.4), blur: 2.4, + color: NSColor.black.withAlphaComponent(0.55).cgColor) + context.setFillColor(NSColor(calibratedWhite: 0.055, alpha: 1).cgColor) + context.addPath(path) + context.fillPath() + context.setShadow(offset: .zero, blur: 0, color: nil) + context.setStrokeColor(NSColor(calibratedWhite: 0.98, alpha: 0.98).cgColor) + context.setLineWidth(2.25) + context.setLineJoin(.round) + context.addPath(path) + context.strokePath() + + // A restrained cool hairline gives the familiar pointer a little life + // without turning it into an annotation or obscuring the target. + context.setStrokeColor(NSColor(calibratedRed: 0.42, green: 0.78, blue: 1, alpha: 0.72).cgColor) + context.setLineWidth(0.62) + context.addPath(path) + context.strokePath() + context.restoreGState() + } +} + +private struct DisplaySurface { + let displayID: CGDirectDisplayID + let quartzFrame: CGRect + let window: NSWindow + let view: CursorView +} + +private final class CursorController { + private var surfaces: [DisplaySurface] = [] + private var activeSurface: Int? + private var currentGlobal: CGPoint? + private var moveStartGlobal = CGPoint.zero + private var moveTargetGlobal = CGPoint.zero + private var moveStartedAt: TimeInterval = 0 + private var moveDuration: TimeInterval = 0 + private var lastTrailGlobal: CGPoint? + private var timer: Timer? + + init() { + rebuildSurfaces() + timer = Timer(timeInterval: 1.0 / 120.0, repeats: true) { [weak self] _ in + self?.tick() + } + RunLoop.main.add(timer!, forMode: .common) + } + + deinit { timer?.invalidate() } + + private func rebuildSurfaces() { + for screen in NSScreen.screens { + guard let number = screen.deviceDescription[NSDeviceDescriptionKey("NSScreenNumber")] as? NSNumber else { + continue + } + let displayID = CGDirectDisplayID(number.uint32Value) + let window = NSWindow( + contentRect: screen.frame, + styleMask: .borderless, + backing: .buffered, + defer: false, + screen: screen + ) + let view = CursorView(frame: CGRect(origin: .zero, size: screen.frame.size)) + window.contentView = view + window.backgroundColor = .clear + window.isOpaque = false + window.hasShadow = false + window.ignoresMouseEvents = true + // Borderless accessory windows otherwise default to sharingState + // 0 and disappear from ScreenCaptureKit. The cursor is presentation + // UI, so make its pixels explicitly capture-visible while keeping + // the surface click-through and absent from window cycling. + window.sharingType = .readOnly + window.level = NSWindow.Level(rawValue: Int(CGWindowLevelForKey(.statusWindow)) + 1) + window.collectionBehavior = [.canJoinAllSpaces, .stationary, .ignoresCycle, .fullScreenAuxiliary] + window.isReleasedWhenClosed = false + surfaces.append(DisplaySurface( + displayID: displayID, + quartzFrame: CGDisplayBounds(displayID), + window: window, + view: view + )) + } + } + + private func surfaceIndex(containing point: CGPoint) -> Int? { + surfaces.firstIndex(where: { $0.quartzFrame.contains(point) }) ?? surfaces.indices.first + } + + private func localPoint(_ global: CGPoint, on surface: DisplaySurface) -> CGPoint { + // CGDisplayBounds and the command protocol both use a top-left origin. + // CursorView is flipped, so subtracting the display origin preserves the + // exact coordinate, including on a secondary display. + CGPoint(x: global.x - surface.quartzFrame.minX, + y: global.y - surface.quartzFrame.minY) + } + + private func setGlobalPoint(_ global: CGPoint, emitTrail: Bool) { + guard let index = surfaceIndex(containing: global) else { return } + if activeSurface != index { + if let old = activeSurface { + surfaces[old].view.isPointerVisible = false + surfaces[old].view.needsDisplay = true + surfaces[old].window.orderOut(nil) + } + activeSurface = index + surfaces[index].window.orderFrontRegardless() + } + + let surface = surfaces[index] + let local = localPoint(global, on: surface) + surface.view.pointer = local + surface.view.isPointerVisible = true + currentGlobal = global + + if emitTrail { + let distance = lastTrailGlobal.map { hypot(global.x - $0.x, global.y - $0.y) } ?? 100 + if distance >= 13 { + emitParticle(at: local, in: surface.view, burst: false) + lastTrailGlobal = global + } + } + surface.view.needsDisplay = true + } + + func move(to target: CGPoint, durationMs: Double) { + let fallback = surfaces.first.map { + CGPoint(x: $0.quartzFrame.midX, y: $0.quartzFrame.midY) + } ?? target + moveStartGlobal = currentGlobal ?? fallback + moveTargetGlobal = target + moveStartedAt = ProcessInfo.processInfo.systemUptime + moveDuration = max(0, durationMs / 1000) + if currentGlobal == nil { setGlobalPoint(moveStartGlobal, emitTrail: false) } + if moveDuration == 0 { setGlobalPoint(target, emitTrail: true) } + } + + func setPressed(_ pressed: Bool) { + guard let index = activeSurface else { return } + surfaces[index].view.isPressed = pressed + surfaces[index].view.needsDisplay = true + } + + func click() { + guard let index = activeSurface else { return } + let surface = surfaces[index] + for _ in 0..<9 { emitParticle(at: surface.view.pointer, in: surface.view, burst: true) } + surface.view.needsDisplay = true + } + + func hide() { + for surface in surfaces { + surface.view.isPointerVisible = false + surface.view.particles.removeAll() + surface.view.needsDisplay = true + surface.window.orderOut(nil) + } + activeSurface = nil + currentGlobal = nil + lastTrailGlobal = nil + } + + private func emitParticle(at point: CGPoint, in view: CursorView, burst: Bool) { + let palette = [ + NSColor(calibratedRed: 0.43, green: 0.82, blue: 1.00, alpha: 1), + NSColor(calibratedRed: 0.73, green: 0.61, blue: 1.00, alpha: 1), + NSColor(calibratedRed: 1.00, green: 0.77, blue: 0.42, alpha: 1), + ] + let angle = CGFloat.random(in: burst ? 0...(2 * .pi) : (0.35 * .pi)...(0.78 * .pi)) + let speed = CGFloat.random(in: burst ? 24...68 : 8...22) + let origin = CGPoint( + x: point.x + CGFloat.random(in: burst ? -2...3 : 2...7), + y: point.y + CGFloat.random(in: burst ? -2...3 : 8...18) + ) + view.particles.append(Particle( + position: origin, + velocity: CGVector(dx: cos(angle) * speed, dy: sin(angle) * speed), + bornAt: ProcessInfo.processInfo.systemUptime, + lifetime: Double.random(in: burst ? 0.34...0.52 : 0.22...0.34), + radius: CGFloat.random(in: burst ? 1.35...2.35 : 0.8...1.45), + color: palette.randomElement()! + )) + } + + private func tick() { + let now = ProcessInfo.processInfo.systemUptime + if moveDuration > 0, now < moveStartedAt + moveDuration { + let t = min(1, max(0, (now - moveStartedAt) / moveDuration)) + let eased = t < 0.5 + ? 4 * t * t * t + : 1 - pow(-2 * t + 2, 3) / 2 + setGlobalPoint(CGPoint( + x: moveStartGlobal.x + (moveTargetGlobal.x - moveStartGlobal.x) * eased, + y: moveStartGlobal.y + (moveTargetGlobal.y - moveStartGlobal.y) * eased + ), emitTrail: true) + } else if moveDuration > 0 { + moveDuration = 0 + setGlobalPoint(moveTargetGlobal, emitTrail: true) + } + + for surface in surfaces { + let oldCount = surface.view.particles.count + surface.view.particles.removeAll { now - $0.bornAt >= $0.lifetime } + if oldCount != surface.view.particles.count || !surface.view.particles.isEmpty { + surface.view.needsDisplay = true + } + } + } +} + +private final class AppDelegate: NSObject, NSApplicationDelegate { + private var controller: CursorController? + + func applicationDidFinishLaunching(_ notification: Notification) { + controller = CursorController() + DispatchQueue.global(qos: .userInteractive).async { [weak self] in + while let line = readLine() { + guard let data = line.data(using: .utf8), + let command = try? JSONDecoder().decode(Command.self, from: data) else { continue } + DispatchQueue.main.async { self?.handle(command) } + } + DispatchQueue.main.async { + self?.controller?.hide() + NSApp.terminate(nil) + } + } + FileHandle.standardOutput.write(Data("ready\n".utf8)) + } + + private func handle(_ command: Command) { + switch command.op { + case "move": + guard let x = command.x, let y = command.y else { return } + controller?.move(to: CGPoint(x: x, y: y), durationMs: command.durationMs ?? 0) + case "down": controller?.setPressed(true) + case "up": controller?.setPressed(false) + case "click": controller?.click() + case "hide": controller?.hide() + case "quit": + controller?.hide() + NSApp.terminate(nil) + default: break + } + } +} + +private let app = NSApplication.shared +app.setActivationPolicy(.accessory) +private let delegate = AppDelegate() +app.delegate = delegate +app.run() diff --git a/captutor/bin/captutor-wallpaper.swift b/captutor/bin/captutor-wallpaper.swift --- a/captutor/bin/captutor-wallpaper.swift +++ b/captutor/bin/captutor-wallpaper.swift @@ -2,6 +2,8 @@ // CaptutorWallpaper — a quiet animated Fuser stage behind every filmed window. import AppKit import CoreGraphics +import Metal +import QuartzCore private struct Particle { let x: CGFloat @@ -133,8 +135,11 @@ (dark ? NSColor.black : NSColor.white).setFill() rect.fill() - if card.phase == "ambient" { - for particle in field { + // Keep the drifting Fuser marks as the shared visual system behind + // every phase. Cards soften that field and add type; they never place + // a separate hero logo in front of the title. + let fieldOpacityScale: CGFloat = card.phase == "ambient" ? 1 : 0.30 + for particle in field { // One shared rise rate keeps the collision-safe layout rigid in Y, // while independent sine waves make the marks sway organically. let progress = (particle.y * 1.28 + t * 0.014).truncatingRemainder(dividingBy: 1.28) @@ -151,10 +156,10 @@ : 0.58 + 0.42 * (sin(t * 0.32 + particle.phase) + 1) / 2 drawLogo( in: NSRect(x: x - logoWidth / 2, y: y - particle.size / 2, width: logoWidth, height: particle.size), - opacity: particle.opacity * breath + opacity: particle.opacity * breath * fieldOpacityScale ) - } - } else { + } + if card.phase != "ambient" { drawCard(in: rect, elapsed: ProcessInfo.processInfo.systemUptime - cardChangedAt) } } @@ -173,19 +178,10 @@ func ease(_ delay: TimeInterval, _ duration: TimeInterval) -> CGFloat { let raw = CGFloat(min(1, max(0, (elapsed - delay) / duration))) return raw * raw * (3 - 2 * raw) } - let logoProgress = ease(0.04, 0.42) - let titleProgress = ease(0.20, 0.52) + let titleProgress = ease(0.12, 0.52) NSGraphicsContext.saveGraphicsState() let portrait = canvas.width < canvas.height - let logoSize: CGFloat = portrait ? 116 : 132 - let logoRect = NSRect( - x: canvas.midX - logoSize / 2, - y: canvas.midY + (portrait ? 38 : 28) - (1 - logoProgress) * 24, - width: logoSize, - height: logoSize - ) - drawLogo(in: logoRect, opacity: logoProgress) let titleSize: CGFloat = portrait ? 48 : 62 NSGraphicsContext.saveGraphicsState() NSGraphicsContext.current?.cgContext.setAlpha(titleProgress) @@ -193,7 +189,7 @@ drawCentered( card.title ?? "", in: NSRect( x: canvas.minX + canvas.width * 0.10, - y: canvas.midY - (portrait ? 190 : 170) - (1 - titleProgress) * 18, + y: canvas.midY - (portrait ? 125 : 110) - (1 - titleProgress) * 18, width: canvas.width * 0.80, height: portrait ? 250 : 220 ), @@ -251,10 +247,545 @@ return image } } +private struct MetaballInstance { + var placement: SIMD4 + var motion: SIMD4 + var appearance: SIMD4 +} + +private struct MetaballUniforms { + var viewport: SIMD2 + var time: Float + var dark: Float +} + +// One transparent, instanced Metal pass renders every logo. Only each logo's +// bounding quad is shaded, and the drawable is intentionally sub-Retina; this +// keeps real-time implicit surfaces practical for a quiet desktop backdrop. +private final class MetaballRenderer { + let layer = CAMetalLayer() + private let device: MTLDevice + private let queue: MTLCommandQueue + private let pipeline: MTLRenderPipelineState + private var instances: [MetaballInstance] = [] + private var timer: Timer? + private var started = ProcessInfo.processInfo.systemUptime + private var dark = false + + init?() { + guard let device = MTLCreateSystemDefaultDevice(), + let queue = device.makeCommandQueue() else { return nil } + self.device = device + self.queue = queue + layer.device = device + layer.pixelFormat = .bgra8Unorm + layer.framebufferOnly = true + layer.isOpaque = false + layer.maximumDrawableCount = 2 + layer.contentsGravity = .resize + + do { + let library = try device.makeLibrary(source: Self.shader, options: nil) + let descriptor = MTLRenderPipelineDescriptor() + descriptor.vertexFunction = library.makeFunction(name: "metaballVertex") + descriptor.fragmentFunction = library.makeFunction(name: "metaballFragment") + descriptor.colorAttachments[0].pixelFormat = layer.pixelFormat + descriptor.colorAttachments[0].isBlendingEnabled = true + descriptor.colorAttachments[0].sourceRGBBlendFactor = .sourceAlpha + descriptor.colorAttachments[0].destinationRGBBlendFactor = .oneMinusSourceAlpha + descriptor.colorAttachments[0].sourceAlphaBlendFactor = .one + descriptor.colorAttachments[0].destinationAlphaBlendFactor = .oneMinusSourceAlpha + pipeline = try device.makeRenderPipelineState(descriptor: descriptor) + } catch { + fputs("Captutor metaball shader unavailable: \(error)\n", stderr) + return nil + } + + timer = Timer.scheduledTimer(withTimeInterval: 1.0 / 30.0, repeats: true) { [weak self] _ in + self?.draw() + } + RunLoop.main.add(timer!, forMode: .common) + } + + deinit { timer?.invalidate() } + + func setAppearance(dark: Bool) { self.dark = dark } + + func setOpacity(_ opacity: Float) { + CATransaction.begin() + CATransaction.setAnimationDuration(0.48) + layer.opacity = opacity + CATransaction.commit() + } + + func setLayout(bounds: CGRect, backingScale: CGFloat, + specs: [(CGFloat, CGFloat, CGFloat, TimeInterval, TimeInterval, CGFloat, Int)]) { + layer.frame = bounds + // Match the display's backing pixels for crisp foreground silhouettes. + // Small marks spend fewer shader steps and use softer shading below. + let renderScale = max(1, backingScale) + layer.contentsScale = renderScale + layer.drawableSize = CGSize(width: max(1, bounds.width * renderScale), + height: max(1, bounds.height * renderScale)) + let logoScale = min(max(bounds.width / 1280, 0.78), 1.30) * renderScale + instances = specs.enumerated().map { index, spec in + let opacity: Float = spec.2 > 100 ? 0.72 : 0.58 + return MetaballInstance( + placement: SIMD4(Float(spec.0), Float(spec.1), Float(spec.2 * logoScale), opacity), + motion: SIMD4(Float(spec.3), Float(spec.4), Float(spec.5 * renderScale), + spec.2 < 70 ? 1 : (spec.2 < 100 ? 0.52 : 0)), + appearance: SIMD4(Float(spec.6), index.isMultiple(of: 3) ? -1 : 1, + Float(index) * 0.713, 0) + ) + } + } + + private func draw() { + guard !instances.isEmpty, layer.opacity > 0.001, + let drawable = layer.nextDrawable(), + let command = queue.makeCommandBuffer(), + let encoder = command.makeRenderCommandEncoder(descriptor: renderPass(for: drawable.texture)) else { return } + var uniforms = MetaballUniforms( + viewport: SIMD2(Float(layer.drawableSize.width), Float(layer.drawableSize.height)), + time: Float(ProcessInfo.processInfo.systemUptime - started), + dark: dark ? 1 : 0 + ) + encoder.setRenderPipelineState(pipeline) + instances.withUnsafeBytes { bytes in + encoder.setVertexBytes(bytes.baseAddress!, length: bytes.count, index: 0) + } + encoder.setVertexBytes(&uniforms, length: MemoryLayout.stride, index: 1) + encoder.setFragmentBytes(&uniforms, length: MemoryLayout.stride, index: 0) + encoder.drawPrimitives(type: .triangleStrip, vertexStart: 0, vertexCount: 4, + instanceCount: instances.count) + encoder.endEncoding() + command.present(drawable) + command.commit() + } + + private func renderPass(for texture: MTLTexture) -> MTLRenderPassDescriptor { + let pass = MTLRenderPassDescriptor() + pass.colorAttachments[0].texture = texture + pass.colorAttachments[0].loadAction = .clear + pass.colorAttachments[0].storeAction = .store + pass.colorAttachments[0].clearColor = MTLClearColorMake(0, 0, 0, 0) + return pass + } + + private static let shader = #""" + #include + using namespace metal; + + struct Instance { float4 placement; float4 motion; float4 appearance; }; + struct Uniforms { float2 viewport; float time; float dark; }; + struct Raster { + float4 position [[position]]; + float2 local; + float4 placement; + float4 motion; + float4 appearance; + }; + + constant float PI = 3.14159265359; + constant float TAU = 6.28318530718; + constant float3 centers[12] = { + float3( 0.40654, 1.21499, 0.00000), float3( 1.21550, 1.21499, 0.05450), + float3(-0.40235, 1.21499, -0.01517), float3(-0.40235, 0.40476, -0.05028), + float3(-1.21530, 0.40867, 0.02924), float3(-1.21139, -0.40476, 0.04214), + float3(-1.21139, -1.21499, -0.04094), float3(-0.40235, -1.21499, -0.03075), + float3( 0.40654, -1.21499, 0.04950), float3( 0.40654, -0.40476, 0.01693), + float3( 1.21550, -0.40476, -0.05421), float3( 1.21550, 0.40476, 0.00153) + }; + + float smoothMin(float a, float b, float k) { + float h = max(k - abs(a - b), 0.0) / k; + return min(a, b) - h * h * k * 0.25; + } + + float field(float3 p) { + float d = length(p - centers[0]) - 0.355; + for (uint i = 1; i < 12; ++i) + d = smoothMin(d, length(p - centers[i]) - 0.355, 0.30); + return d; + } + + float3 rotateX(float3 p, float a) { + float c = cos(a), s = sin(a); + return float3(p.x, p.y * c - p.z * s, p.y * s + p.z * c); + } + float3 rotateY(float3 p, float a) { + float c = cos(a), s = sin(a); + return float3(p.x * c + p.z * s, p.y, -p.x * s + p.z * c); + } + float3 rotateZ(float3 p, float a) { + float c = cos(a), s = sin(a); + return float3(p.x * c - p.y * s, p.x * s + p.y * c, p.z); + } + float3 inverseTurn(float3 p, float3 angle) { + p = rotateZ(p, -angle.z); + p = rotateY(p, -angle.y); + return rotateX(p, -angle.x); + } + + vertex Raster metaballVertex(uint vertexID [[vertex_id]], uint instanceID [[instance_id]], + constant Instance *instances [[buffer(0)]], + constant Uniforms &u [[buffer(1)]]) { + const float2 corners[4] = { float2(-1,-1), float2(1,-1), float2(-1,1), float2(1,1) }; + Instance item = instances[instanceID]; + float2 corner = corners[vertexID]; + float size = item.placement.z; + float cycle = fract(item.placement.y + u.time / item.motion.x); + // A cosine lift keeps the volume fully visible and reverses gently at + // the vertical safe bounds instead of cropping or fading at the edge. + float rise = 0.5 - 0.5 * cos(cycle * TAU); + // Reserve about 3% of the display height beyond the rotating quad. This + // clears the menu-bar/dock occlusion as well as the raw pixel boundary. + float margin = size * 0.54 + max(8.0, u.viewport.y * 0.03); + float2 center = float2(item.placement.x * u.viewport.x, + mix(margin, u.viewport.y - margin, rise)); + center.x += sin(u.time * 0.18 + item.placement.y * TAU) * item.motion.z; + center.x = clamp(center.x, margin, u.viewport.x - margin); + float2 ndc = float2(center.x / u.viewport.x * 2.0 - 1.0, + center.y / u.viewport.y * 2.0 - 1.0); + ndc += corner * float2(size / u.viewport.x, size / u.viewport.y); + Raster out; + out.position = float4(ndc, 0, 1); + out.local = corner * 1.73; + out.placement = item.placement; + out.motion = item.motion; + out.appearance = item.appearance; + return out; + } + + fragment float4 metaballFragment(Raster in [[stage_in]], constant Uniforms &u [[buffer(0)]]) { + float direction = in.appearance.y; + float seed = in.appearance.z; + float turn = direction * (u.time / in.motion.y * TAU) + in.placement.y * TAU; + // Non-commensurate angular rates prevent the mark from remaining upright + // or repeating a simple front/back flip. + float3 angle = float3(turn * 0.67 + seed, turn, turn * 0.39 - seed * 0.6); + float3 ro = inverseTurn(float3(in.local, 3.0), angle); + float3 rd = inverseTurn(float3(0, 0, -1), angle); + + // Skip empty corners analytically before entering the implicit field. + float b = dot(ro, rd); + float c = dot(ro, ro) - 1.72 * 1.72; + float discriminant = b * b - c; + if (discriminant < 0.0) discard_fragment(); + float traveled = max(0.0, -b - sqrt(discriminant)); + float3 p = ro + rd * traveled; + float blur = in.motion.w; + float hitEpsilon = mix(0.0045, 0.012, blur); + uint stepLimit = blur > 0.75 ? 26 : (blur > 0.1 ? 32 : 40); + bool hit = false; + for (uint step = 0; step < stepLimit; ++step) { + float d = field(p); + if (d < hitEpsilon) { hit = true; break; } + traveled += max(d * 0.74, hitEpsilon); + if (traveled > 6.0) break; + p = ro + rd * traveled; + } + if (!hit) discard_fragment(); + + float e = mix(0.006, 0.016, blur); + float3 normal = normalize(float3( + field(p + float3(e,0,0)) - field(p - float3(e,0,0)), + field(p + float3(0,e,0)) - field(p - float3(0,e,0)), + field(p + float3(0,0,e)) - field(p - float3(0,0,e)) + )); + float3 worldNormal = rotateX(normal, angle.x); + worldNormal = rotateY(worldNormal, angle.y); + worldNormal = rotateZ(worldNormal, angle.z); + float3 light = normalize(float3(-0.48, 0.72, 0.62)); + float3 fill = normalize(float3(0.68, -0.22, 0.70)); + float diffuse = max(dot(worldNormal, light), 0.0); + float fillLight = max(dot(worldNormal, fill), 0.0); + float specular = pow(max(dot(worldNormal, normalize(light + float3(0,0,1))), 0.0), + mix(52.0, 18.0, blur)); + float rim = pow(1.0 - abs(worldNormal.z), 2.25); + + int variant = int(in.appearance.x + 0.5); + float baseValue; + if (u.dark > 0.5) + baseValue = variant == 0 ? 0.82 : (variant == 1 ? 0.48 : 0.68); + else + baseValue = variant == 0 ? 0.035 : (variant == 1 ? 0.72 : 0.22); + float lightShape = 0.25 + diffuse * 0.70 + fillLight * 0.18; + lightShape = mix(lightShape, 0.54 + diffuse * 0.36, blur * 0.72); + float value = baseValue * lightShape; + value += rim * mix(u.dark > 0.5 ? 0.42 : 0.30, 0.18, blur) + + specular * mix(0.92, 0.46, blur); + float3 color = float3(clamp(value, 0.0, 1.0)); + return float4(color, in.placement.w); + } + """# +} + +// Dimensional Fuser backdrop. The production SVG's twelve nodes become one +// live smooth-union surface, tumbling through yaw, pitch, and roll in Metal. +private final class FuserDimensionalView: NSView { + private struct MarkSpec { + let x: CGFloat + let phase: CGFloat + let size: CGFloat + let riseSeconds: TimeInterval + let turnSeconds: TimeInterval + let sway: CGFloat + let variant: Int + } + + private static let specs = [ + // A fixed staggered 4×4 field. Small phase offsets soften the rows while + // the shared 48-second rise keeps every protected gap invariant. + MarkSpec(x: 0.07, phase: 0.04, size: 138, riseSeconds: 48, turnSeconds: 25, sway: 12, variant: 0), + MarkSpec(x: 0.34, phase: 0.08, size: 48, riseSeconds: 48, turnSeconds: 18, sway: 9, variant: 1), + MarkSpec(x: 0.61, phase: 0.02, size: 92, riseSeconds: 48, turnSeconds: 22, sway: 11, variant: 2), + MarkSpec(x: 0.88, phase: 0.06, size: 60, riseSeconds: 48, turnSeconds: 20, sway: 9, variant: 0), + + MarkSpec(x: 0.18, phase: 0.29, size: 52, riseSeconds: 48, turnSeconds: 17, sway: 9, variant: 1), + MarkSpec(x: 0.45, phase: 0.33, size: 116, riseSeconds: 48, turnSeconds: 28, sway: 12, variant: 2), + MarkSpec(x: 0.72, phase: 0.27, size: 44, riseSeconds: 48, turnSeconds: 16, sway: 8, variant: 0), + MarkSpec(x: 0.95, phase: 0.31, size: 82, riseSeconds: 48, turnSeconds: 23, sway: 10, variant: 1), + + MarkSpec(x: 0.06, phase: 0.54, size: 76, riseSeconds: 48, turnSeconds: 21, sway: 10, variant: 2), + MarkSpec(x: 0.31, phase: 0.58, size: 46, riseSeconds: 48, turnSeconds: 15, sway: 8, variant: 0), + MarkSpec(x: 0.58, phase: 0.52, size: 146, riseSeconds: 48, turnSeconds: 31, sway: 13, variant: 1), + MarkSpec(x: 0.84, phase: 0.56, size: 58, riseSeconds: 48, turnSeconds: 19, sway: 9, variant: 2), + + MarkSpec(x: 0.16, phase: 0.79, size: 48, riseSeconds: 48, turnSeconds: 16, sway: 8, variant: 0), + MarkSpec(x: 0.42, phase: 0.83, size: 104, riseSeconds: 48, turnSeconds: 27, sway: 11, variant: 1), + MarkSpec(x: 0.68, phase: 0.77, size: 64, riseSeconds: 48, turnSeconds: 20, sway: 9, variant: 2), + MarkSpec(x: 0.93, phase: 0.81, size: 126, riseSeconds: 48, turnSeconds: 29, sway: 12, variant: 0), + ] + + private let background = CAGradientLayer() + private let glowA = CAGradientLayer() + private let glowB = CAGradientLayer() + private let ambient = CALayer() + private let metaballRenderer = MetaballRenderer() + private let cardLayer = CALayer() + private let cardMark = CALayer() + private let cardTitle = CATextLayer() + private let metaballSheets: [CGImage] + private let cardURL = FileManager.default.homeDirectoryForCurrentUser + .appendingPathComponent(".local/share/captutor/wallpaper-card.json") + private var card = CardState.ambient + private var cardPayload = Data() + private var timer: Timer? + private var lastSize = CGSize.zero + + override init(frame frameRect: NSRect) { + metaballSheets = (0..<3).map(Self.loadMetaballSheet(variant:)) + super.init(frame: frameRect) + wantsLayer = true + let root = CALayer() + root.masksToBounds = true + layer = root + + background.startPoint = CGPoint(x: 0.04, y: 0.94) + background.endPoint = CGPoint(x: 0.96, y: 0.06) + root.addSublayer(background) + + for glow in [glowA, glowB] { + glow.type = .radial + glow.startPoint = CGPoint(x: 0.5, y: 0.5) + glow.endPoint = CGPoint(x: 0.96, y: 0.96) + glow.locations = [0, 0.48, 1] + root.addSublayer(glow) + } + root.addSublayer(ambient) + if let metaballRenderer { + root.addSublayer(metaballRenderer.layer) + } + + cardLayer.opacity = 0 + cardLayer.addSublayer(cardMark) + cardTitle.alignmentMode = .center + cardTitle.isWrapped = true + cardTitle.contentsScale = NSScreen.main?.backingScaleFactor ?? 2 + cardTitle.font = NSFont.systemFont(ofSize: 60, weight: .black) + cardLayer.addSublayer(cardTitle) + root.addSublayer(cardLayer) + + applyAppearance() + timer = Timer.scheduledTimer(withTimeInterval: 0.20, repeats: true) { [weak self] _ in + self?.reloadCard() + } + } + + required init?(coder: NSCoder) { nil } + deinit { timer?.invalidate() } + + override func viewDidChangeEffectiveAppearance() { + super.viewDidChangeEffectiveAppearance() + applyAppearance() + } + + override func layout() { + super.layout() + background.frame = bounds + let glowSize = max(bounds.width, bounds.height) * 0.82 + glowA.frame = CGRect(x: -glowSize * 0.38, y: bounds.height - glowSize * 0.56, + width: glowSize, height: glowSize) + glowB.frame = CGRect(x: bounds.width - glowSize * 0.62, y: -glowSize * 0.42, + width: glowSize, height: glowSize) + metaballRenderer?.setLayout(bounds: bounds, + backingScale: window?.backingScaleFactor + ?? NSScreen.main?.backingScaleFactor ?? 2, + specs: Self.specs.map { + ($0.x, $0.phase, $0.size, $0.riseSeconds, $0.turnSeconds, $0.sway, $0.variant) + }) + cardLayer.frame = bounds + let portrait = bounds.width < bounds.height + let markSize: CGFloat = portrait ? 150 : 170 + cardMark.frame = CGRect(x: bounds.midX - markSize / 2, + y: bounds.midY + (portrait ? 55 : 25), + width: markSize, height: markSize) + cardTitle.frame = CGRect(x: bounds.width * 0.09, + y: bounds.midY - (portrait ? 230 : 195), + width: bounds.width * 0.82, + height: portrait ? 235 : 210) + guard bounds.size != lastSize else { return } + lastSize = bounds.size + rebuildAmbient() + } + + private func rebuildAmbient() { + ambient.sublayers?.forEach { $0.removeFromSuperlayer() } + ambient.frame = bounds + guard metaballRenderer == nil else { return } + let now = CACurrentMediaTime() + + for spec in Self.specs { + let scale = min(max(bounds.width / 1280, 0.78), 1.30) + let size = spec.size * scale + let travel = CALayer() + travel.bounds = CGRect(x: 0, y: 0, width: size, height: size) + travel.opacity = spec.size > 100 ? 0.72 : 0.58 + travel.allowsEdgeAntialiasing = true + + let solid = CALayer() + solid.frame = travel.bounds + solid.anchorPoint = CGPoint(x: 0.5, y: 0.5) + solid.position = CGPoint(x: size / 2, y: size / 2) + travel.addSublayer(solid) + + let turnStart = now - spec.turnSeconds * Double(spec.phase) + let sheet = metaballSheets[spec.variant] + let sprite = CALayer() + sprite.frame = solid.bounds + sprite.contents = sheet + sprite.contentsGravity = .resizeAspectFill + sprite.contentsScale = 2 + solid.addSublayer(sprite) + let turn = CABasicAnimation(keyPath: "transform.rotation.z") + turn.fromValue = spec.variant == 1 ? CGFloat.pi * 2 : 0 + turn.toValue = spec.variant == 1 ? 0 : CGFloat.pi * 2 + turn.duration = spec.turnSeconds + turn.repeatCount = .infinity + turn.timingFunction = CAMediaTimingFunction(name: .linear) + turn.beginTime = turnStart + turn.isRemovedOnCompletion = false + solid.add(turn, forKey: "metaballTurn") + + let x = bounds.width * spec.x + // All marks traverse exactly the same vertical span and duration. + // Their relative layout therefore survives every wrap unchanged. + let travelMargin = 190 * scale + let lowY = -travelMargin + let highY = bounds.height + travelMargin + let path = CGMutablePath() + path.move(to: CGPoint(x: x, y: lowY)) + path.addCurve(to: CGPoint(x: x - spec.sway * 0.45, y: bounds.height * 0.44), + control1: CGPoint(x: x + spec.sway, y: bounds.height * 0.13), + control2: CGPoint(x: x - spec.sway, y: bounds.height * 0.31)) + path.addCurve(to: CGPoint(x: x, y: highY), + control1: CGPoint(x: x + spec.sway, y: bounds.height * 0.68), + control2: CGPoint(x: x - spec.sway * 0.55, y: bounds.height * 0.88)) + travel.position = CGPoint(x: x, y: lowY + (highY - lowY) * spec.phase) + let rise = CAKeyframeAnimation(keyPath: "position") + rise.path = path + rise.calculationMode = .paced + rise.duration = spec.riseSeconds + rise.repeatCount = .infinity + rise.beginTime = now - spec.riseSeconds * Double(spec.phase) + rise.isRemovedOnCompletion = false + travel.add(rise, forKey: "fuserRise") + + ambient.addSublayer(travel) + } + } + + private func applyAppearance() { + let dark = effectiveAppearance.bestMatch(from: [.darkAqua, .aqua]) == .darkAqua + metaballRenderer?.setAppearance(dark: dark) + CATransaction.begin() + CATransaction.setAnimationDuration(0.55) + background.colors = (dark ? [ + NSColor(hex: 0x050505), NSColor(hex: 0x141414), NSColor(hex: 0x080808), + ] : [ + NSColor(hex: 0xFFFFFF), NSColor(hex: 0xECECEC), NSColor(hex: 0xFAFAFA), + ]).map(\.cgColor) + let smoke = NSColor(hex: dark ? 0xFFFFFF : 0x111111) + let silver = NSColor(hex: dark ? 0xA8A8A8 : 0x8E8E8E) + glowA.colors = [smoke.withAlphaComponent(dark ? 0.12 : 0.09).cgColor, + smoke.withAlphaComponent(dark ? 0.045 : 0.032).cgColor, + smoke.withAlphaComponent(0).cgColor] + glowB.colors = [silver.withAlphaComponent(dark ? 0.10 : 0.08).cgColor, + silver.withAlphaComponent(dark ? 0.038 : 0.028).cgColor, + silver.withAlphaComponent(0).cgColor] + cardTitle.foregroundColor = (dark ? NSColor.white : NSColor(hex: 0x171717)).cgColor + cardMark.contents = metaballSheets[0] + cardMark.contentsGravity = .resizeAspect + cardMark.shadowColor = (dark ? NSColor.white : NSColor.black).cgColor + cardMark.shadowRadius = 28 + cardMark.shadowOpacity = 0.46 + CATransaction.commit() + } + + private func reloadCard() { + guard let data = try? Data(contentsOf: cardURL), data != cardPayload, + let decoded = try? JSONDecoder().decode(CardState.self, from: data) else { return } + cardPayload = data + card = decoded + cardTitle.string = decoded.title ?? "" + CATransaction.begin() + CATransaction.setAnimationDuration(0.48) + ambient.opacity = decoded.phase == "ambient" ? 1 : 0.10 + metaballRenderer?.setOpacity(decoded.phase == "ambient" ? 1 : 0.10) + cardLayer.opacity = decoded.phase == "ambient" ? 0 : 1 + CATransaction.commit() + } + + private static func loadMetaballSheet(variant: Int) -> CGImage { + guard let url = Bundle.main.url(forResource: "fuser-metaballs-\(variant)", withExtension: "png"), + let image = NSImage(contentsOf: url), + let cgImage = image.cgImage(forProposedRect: nil, context: nil, hints: nil) else { + fatalError("Captutor Wallpaper is missing fuser-metaballs-\(variant).png") + } + return cgImage + } +} + +private extension NSColor { + convenience init(hex: UInt32) { + self.init(srgbRed: CGFloat((hex >> 16) & 0xFF) / 255, + green: CGFloat((hex >> 8) & 0xFF) / 255, + blue: CGFloat(hex & 0xFF) / 255, + alpha: 1) + } +} + private final class AppDelegate: NSObject, NSApplicationDelegate { private var windows: [NSWindow] = [] func applicationDidFinishLaunching(_ notification: Notification) { + let args = CommandLine.arguments + let brandIndex = args.firstIndex(of: "--brand") + let brand = brandIndex.flatMap { index in + args.indices.contains(index + 1) ? args[index + 1].lowercased() : nil + } ?? "fuser" + let prototype = args.contains("--prototype") for screen in NSScreen.screens { let window = NSWindow( contentRect: screen.frame, @@ -263,15 +794,24 @@ backing: .buffered, defer: false, screen: screen ) - window.level = NSWindow.Level(rawValue: Int(CGWindowLevelForKey(.desktopWindow)) + 1) + window.level = NSWindow.Level(rawValue: prototype + ? Int(CGWindowLevelForKey(.desktopIconWindow)) - 1 + : Int(CGWindowLevelForKey(.desktopWindow)) + 1) window.collectionBehavior = [.canJoinAllSpaces, .stationary, .ignoresCycle] window.ignoresMouseEvents = true window.isOpaque = true window.hasShadow = false - window.contentView = WallpaperView(frame: NSRect(origin: .zero, size: screen.frame.size)) + if brand == "classic" { + window.contentView = WallpaperView(frame: NSRect(origin: .zero, size: screen.frame.size)) + } else { + window.contentView = FuserDimensionalView(frame: NSRect(origin: .zero, size: screen.frame.size)) + } window.orderFrontRegardless() windows.append(window) } + print("Captutor Wallpaper brand=\(brand) prototype=\(prototype) " + + "renderer=\(brand == "classic" ? "vector-field" : "realtime-metal-metaballs")") + fflush(stdout) } } diff --git a/captutor/bin/cdp-frame.mjs b/captutor/bin/cdp-frame.mjs new file mode 100644 --- /dev/null +++ b/captutor/bin/cdp-frame.mjs @@ -0,0 +1,52 @@ +#!/usr/bin/env node +// A bounded, invisible browser preflight for Captutor pathfinding. +// +// One CDP connection returns the current URL, visible controls, stable locator +// candidates, focus, viewport, and React Flow nodes/handles/edges. Optionally +// save a compositor screenshot without putting any Frame/Puppet UI on screen. + +import { writeFileSync } from "node:fs"; +import { resolve } from "node:path"; +import { withSession } from "../lib/cdp.mjs"; + +const args = process.argv.slice(2); +const value = (flag, fallback = null) => { + const index = args.indexOf(flag); + return index >= 0 ? args[index + 1] : fallback; +}; +const match = value("--match", "fuser.studio"); +const screenshotPath = value("--screenshot"); +const compact = args.includes("--compact"); + +if (args.includes("--help") || args.includes("-h")) { + console.log("usage: node bin/cdp-frame.mjs [--match text] [--screenshot path.png] [--compact]"); + process.exit(0); +} + +const timeoutMs = Number(value("--timeout", "15000")); +const watchdog = setTimeout(() => { + console.error(`CDP frame timed out after ${timeoutMs}ms`); + process.exit(124); +}, timeoutMs); +try { + const result = await withSession(match, async (cdp) => { + const frame = await cdp.frame(); + if (screenshotPath) { + const out = resolve(screenshotPath); + writeFileSync(out, await cdp.screenshot()); + frame.screenshot = out; + } + return frame; + }); + console.log(JSON.stringify(result, null, compact ? 0 : 2)); +} catch (error) { + console.error(JSON.stringify({ + ok:false, + code:error.code || "CDP_FRAME_FAILED", + message:error.message, + details:error.details || null, + })); + process.exitCode = 1; +} finally { + clearTimeout(watchdog); +} diff --git a/captutor/bin/install.sh b/captutor/bin/install.sh --- a/captutor/bin/install.sh +++ b/captutor/bin/install.sh @@ -13,7 +13,10 @@ "$SOURCE/" "$DEST/" mkdir -p "$HOME/.local/bin" "$HOME/Desktop/outbox" install -m 755 "$SOURCE/vendor/reel.mjs" "$HOME/.local/bin/reel.mjs" +/usr/bin/swiftc -O "$SOURCE/bin/captutor-pointer.swift" -o "$HOME/.local/bin/captutor-pointer" +/usr/bin/swiftc -O "$SOURCE/bin/captutor-cursor.swift" -o "$HOME/.local/bin/captutor-cursor" echo "✓ Captutor installed at $DEST" echo "✓ reel controller installed at $HOME/.local/bin/reel.mjs" +echo "✓ native tutorial cursor installed at $HOME/.local/bin/captutor-cursor" echo "✓ delivery outbox at $HOME/Desktop/outbox" diff --git a/captutor/bin/render-fuser-metaballs.swift b/captutor/bin/render-fuser-metaballs.swift new file mode 100644 --- /dev/null +++ b/captutor/bin/render-fuser-metaballs.swift @@ -0,0 +1,189 @@ +// Bake the canonical twelve-node Fuser mark as a smooth 3D metaball surface. +// Runtime Captutor rotates these stills; raymarching never runs while filming. + +import AppKit +import Foundation +import simd + +guard CommandLine.arguments.count == 2 else { + fputs("usage: render-fuser-metaballs \n", stderr) + exit(2) +} + +private let output = URL(fileURLWithPath: CommandLine.arguments[1], isDirectory: true) +// One generous source image lets Core Animation provide continuous rotation +// without atlas stepping or the memory cost of hundreds of decoded views. +private let frameSize = 384 +private let columns = 1 +private let rows = 1 +private let frameCount = columns * rows +private let sheetWidth = columns * frameSize +private let sheetHeight = rows * frameSize + +private struct Palette { + let base: SIMD3 + let accent: SIMD3 + let edge: SIMD3 +} + +private let palettes = [ + // Obsidian, pearl, and graphite remain strictly neutral. Their different + // values make overlapping marks legible without leaving Fuser's B/W system. + Palette(base: SIMD3(repeating: 0.035), accent: SIMD3(repeating: 1.00), edge: SIMD3(repeating: 0.005)), + Palette(base: SIMD3(repeating: 0.76), accent: SIMD3(repeating: 1.00), edge: SIMD3(repeating: 0.16)), + Palette(base: SIMD3(repeating: 0.22), accent: SIMD3(repeating: 0.94), edge: SIMD3(repeating: 0.018)), +] + +// Centers are taken directly from captutor/assets/fuser-mark.svg. Normalizing +// around its 24×24 viewBox preserves the production mark's proportions. +private let svgCenters: [(Float, Float)] = [ + (15.1507, 2.5838), (21.4201, 2.5838), (8.8818, 2.5838), + (8.8818, 8.86312), (2.58145, 8.83282), (2.61171, 15.1369), + (2.61171, 21.4162), (8.8818, 21.4162), (15.1507, 21.4162), + (15.1507, 15.1369), (21.4201, 15.1369), (21.4201, 8.86312), +] +private let sourceCenters: [SIMD3] = svgCenters.enumerated().map { index, point in + let x = (point.0 - 12) / 7.75 + let y = (12 - point.1) / 7.75 + // A tiny alternating depth keeps the front silhouette faithful while the + // side view reveals that this is a living cluster, not a flat extrusion. + let z = sin(Float(index) * 1.71) * 0.055 + return SIMD3(x, y, z) +} + +@Sendable @inline(__always) private func rotate(_ point: SIMD3, yaw: Float, pitch: Float) -> SIMD3 { + let cy = cos(yaw), sy = sin(yaw) + let cp = cos(pitch), sp = sin(pitch) + let yTurned = SIMD3(point.x * cy + point.z * sy, point.y, -point.x * sy + point.z * cy) + return SIMD3(yTurned.x, yTurned.y * cp - yTurned.z * sp, + yTurned.y * sp + yTurned.z * cp) +} + +@Sendable @inline(__always) private func smoothMin(_ a: Float, _ b: Float, _ blend: Float) -> Float { + let h = max(blend - abs(a - b), 0) / blend + return min(a, b) - h * h * blend * 0.25 +} + +@Sendable @inline(__always) private func fieldDistance(_ point: SIMD3, centers: [SIMD3]) -> Float { + let radius: Float = 0.355 + var distance = simd_length(point - centers[0]) - radius + for center in centers.dropFirst() { + distance = smoothMin(distance, simd_length(point - center) - radius, 0.30) + } + return distance +} + +@Sendable @inline(__always) private func surfaceNormal(_ point: SIMD3, centers: [SIMD3]) -> SIMD3 { + let epsilon: Float = 0.005 + let x = fieldDistance(point + SIMD3(epsilon, 0, 0), centers: centers) + - fieldDistance(point - SIMD3(epsilon, 0, 0), centers: centers) + let y = fieldDistance(point + SIMD3(0, epsilon, 0), centers: centers) + - fieldDistance(point - SIMD3(0, epsilon, 0), centers: centers) + let z = fieldDistance(point + SIMD3(0, 0, epsilon), centers: centers) + - fieldDistance(point - SIMD3(0, 0, epsilon), centers: centers) + return simd_normalize(SIMD3(x, y, z)) +} + +@Sendable @inline(__always) private func clamp01(_ value: Float) -> Float { min(max(value, 0), 1) } +@Sendable @inline(__always) private func mix(_ a: SIMD3, _ b: SIMD3, _ t: Float) -> SIMD3 { + a + (b - a) * clamp01(t) +} + +@Sendable private func shade(pixelX: Int, pixelY: Int, centers: [SIMD3], palette: Palette) -> SIMD4 { + let u = (Float(pixelX) + 0.5) / Float(frameSize) + let v = (Float(pixelY) + 0.5) / Float(frameSize) + let viewScale: Float = 3.45 + var point = SIMD3((u - 0.5) * viewScale, (0.5 - v) * viewScale, 3.0) + let ray = SIMD3(0, 0, -1) + var traveled: Float = 0 + var hit = false + + for _ in 0..<88 { + let distance = fieldDistance(point, centers: centers) + if distance < 0.0035 { + hit = true + break + } + let advance = max(distance * 0.72, 0.006) + traveled += advance + if traveled > 6.2 { break } + point += ray * advance + } + guard hit else { return SIMD4(0, 0, 0, 0) } + + let normal = surfaceNormal(point, centers: centers) + let light = simd_normalize(SIMD3(-0.48, 0.72, 0.62)) + let fill = simd_normalize(SIMD3(0.68, -0.22, 0.70)) + let view = SIMD3(0, 0, 1) + let diffuse = max(simd_dot(normal, light), 0) + let fillLight = max(simd_dot(normal, fill), 0) + let halfVector = simd_normalize(light + view) + let specular = pow(max(simd_dot(normal, halfVector), 0), 54) + let rim = pow(1 - max(simd_dot(normal, view), 0), 2.35) + let facing = clamp01(normal.z * 0.5 + 0.5) + var color = mix(palette.edge, palette.base, 0.28 + facing * 0.72) + color *= 0.24 + diffuse * 0.71 + fillLight * 0.18 + color += palette.accent * (rim * 0.42) + color += SIMD3(repeating: 1) * (specular * 0.88) + + // Contact darkening where neighboring balls merge helps the lobes remain + // legible without drawing seams into the implicit surface. + let nearest = centers.map { simd_length(point - $0) }.sorted() + if nearest.count > 1 { + let junction = clamp01((0.53 - nearest[1]) * 2.6) + color *= 1 - junction * 0.14 + } + + return SIMD4(UInt8(clamp01(color.x) * 255), + UInt8(clamp01(color.y) * 255), + UInt8(clamp01(color.z) * 255), 255) +} + +try FileManager.default.createDirectory(at: output, withIntermediateDirectories: true) +let rotatedCenters: [[SIMD3]] = (0...allocate(capacity: byteCount) + pixels.initialize(repeating: 0, count: byteCount) + + DispatchQueue.concurrentPerform(iterations: sheetHeight) { outputY in + let cellYFromTop = outputY / frameSize + let localY = outputY % frameSize + // Frame zero occupies the bottom sheet row, matching CALayer contentsRect. + let frameRow = rows - 1 - cellYFromTop + for outputX in 0..?] = [pixels] + guard let bitmap = NSBitmapImageRep(bitmapDataPlanes: &planes, + pixelsWide: sheetWidth, pixelsHigh: sheetHeight, + bitsPerSample: 8, samplesPerPixel: 4, + hasAlpha: true, isPlanar: false, + colorSpaceName: .deviceRGB, + bitmapFormat: .alphaNonpremultiplied, + bytesPerRow: sheetWidth * 4, bitsPerPixel: 32), + let png = bitmap.representation(using: .png, properties: [.compressionFactor: 0.86]) else { + pixels.deallocate() + fatalError("could not encode metaball sheet") + } + let destination = output.appendingPathComponent("fuser-metaballs-\(variant).png") + try png.write(to: destination) + pixels.deallocate() + print("✓ \(destination.lastPathComponent) · \(png.count / 1024) KB") +} diff --git a/captutor/bin/stage.mjs b/captutor/bin/stage.mjs --- a/captutor/bin/stage.mjs +++ b/captutor/bin/stage.mjs @@ -6,16 +6,21 @@ import { existsSync, readFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { dirname, join, resolve } from "node:path"; import { fileURLToPath } from "node:url"; -import { enterStageMode, exitStageMode } from "../lib/stage-mode.mjs"; +import { enterStageMode, exitStageMode, parseStageFlags } from "../lib/stage-mode.mjs"; const HERE = dirname(fileURLToPath(import.meta.url)); const REEL = resolve(HERE, "../vendor/reel.mjs"); const REEL_STATE = join(process.env.HOME, ".local", "share", "slab", "state", "reel.state"); const rawArgs = process.argv.slice(2); -const vertical = rawArgs.includes("--vertical"); -const args = rawArgs.filter((arg) => arg !== "--vertical"); +let parsed; +try { parsed = parseStageFlags(rawArgs); } +catch (error) { + console.error(error.message); + process.exit(2); +} +const { vertical, brand, args } = parsed; if (!args.length) { - console.error("usage: node bin/stage.mjs [--vertical] render [captutor options]"); + console.error("usage: node bin/stage.mjs [--vertical] [--brand fuser|classic] render [captutor options]"); process.exit(2); } @@ -28,6 +33,34 @@ }; process.on("SIGINT", () => forward("SIGINT")); process.on("SIGTERM", () => forward("SIGTERM")); +function verifyFilmingPermissions() { + const recorder = spawnSync(process.execPath, [REEL, "status"], { + encoding: "utf8", + env: process.env, + }); + if (recorder.status !== 0) { + throw new Error( + `SlabMenubar recording bridge is unavailable: ${(recorder.stderr || recorder.stdout || "").trim()}`, + ); + } + let status; + try { status = JSON.parse(recorder.stdout || "{}"); } + catch { throw new Error("SlabMenubar recording bridge returned invalid status"); } + if (typeof status.recording !== "boolean") { + throw new Error("SlabMenubar recording bridge did not report recording permission state"); + } + + const accessibility = spawnSync("/usr/bin/osascript", [ + "-e", 'tell application "System Events" to get UI elements enabled', + ], { encoding:"utf8", env:process.env }); + if (accessibility.status !== 0 || accessibility.stdout.trim() !== "true") { + throw new Error( + "System Events Accessibility is not enabled for Captutor; allow it in System Settings before filming", + ); + } + console.log("✓ filming permissions — SlabMenubar recorder + System Events Accessibility"); +} + function stopOwnedReelIfNeeded() { if (!child || !existsSync(REEL_STATE)) return; let state; @@ -46,9 +79,10 @@ } let code = 1; try { + verifyFilmingPermissions(); // Enter is inside the guarded region deliberately: if a preference change // fails halfway through, the state file still lets `finally` unwind it. - await enterStageMode({ vertical }); + await enterStageMode({ vertical, brand }); if (vertical) { // Rotation can leave Chrome's process alive with no page window. Relaunch // the dedicated filming profile only when its Fuser target disappeared. @@ -68,9 +102,10 @@ env: { ...process.env, CAPTUTOR_STAGE_MODE: "1", CAPTUTOR_VERTICAL_MODE: vertical ? "1" : "0", - // Accessibility-free filming seats can retain Captutor's deterministic - // in-page pointer while still using the full native Stage desktop. - CAPTUTOR_REAL_CURSOR: process.env.CAPTUTOR_REAL_CURSOR ?? "1", + CAPTUTOR_BRAND: brand, + // The capture-visible Swift overlay is the normal filmed pointer. Keep + // the system cursor opt-in for explicitly human-driven takes only. + CAPTUTOR_REAL_CURSOR: process.env.CAPTUTOR_REAL_CURSOR ?? "0", CDP_PORT: process.env.CDP_PORT || "9333", PATH: `/opt/homebrew/bin:${process.env.HOME}/.local/bin:/usr/bin:/bin:/usr/sbin:/sbin`, }, diff --git a/captutor/captutor.mjs b/captutor/captutor.mjs --- a/captutor/captutor.mjs +++ b/captutor/captutor.mjs @@ -37,8 +37,10 @@ import { dirname, join, resolve, basename } from "node:path"; import { fileURLToPath, pathToFileURL } from "node:url"; import { narrate } from "./lib/narrate.mjs"; -import { attach } from "./lib/cdp.mjs"; -import { clickOn, dragBetween, pointAt, typeInto, INSTALL } from "./lib/cursor.mjs"; +import { attach, BrowserCrashError } from "./lib/cdp.mjs"; +import { + clickOn, dragBetween, pointAt, stopNativeCursor, typeInto, +} from "./lib/cursor.mjs"; import { spotlight, outline, burst, zoom, resetCamera, clearEffects, } from "./lib/effects.mjs"; @@ -50,6 +52,7 @@ import { ensureSignedIn, WORKSPACE } from "./lib/login.mjs"; import * as credits from "./lib/credits.mjs"; import { publishToOutbox } from "./lib/outbox.mjs"; import { presentSignboard, setAmbient } from "./lib/signboard.mjs"; +import { assertHiDPIStage } from "./lib/stage-contract.mjs"; const HERE = dirname(fileURLToPath(import.meta.url)); @@ -60,6 +63,12 @@ // path is an env override. Iris runs on panda; this is what lets her film. const INSTALLED_REEL = join(process.env.HOME, ".local", "bin", "reel.mjs"); const REEL = process.env.CAPTUTOR_REEL || (existsSync(INSTALLED_REEL) ? INSTALLED_REEL : join(resolve(HERE, "../../.."), "slab", "bin", "reel.mjs")); +const INSTALLED_FRAME = join(process.env.HOME, ".local", "bin", "frame.mjs"); +const REPO_FRAME = join(resolve(HERE, ".."), "slab", "bin", "frame.mjs"); +const FRAME = process.env.CAPTUTOR_FRAME + || (existsSync(INSTALLED_FRAME) ? INSTALLED_FRAME + : existsSync(REPO_FRAME) ? REPO_FRAME + : join(resolve(HERE, "../../.."), "slab", "bin", "frame.mjs")); const FUSER = process.env.FUSER_REPO || `${process.env.HOME}/Developer/fuser`; const DOCS_PUBLIC = join(FUSER, "apps", "docs", "public"); @@ -67,6 +76,23 @@ const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); const now = () => Date.now() / 1000; const REAL_CURSOR = process.env.CAPTUTOR_REAL_CURSOR === "1"; const STAGE_MODE = process.env.CAPTUTOR_STAGE_MODE === "1"; +// CAPTUTOR_TASK_GID is present on every Iris mission, including a worker that +// was already running when this invariant was deployed. Local development +// renders remain possible without Stage; fleet takes do not. +const REQUIRE_HIDPI = process.env.CAPTUTOR_REQUIRE_HIDPI === "1" + || Boolean(process.env.CAPTUTOR_TASK_GID); +const VERTICAL_MODE = process.env.CAPTUTOR_VERTICAL_MODE === "1"; + +// Frame's native OCR/target surfaces live above app windows, so even a window +// capture can film one. Retire every Frame-owned transient immediately before +// the reel starts. If Frame is installed, failure is a capture-safety failure: +// it is better to abort a take than ship tooling UI inside the tutorial. +function clearFrameOverlays() { + if (!existsSync(FRAME)) return; + execFileSync(process.execPath, [ + FRAME, "local", "--clear-overlays", "--quiet-overlay", "--no-ocr", "--json", + ], { encoding: "utf8", timeout: 10_000, stdio: ["ignore", "pipe", "pipe"] }); +} // Some filming seats (notably clamshell Macs on native-only external panels) // cannot expose Captutor's usual 2560×1440/1280×720 HiDPI pair. Keep those @@ -153,6 +179,27 @@ appendFileSync(FAILURE_LOG, `${JSON.stringify(record)}\n`); return record; } +function logBrowserFailure({ sp, locale, format, attempt, error, beat, elapsed, aborted }) { + mkdirSync(dirname(FAILURE_LOG), { recursive: true }); + const record = { + schema:"captutor-failure/v1", + at:new Date().toISOString(), + screenplay:sp.slug, + locale, + format, + attempt, + reason:"browser-renderer-crash", + message:error.message, + beat:Math.max(0, beat) + 1, + elapsedSec:Number(elapsed.toFixed(3)), + abortedVideo:aborted, + signal:error.details?.signal || null, + action:"abort-and-restart-browser", + }; + appendFileSync(FAILURE_LOG, `${JSON.stringify(record)}\n`); + return record; +} + // `since` is the load-bearing value: the wall-clock instant the recorder's first // frame exists, on the same machine and the same epoch as our own Date.now(). // Every beat offset is measured against it, so audio and video share an origin. @@ -247,6 +294,11 @@ await new Promise((r) => setTimeout(r, 900)); // let the layout settle } async function cmdRender(sp, workDir, locale, format, attempt = 1) { + assertHiDPIStage({ + required:REQUIRE_HIDPI, + stageMode:STAGE_MODE, + vertical:VERTICAL_MODE, + }); const beats = await cmdNarrate(sp, workDir, locale); const t = translator(locale); const s = selectors(t); @@ -279,14 +331,23 @@ ["Overlay.setShowViewportSizeOnResize", { show: false }], ["Overlay.setShowAdHighlights", { show: false }], ].map(([method, params]) => cdp.send(method, params).catch(() => {}))); - if (REAL_CURSOR) { - await cdp.eval(`(() => { - document.getElementById('__captutor_cursor')?.remove(); - delete window.__captutor; - })()`); - } else { - await cdp.eval(INSTALL); - } + // Puppet and the shared analysis layer draw directly into the page at the + // highest z-index. They normally self-fade, but a tutorial take must not + // depend on a timeout or on which automation client touched the tab last. + await cdp.eval(`(() => { + document.getElementById('__puppet_cursor')?.remove(); + document.getElementById('__analysis_overlay')?.remove(); + clearTimeout(window.__pcTimer); + delete window.__pcTimer; + })()`); + + // Captutor's visible pointer is a native click-through Swift surface. Remove + // any cursor left in the page by an older build; trusted input still travels + // through CDP and therefore remains independent from the presentation layer. + await cdp.eval(`(() => { + document.getElementById('__captutor_cursor')?.remove(); + delete window.__captutor; + })()`); // The screenplay says `click('[data-testid=fuse]')`, not // `click(cdp, '[data-testid=fuse]')` — the session is plumbing, and a @@ -437,6 +498,30 @@ // stutter the drawn cursor and can leave `reel` filming a stale surface. await cdp.send("Page.bringToFront"); await sleep(600); + clearFrameOverlays(); + + // /json retains the original Fuser URL and title after a renderer dies, so + // those fields are not a health check. Require the page itself to answer just + // before the camera starts; this catches a pre-existing "Aw, Snap!" without + // filming it or debiting a generation. + try { + await cdp.assertHealthy("pre-record"); + const screen = await cdp.eval(`({ + width: screen.width, + height: screen.height, + dpr: window.devicePixelRatio, + })`); + assertHiDPIStage({ + required:REQUIRE_HIDPI, + stageMode:STAGE_MODE, + vertical:VERTICAL_MODE, + screen, + }); + } catch (error) { + await cdp.close(); + throw error; + } + const stageDisplay = STAGE_MODE && F.compose?.fullDesktop; console.log(`\n● recording (${stageDisplay ? "full Stage desktop" : `window: ${sp.window || "whole display"}`})`); const state = reelStart({ @@ -456,6 +541,7 @@ let activeBeat = -1; const stopRecording = (out) => { if (!recording) return out; recording = false; + stopNativeCursor(); return reelStop(out); }; @@ -463,7 +549,7 @@ const take = (async () => { await sleep((sp.leadInMs ?? 700)); // a beat of stillness before we start moving if (sp.openingCard) { const card = { - phase: "title", ...localizeCard(sp.openingCard), + phase: "title", ...localizeCard(sp.openingCard), title: "Learn Fuser", }; await perform("signboard", { card, role:"opening" }, () => presentSignboard(cdp, card, { @@ -521,9 +607,17 @@ } return null; })(); + const browserGuard = (async () => { + while (recording) { + await cdp.assertHealthy(`recording beat ${Math.max(0, activeBeat) + 1}`); + await sleep(500); + } + return null; + })(); + let timed; try { - timed = await Promise.race([take, upgradeGuard]); + timed = await Promise.race([take, upgradeGuard, browserGuard]); } catch (err) { if (err instanceof UpgradeInterruption) { // The camera is already stopped. Let any in-flight screenplay promise @@ -542,6 +636,25 @@ } console.warn(`\n↻ logged and discarded interrupted take; retrying cleanly (${attempt}/${AUTO_RETRIES})`); await sleep(900); return cmdRender(sp, workDir, locale, format, attempt + 1); + } + + if (err instanceof BrowserCrashError) { + const stamp = new Date().toISOString().replaceAll(/[:.]/g, "-"); + const aborted = stopRecording(join(workDir, `aborted-browser-${stamp}.mp4`)); + logBrowserFailure({ + sp, locale, format, attempt, error:err, + beat:activeBeat, elapsed:now() - since, aborted, + }); + console.error(`\n✗ browser renderer crashed at beat ${Math.max(0, activeBeat) + 1}; take aborted`); + if (purse) { + await credits.settle(cdp, purse, { + slug:sp.slug, locale, format, aborted:true, reason:"browser-renderer-crash", + }).catch((settleError) => { + console.warn(` credit settlement unavailable after crash: ${settleError.message}`); + }); + } + await cdp.close(); + throw err; } console.error(`\n✗ beat ${activeBeat + 1} failed: ${err.message}`); diff --git a/captutor/lib/cdp.mjs b/captutor/lib/cdp.mjs --- a/captutor/lib/cdp.mjs +++ b/captutor/lib/cdp.mjs @@ -13,8 +13,22 @@ // click themselves with no pointer in sight reads as a bug. // (WebSocket is a Node >= 22 global — no import, no `ws` dependency.) +import { basename } from "node:path"; + const HOST = process.env.CDP_HOST || "127.0.0.1"; const PORT = process.env.CDP_PORT || "9222"; +const COMMAND_TIMEOUT_MS = Number(process.env.CAPTUTOR_CDP_COMMAND_TIMEOUT_MS || 15_000); +const HEALTH_TIMEOUT_MS = Number(process.env.CAPTUTOR_CDP_HEALTH_TIMEOUT_MS || 4_000); +const CRASH_EVENTS = new Set(["Inspector.targetCrashed", "Target.targetCrashed"]); + +export class BrowserCrashError extends Error { + constructor(message, details = {}) { + super(message); + this.name = "BrowserCrashError"; + this.code = "BROWSER_RENDERER_CRASH"; + this.details = details; + } +} export async function attach(urlMatch) { const list = await (await fetch(`http://${HOST}:${PORT}/json`)).json(); @@ -27,38 +41,160 @@ throw new Error( `no CDP page${urlMatch ? ` matching "${urlMatch}"` : ""}. open pages:\n` + pages.map((p) => ` ${p.url}`).join("\n")); } - return new Session(target.webSocketDebuggerUrl); + const session = new Session(target.webSocketDebuggerUrl); + try { + await session.monitorCrashes(); + return session; + } catch (error) { + await session.close(); + throw error; + } } -class Session { +// Short-lived inspection should always use this wrapper. A bare `attach()` in +// a one-off node script leaves the WebSocket holding the process open, which +// turns a two-second preflight into a tool timeout. Long-running screenplays +// still own their Session directly and close it in Captutor's render teardown. +export async function withSession(urlMatch, action) { + const session = await attach(urlMatch); + try { return await action(session); } + finally { await session.close(); } +} + +export class Session { constructor(wsUrl) { this.wsUrl = wsUrl; this.id = 0; this.pending = new Map(); + this.crashEvent = null; + this.closing = false; + const entrypoint = process.argv[1] ? basename(process.argv[1]) : ""; + // Claude frequently explores with `node -e` or a throwaway *probe*. Those + // scripts used to print their answer and then live forever because nobody + // closed CDP. Give only those explicitly ephemeral entrypoints a brief idle + // reaper; production captutor.mjs sessions remain fully caller-owned. + this.ephemeralIdleMs = ( + process.env.CAPTUTOR_CDP_EPHEMERAL === "1" + || !entrypoint + || /(?:^|[-_.])probe/i.test(entrypoint) + ) ? 5_000 : 0; + this.idleTimer = null; this.ready = new Promise((res, rej) => { // node:ws is not built in; use the global WebSocket (node >= 22). this.ws = new globalThis.WebSocket(wsUrl); this.ws.addEventListener("open", () => res()); this.ws.addEventListener("error", (e) => rej(e)); + this.ws.addEventListener("close", () => { + if (!this.closing) this.markCrashed("WebSocket.closed", {}); + else this.rejectPending(new Error("CDP session closed")); + }); this.ws.addEventListener("message", (ev) => { const msg = JSON.parse(ev.data); + if (CRASH_EVENTS.has(msg.method)) { + this.markCrashed(msg.method, msg.params || {}); + return; + } const p = this.pending.get(msg.id); if (!p) return; this.pending.delete(msg.id); - msg.error ? p.rej(new Error(JSON.stringify(msg.error))) : p.res(msg.result); + clearTimeout(p.timer); + if (msg.error) { + const error = new Error(JSON.stringify(msg.error)); + if (/target.*crash|renderer.*crash/i.test(msg.error.message || "")) { + this.markCrashed("CDP.error", { method:p.method, error:msg.error }); + p.rej(this.crashError()); + } else { + p.rej(error); + } + } else { + p.res(msg.result); + } + this.armIdleClose(); }); }); } - async send(method, params = {}) { + rejectPending(error) { + for (const { rej: reject, timer } of this.pending.values()) { + clearTimeout(timer); + reject(error); + } + this.pending.clear(); + } + + markCrashed(method, params) { + if (!this.crashEvent) { + this.crashEvent = { method, params, at:new Date().toISOString() }; + } + this.rejectPending(this.crashError()); + } + + crashError(context = "") { + const where = context ? ` during ${context}` : ""; + const signal = this.crashEvent?.method || "unresponsive target"; + return new BrowserCrashError( + `browser renderer unavailable${where} (${signal})`, + { context, signal, event:this.crashEvent }, + ); + } + + armIdleClose() { + clearTimeout(this.idleTimer); + if (!this.ephemeralIdleMs || this.pending.size) return; + this.idleTimer = setTimeout(() => { void this.close(); }, this.ephemeralIdleMs); + } + + async send(method, params = {}, { timeoutMs = COMMAND_TIMEOUT_MS } = {}) { await this.ready; + if (this.crashEvent) throw this.crashError(method); + clearTimeout(this.idleTimer); const id = ++this.id; return new Promise((res, rej) => { - this.pending.set(id, { res, rej }); + const timer = timeoutMs > 0 ? setTimeout(() => { + this.pending.delete(id); + rej(new Error(`CDP ${method} timed out after ${timeoutMs}ms`)); + this.armIdleClose(); + }, timeoutMs) : null; + this.pending.set(id, { res, rej, timer, method }); this.ws.send(JSON.stringify({ id, method, params })); }); } + // Inspector.targetCrashed is the authoritative live signal. The bounded + // Runtime heartbeat covers two less-obvious cases: attaching after the crash + // already happened, and Chrome keeping a dead target in /json with its old + // title and URL (which is exactly how an "Aw, Snap!" page fooled Captutor). + async monitorCrashes() { + try { + await this.send("Inspector.enable", {}, { timeoutMs:HEALTH_TIMEOUT_MS }); + await this.assertHealthy("attach"); + } catch (error) { + if (error instanceof BrowserCrashError) throw error; + this.markCrashed("CDP.unresponsive", { + phase:"attach", message:error.message, + }); + throw this.crashError("attach"); + } + } + + async assertHealthy(context = "browser", { timeoutMs = HEALTH_TIMEOUT_MS } = {}) { + if (this.crashEvent) throw this.crashError(context); + try { + const result = await this.send("Runtime.evaluate", { + expression:"({ readyState:document.readyState, href:location.href })", + returnByValue:true, + }, { timeoutMs }); + if (!result?.result || result.exceptionDetails) { + throw new Error("health expression did not return a page state"); + } + return result.result.value; + } catch (error) { + if (error instanceof BrowserCrashError) throw error; + this.markCrashed("CDP.unresponsive", { context, message:error.message }); + throw this.crashError(context); + } + } + /// Evaluate ONE expression. Wrap statements in an IIFE — top-level `const` /// returns undefined and silently swallows what you meant to return. async eval(expression) { @@ -166,5 +302,112 @@ await this.send("Input.dispatchKeyEvent", { type: "keyDown", ...base }); await this.send("Input.dispatchKeyEvent", { type: "keyUp", ...base }); } - close() { try { this.ws.close(); } catch {} } + // A private, pixels-and-DOM "frame" of the browser target. It never raises + // a native window, draws OCR boxes, moves the pointer, or mutates the page. + // Pathfinding agents get the controls and React Flow topology they actually + // need in one bounded call instead of repeatedly guessing selectors. + async frame() { + return this.eval(`(() => { + const clean = (value, limit = 140) => String(value || '') + .replace(/\\s+/g, ' ').trim().slice(0, limit); + const visible = (element) => { + const style = getComputedStyle(element); + const rect = element.getBoundingClientRect(); + return style.display !== 'none' && style.visibility !== 'hidden' && + Number(style.opacity || 1) > 0 && rect.width > 0 && rect.height > 0 && + rect.bottom >= 0 && rect.right >= 0 && rect.top <= innerHeight && rect.left <= innerWidth; + }; + const box = (element) => { + const rect = element.getBoundingClientRect(); + return { + x: Math.round(rect.left), y: Math.round(rect.top), + width: Math.round(rect.width), height: Math.round(rect.height), + cx: Math.round(rect.left + rect.width / 2), + cy: Math.round(rect.top + rect.height / 2), + }; + }; + const locator = (element) => { + if (element.id) return '#' + CSS.escape(element.id); + const testid = element.getAttribute('data-testid'); + if (testid) return '[data-testid=' + JSON.stringify(testid) + ']'; + const aria = element.getAttribute('aria-label'); + if (aria) return '[aria-label=' + JSON.stringify(aria) + ']'; + const role = element.getAttribute('role'); + const label = clean(element.innerText || element.textContent, 80); + if (label && ['BUTTON', 'A'].includes(element.tagName)) return 'text=' + label; + return role ? '[role=' + JSON.stringify(role) + ']' : element.tagName.toLowerCase(); + }; + const describe = (element) => ({ + tag: element.tagName.toLowerCase(), + role: element.getAttribute('role') || '', + text: clean(element.innerText || element.textContent), + ariaLabel: clean(element.getAttribute('aria-label')), + placeholder: clean(element.getAttribute('placeholder')), + testId: element.getAttribute('data-testid') || '', + disabled: Boolean(element.disabled || element.getAttribute('aria-disabled') === 'true'), + locator: locator(element), + rect: box(element), + }); + const controls = [...document.querySelectorAll( + 'button,a,input,textarea,select,[role=button],[role=option],[role=menuitem],[role=dialog]' + )].filter(visible).slice(0, 180).map(describe); + const nodes = [...document.querySelectorAll('.react-flow__node')] + .filter(visible).map((node) => ({ + id: node.getAttribute('data-id') || node.id || '', + type: [...node.classList].find((name) => name.startsWith('react-flow__node-')) + ?.replace('react-flow__node-', '') || '', + text: clean(node.innerText || node.textContent, 220), + rect: box(node), + handles: [...node.querySelectorAll('.react-flow__handle')].filter(visible).map((handle) => ({ + id: handle.getAttribute('data-handleid') || handle.getAttribute('data-nodeid') || '', + kind: handle.classList.contains('source') ? 'source' + : handle.classList.contains('target') ? 'target' : '', + position: ['left','right','top','bottom'].find((side) => handle.classList.contains(side)) || '', + ariaLabel: clean(handle.getAttribute('aria-label')), + rect: box(handle), + })), + })); + const focus = document.activeElement && document.activeElement !== document.body + ? describe(document.activeElement) : null; + return { + schema: 'captutor-cdp-frame/v1', + capturedAt: new Date().toISOString(), + url: location.href, + title: document.title, + readyState: document.readyState, + viewport: { width: innerWidth, height: innerHeight, dpr: devicePixelRatio }, + focus, + controls, + graph: { + present: Boolean(document.querySelector('.react-flow')), + nodeCount: nodes.length, + edgeCount: document.querySelectorAll('.react-flow__edge').length, + nodes, + }, + }; + })()`); + } + + // Page.captureScreenshot reads the compositor invisibly. Unlike fleet Frame, + // it cannot create an on-screen overlay that leaks into a subsequent take. + async screenshot({ format = "png", quality } = {}) { + await this.send("Page.enable"); + const params = { format, fromSurface: true, captureBeyondViewport: false }; + if (format === "jpeg" && quality != null) params.quality = quality; + const { data } = await this.send("Page.captureScreenshot", params); + return Buffer.from(data, "base64"); + } + + async close({ timeoutMs = 1000 } = {}) { + clearTimeout(this.idleTimer); + this.closing = true; + try { await this.ready; } catch { return; } + if (!this.ws || this.ws.readyState >= 2) return; + const closed = new Promise((resolve) => this.ws.addEventListener("close", resolve, { once: true })); + try { this.ws.close(); } catch { return; } + await Promise.race([ + closed, + new Promise((resolve) => setTimeout(resolve, timeoutMs)), + ]); + } } diff --git a/captutor/lib/cursor.mjs b/captutor/lib/cursor.mjs --- a/captutor/lib/cursor.mjs +++ b/captutor/lib/cursor.mjs @@ -1,87 +1,83 @@ // cursor — the pointer the viewer actually sees. // // CDP clicks do not move the macOS cursor, while `reel` films the real screen. -// Stage Mode therefore drives the native pointer through a tiny smooth-motion -// helper and lands the trusted CDP click at the same page coordinate. Outside -// Stage Mode the original shadow-DOM tutorial pointer remains available. +// A small native Swift overlay paints the filmed pointer and its particles. +// Browser interaction stays on Chrome's trusted CDP channel, so presentation +// never changes hit-testing or becomes browser-bound markup. // -import { execFileSync } from "node:child_process"; +import { execFileSync, spawn } from "node:child_process"; +import { existsSync } from "node:fs"; import { homedir } from "node:os"; import { join } from "node:path"; const REAL_CURSOR = process.env.CAPTUTOR_REAL_CURSOR === "1"; const POINTER_BIN = process.env.CAPTUTOR_POINTER || join(homedir(), ".local", "bin", "captutor-pointer"); +const NATIVE_CURSOR_BIN = process.env.CAPTUTOR_NATIVE_CURSOR + || join(homedir(), ".local", "bin", "captutor-cursor"); +const NO_CURSOR = process.env.CAPTUTOR_CURSOR === "none"; +const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); -// Everything lives in a shadow-DOM host at the top layer, so fuser's own styles -// can neither restyle it nor stack above it, and it is pointer-events:none so it -// can never eat the very click it is illustrating. +let nativeCursor = null; +let nativeCursorFailure = null; +let nativeCursorReady = null; -export const INSTALL = `(() => { - if (window.__captutor) return true; - const host = document.createElement('div'); - host.id = '__captutor_cursor'; - Object.assign(host.style, { - position: 'fixed', inset: '0', pointerEvents: 'none', zIndex: '2147483647', +export async function startNativeCursor() { + if (REAL_CURSOR || NO_CURSOR) return; + if (nativeCursor) return nativeCursorReady; + if (!existsSync(NATIVE_CURSOR_BIN)) { + throw new Error( + `native Captutor cursor is not installed at ${NATIVE_CURSOR_BIN}; enter Stage Mode or run captutor/bin/install.sh`, + ); + } + nativeCursorFailure = null; + const child = spawn(NATIVE_CURSOR_BIN, [], { + stdio: ["pipe", "pipe", "inherit"], }); - document.documentElement.appendChild(host); - const root = host.attachShadow({ mode: 'open' }); - root.innerHTML = \` - - - - -
\`; - const ptr = root.querySelector('.p'); - const ring = root.querySelector('.r'); + nativeCursor = child; + nativeCursorReady = new Promise((resolve, reject) => { + const timeout = setTimeout(() => reject(new Error("native Captutor cursor did not become ready")), 3000); + child.stdout.once("data", (chunk) => { + clearTimeout(timeout); + if (String(chunk).includes("ready")) resolve(); + else reject(new Error("native Captutor cursor returned an invalid readiness response")); + }); + child.once("error", (error) => { + clearTimeout(timeout); + reject(error); + }); + }); + child.once("error", (error) => { + if (nativeCursor === child) nativeCursorFailure = error; + }); + child.once("exit", (code, signal) => { + if (nativeCursor === child) { + if (code && !nativeCursorFailure) { + nativeCursorFailure = new Error(`native Captutor cursor exited (${code}${signal ? `, ${signal}` : ""})`); + } + nativeCursor = null; + nativeCursorReady = null; + } + }); + return nativeCursorReady; +} - const state = { x: window.innerWidth / 2, y: window.innerHeight / 2 }; - const put = (x, y) => { - state.x = x; state.y = y; - ptr.style.transform = \`translate(\${x - 3}px, \${y - 3}px)\`; - ring.style.transform = \`translate(\${x}px, \${y}px)\`; - }; - put(state.x, state.y); +function nativeCommand(op, values = {}) { + if (REAL_CURSOR || NO_CURSOR) return; + if (nativeCursorFailure) throw nativeCursorFailure; + if (!nativeCursor?.stdin?.writable) throw new Error("native Captutor cursor is unavailable"); + nativeCursor.stdin.write(`${JSON.stringify({ op, ...values })}\n`); +} - // easeInOutCubic — accelerate away, coast, settle. A linear glide looks - // robotic; this reads as a hand. - const ease = (t) => (t < 0.5 ? 4 * t * t * t : 1 - Math.pow(-2 * t + 2, 3) / 2); - - window.__captutor = { - pos: () => ({ x: state.x, y: state.y }), - moveTo: (x, y, ms = 520) => new Promise((done) => { - const x0 = state.x, y0 = state.y, t0 = performance.now(); - const step = (now) => { - const t = Math.min(1, (now - t0) / ms); - const k = ease(t); - put(x0 + (x - x0) * k, y0 + (y - y0) * k); - t < 1 ? requestAnimationFrame(step) : done(); - }; - requestAnimationFrame(step); - }), - ripple: () => new Promise((done) => { - ring.style.transition = 'none'; - ring.style.opacity = '1'; - ring.style.transform = \`translate(\${state.x}px, \${state.y}px) scale(.5)\`; - requestAnimationFrame(() => { - ring.style.transition = 'transform .42s ease-out, opacity .42s ease-out'; - ring.style.opacity = '0'; - ring.style.transform = \`translate(\${state.x}px, \${state.y}px) scale(2.6)\`; - setTimeout(done, 420); - }); - }), - }; - return true; -})()`; +export function stopNativeCursor() { + if (!nativeCursor) return; + if (nativeCursor.stdin.writable) { + nativeCursor.stdin.write(`${JSON.stringify({ op: "hide" })}\n`); + nativeCursor.stdin.write(`${JSON.stringify({ op: "quit" })}\n`); + nativeCursor.stdin.end(); + } + nativeCursor = null; + nativeCursorReady = null; +} async function moveRealPointer(cdp, point, durationMs) { const geometry = await cdp.eval(`({ @@ -90,12 +86,28 @@ })`); moveRealPointerWithGeometry(geometry, point, durationMs); } -function moveRealPointerWithGeometry(geometry, point, durationMs) { +export function pagePointToScreen(geometry, point) { const borderX = Math.max(0, (geometry.outerWidth - geometry.innerWidth) / 2); const chromeY = Math.max(0, geometry.outerHeight - geometry.innerHeight - borderX); - const x = geometry.screenX + borderX + point.x; - const y = geometry.screenY + chromeY + point.y; + return { + x: geometry.screenX + borderX + point.x, + y: geometry.screenY + chromeY + point.y, + }; +} + +function moveRealPointerWithGeometry(geometry, point, durationMs) { + const { x, y } = pagePointToScreen(geometry, point); execFileSync(POINTER_BIN, [String(x), String(y), String(durationMs)]); +} + +async function moveNativePointer(cdp, point, durationMs, geometry = null) { + await startNativeCursor(); + const measured = geometry || await cdp.eval(`({ + screenX, screenY, outerWidth, outerHeight, innerWidth, innerHeight + })`); + const screen = pagePointToScreen(measured, point); + nativeCommand("move", { x: screen.x, y: screen.y, durationMs }); + await sleep(durationMs); } async function pointWithin(cdp, selector, { anchorX = 0.5, anchorY = 0.5 } = {}) { @@ -118,16 +130,13 @@ }; })()`); } -/// Glide the drawn pointer to an element, ripple, and land a TRUSTED click at -/// the same spot. The ripple fires just before the real click so the highlight -/// is already blooming when the UI reacts — click-then-ripple reads as lag. +/// Glide the native pointer to an element and land a TRUSTED click at the same +/// tip coordinate. Swift adds the small visual response after the UI commits. export async function clickOn( cdp, selector, { moveMs = 520, settleMs = 140, anchorX = 0.5, anchorY = 0.5 } = {}, ) { - if (!REAL_CURSOR) await cdp.eval(INSTALL); - // Measure, glide, then MEASURE AGAIN before committing the click. // // The glide takes ~half a second, and half a second is a long time in a React @@ -143,22 +152,25 @@ // CGWarpMouseCursorPosition can leave one final native hover event queued. // Let it drain before the trusted CDP click, or that late event can land on // the canvas pane and immediately clear the node selection we just made. await new Promise((resolve) => setTimeout(resolve, 180)); - } - else await cdp.eval(`window.__captutor.moveTo(${first.x}, ${first.y}, ${moveMs})`); + } else if (!NO_CURSOR) await moveNativePointer(cdp, first, moveMs); const now = await pointWithin(cdp, selector, { anchorX, anchorY }); if (Math.hypot(now.x - first.x, now.y - first.y) > 2) { if (REAL_CURSOR) { await moveRealPointer(cdp, now, 120); await new Promise((resolve) => setTimeout(resolve, 180)); - } - else await cdp.eval(`window.__captutor.moveTo(${now.x}, ${now.y}, 120)`); + } else if (!NO_CURSOR) await moveNativePointer(cdp, now, 120); } - if (!REAL_CURSOR) await cdp.eval(`window.__captutor.ripple()`); - await cdp.mouse("mouseMoved", now.x, now.y); - await cdp.mouse("mousePressed", now.x, now.y); - await cdp.mouse("mouseReleased", now.x, now.y); + nativeCommand("down"); + try { + await cdp.mouse("mouseMoved", now.x, now.y); + await cdp.mouse("mousePressed", now.x, now.y); + await cdp.mouse("mouseReleased", now.x, now.y); + } finally { + nativeCommand("up"); + } + nativeCommand("click"); await new Promise((r) => setTimeout(r, settleMs)); return now; // where the action landed — the vertical cut crops to follow it } @@ -170,7 +182,6 @@ cdp, selector, { moveMs = 620, anchorX = 0.5, anchorY = 0.5, offsetX = 0, offsetY = 0 } = {}, ) { - if (!REAL_CURSOR) await cdp.eval(INSTALL); const base = selector.startsWith("text=") || selector.startsWith("js=") ? await cdp.center(selector) : await pointWithin(cdp, selector, { anchorX, anchorY }); @@ -180,7 +191,7 @@ // on its right edge). Clamp only to the visible filming viewport. const x = Math.max(12, Math.min(viewport.width - 28, base.x + Number(offsetX))); const y = Math.max(12, Math.min(viewport.height - 28, base.y + Number(offsetY))); if (REAL_CURSOR) await moveRealPointer(cdp, { x, y }, moveMs); - else await cdp.eval(`window.__captutor.moveTo(${x}, ${y}, ${moveMs})`); + else if (!NO_CURSOR) await moveNativePointer(cdp, { x, y }, moveMs); // Keep the page's pointer state aligned with the native pointer. Besides // making hover treatments truthful, this gives ScreenCaptureKit a compositor // change to record during an otherwise static, pointer-only beat. @@ -198,52 +209,63 @@ fromSelector, toSelector, { moveMs = 520, dragMs = 760, steps = 24, settleMs = 220 } = {}, ) { - if (!REAL_CURSOR) await cdp.eval(INSTALL); - const first = await cdp.center(fromSelector); if (REAL_CURSOR) await moveRealPointer(cdp, first, moveMs); - else await cdp.eval(`window.__captutor.moveTo(${first.x}, ${first.y}, ${moveMs})`); + else if (!NO_CURSOR) await moveNativePointer(cdp, first, moveMs); // Re-measure after the pointer arrives. Hovering a Fuser socket enlarges it, // and React can settle the target node during the approach. const from = await cdp.center(fromSelector, { waitMs: 1500 }); const to = await cdp.center(toSelector, { waitMs: 1500 }); - const geometry = REAL_CURSOR + const geometry = (REAL_CURSOR || !NO_CURSOR) ? await cdp.eval(`({ screenX, screenY, outerWidth, outerHeight, innerWidth, innerHeight })`) : null; await cdp.send("Input.dispatchMouseEvent", { type: "mouseMoved", x: from.x, y: from.y, button: "left", buttons: 0, }); - await cdp.send("Input.dispatchMouseEvent", { - type: "mousePressed", x: from.x, y: from.y, - button: "left", buttons: 1, clickCount: 1, - }); + nativeCommand("down"); + let released = false; + try { + await cdp.send("Input.dispatchMouseEvent", { + type: "mousePressed", x: from.x, y: from.y, + button: "left", buttons: 1, clickCount: 1, + }); - const ease = (t) => (t < 0.5 - ? 4 * t * t * t - : 1 - Math.pow(-2 * t + 2, 3) / 2); - for (let index = 1; index <= steps; index += 1) { - const k = ease(index / steps); - const point = { - x: from.x + (to.x - from.x) * k, - y: from.y + (to.y - from.y) * k, - }; - if (REAL_CURSOR) { - moveRealPointerWithGeometry(geometry, point, Math.max(12, dragMs / steps)); - } else { - await cdp.eval(`window.__captutor.moveTo(${point.x}, ${point.y}, ${Math.max(12, dragMs / steps)})`); + const ease = (t) => (t < 0.5 + ? 4 * t * t * t + : 1 - Math.pow(-2 * t + 2, 3) / 2); + for (let index = 1; index <= steps; index += 1) { + const k = ease(index / steps); + const point = { + x: from.x + (to.x - from.x) * k, + y: from.y + (to.y - from.y) * k, + }; + if (REAL_CURSOR) { + moveRealPointerWithGeometry(geometry, point, Math.max(12, dragMs / steps)); + } else if (!NO_CURSOR) { + await moveNativePointer(cdp, point, Math.max(12, dragMs / steps), geometry); + } + await cdp.send("Input.dispatchMouseEvent", { + type: "mouseMoved", x: point.x, y: point.y, + button: "left", buttons: 1, + }); } + await cdp.send("Input.dispatchMouseEvent", { - type: "mouseMoved", x: point.x, y: point.y, - button: "left", buttons: 1, + type: "mouseReleased", x: to.x, y: to.y, + button: "left", buttons: 0, clickCount: 1, }); + released = true; + } finally { + if (!released) { + await cdp.send("Input.dispatchMouseEvent", { + type: "mouseReleased", x: to.x, y: to.y, + button: "left", buttons: 0, clickCount: 1, + }).catch(() => {}); + } + nativeCommand("up"); } - - await cdp.send("Input.dispatchMouseEvent", { - type: "mouseReleased", x: to.x, y: to.y, - button: "left", buttons: 0, clickCount: 1, - }); await new Promise((resolve) => setTimeout(resolve, settleMs)); return { from, to }; } diff --git a/captutor/lib/onboarding.mjs b/captutor/lib/onboarding.mjs new file mode 100644 --- /dev/null +++ b/captutor/lib/onboarding.mjs @@ -0,0 +1,149 @@ +// onboarding — the shared DOM/bridge contract between Fuser tutorials and Captutor. +// +// Product code owns tutorial IDs, ordering, requirements, and state. Captutor +// discovers that contract from semantic data attributes and the local-only +// `window.__fuserOnboardingAudit` bridge. This keeps screenplays away from +// translated copy, generated class names, portal structure, and Zustand internals. + +const ACTIVE_OVERLAY = '[data-onboarding-overlay]'; +const ACTIVE_TARGET = '[data-onboarding-active="true"]'; +const INITIAL_DIALOG = '[data-onboarding-surface="initial-dialog"]'; + +const sleep = ms => new Promise(resolve => setTimeout(resolve, ms)); + +const visibleAction = action => `js=(() => { + const roots = [ + document.querySelector(${JSON.stringify(ACTIVE_OVERLAY)}), + document.querySelector(${JSON.stringify(INITIAL_DIALOG)}), + document, + ].filter(Boolean); + for (const root of roots) { + const button = [...root.querySelectorAll( + '[data-onboarding-action=${JSON.stringify(action)}]' + )].find(element => element.getClientRects().length > 0 && !element.disabled); + if (button) return button; + } + return null; +})()`; + +export const onboardingSelectors = Object.freeze({ + activeOverlay: ACTIVE_OVERLAY, + activeTarget: ACTIVE_TARGET, + initialDialog: INITIAL_DIALOG, + replayControl: '[data-neo-anchor="replay-editor-tutorial"]', + nextAction: visibleAction('next'), + finishAction: visibleAction('finish'), + previousAction: visibleAction('previous'), +}); + +export async function readOnboardingState(cdp) { + return cdp.eval(`(() => { + const dialog = document.querySelector(${JSON.stringify(INITIAL_DIALOG)}); + const overlay = document.querySelector(${JSON.stringify(ACTIVE_OVERLAY)}); + const target = document.querySelector(${JSON.stringify(ACTIVE_TARGET)}); + const bridge = window.__fuserOnboardingAudit; + const store = bridge?.snapshot?.() ?? null; + const source = overlay || dialog; + return { + contractVersion: bridge?.version ?? null, + engine: overlay?.dataset.onboardingEngine || (dialog ? 'classic-dialog' : 'classic'), + step: source?.dataset.onboardingOverlay || source?.dataset.onboardingStep || + store?.currentStepper || null, + group: target?.dataset.onboardingGroup || null, + contentIndex: Number( + source?.dataset.onboardingContentIndex ?? store?.currentStepperContentIndex ?? 0, + ), + requirement: overlay?.dataset.onboardingRequirement || null, + requirementMet: overlay?.dataset.onboardingRequirementMet === 'true', + actions: [...document.querySelectorAll('[data-onboarding-action]')] + .filter(element => element.getClientRects().length > 0 && !element.disabled) + .map(element => element.dataset.onboardingAction), + targetAnchor: target?.dataset.neoAnchor || null, + store, + }; + })()`); +} + +export async function waitForOnboardingStep( + cdp, + { step, contentIndex, timeoutMs = 20000 } = {}, +) { + const deadline = Date.now() + timeoutMs; + let last = null; + while (Date.now() < deadline) { + last = await readOnboardingState(cdp); + const stepMatches = step === undefined || last.step === step; + const indexMatches = + contentIndex === undefined || last.contentIndex === contentIndex; + if (last.step && stepMatches && indexMatches) return last; + await sleep(100); + } + throw new Error( + `onboarding step timed out: expected ${JSON.stringify({ step, contentIndex })}, ` + + `last ${JSON.stringify(last)}`, + ); +} + +export async function satisfyOnboardingRequirement(cdp, requirement) { + const result = await cdp.eval(`(() => { + if (!['localhost', '127.0.0.1'].includes(location.hostname)) { + return { ok:false, reason:'audit bridge is local-only' }; + } + const bridge = window.__fuserOnboardingAudit; + if (!bridge || bridge.version !== 1) { + return { ok:false, reason:'onboarding audit bridge v1 is unavailable' }; + } + bridge.completeRequirement(${JSON.stringify(requirement)}); + return { ok:true }; + })()`); + if (!result?.ok) throw new Error(result?.reason || 'could not satisfy requirement'); + return result; +} + +export async function advanceOnboarding( + cdp, + { click, satisfyRequirements = false, timeoutMs = 20000 } = {}, +) { + if (typeof click !== 'function') throw new Error('advanceOnboarding needs Captutor click'); + const before = await waitForOnboardingStep(cdp, { timeoutMs }); + const beforeKey = `${before.step}:${before.contentIndex}`; + + if (before.requirement && !before.requirementMet) { + if (!satisfyRequirements) { + throw new Error(`onboarding step ${beforeKey} needs ${before.requirement}`); + } + await satisfyOnboardingRequirement(cdp, before.requirement); + } else if (before.actions.includes('next')) { + await click(onboardingSelectors.nextAction); + } else if (before.actions.includes('finish')) { + await click(onboardingSelectors.finishAction); + } else { + throw new Error(`onboarding step ${beforeKey} has no forward action`); + } + + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + const after = await readOnboardingState(cdp); + if (!after.step || `${after.step}:${after.contentIndex}` !== beforeKey) { + return { before, after }; + } + await sleep(100); + } + throw new Error(`onboarding did not advance from ${beforeKey}`); +} + +export async function startOnboardingReplay(cdp) { + const result = await cdp.eval(`(() => { + if (!['localhost', '127.0.0.1'].includes(location.hostname)) { + return { ok:false, reason:'audit bridge is local-only' }; + } + const bridge = window.__fuserOnboardingAudit; + if (!bridge || bridge.version !== 1) { + return { ok:false, reason:'onboarding audit bridge v1 is unavailable' }; + } + bridge.startReplay(); + return { ok:true }; + })()`); + if (!result?.ok) throw new Error(result?.reason || 'could not start replay'); + return waitForOnboardingStep(cdp); +} diff --git a/captutor/lib/stage-contract.mjs b/captutor/lib/stage-contract.mjs new file mode 100644 --- /dev/null +++ b/captutor/lib/stage-contract.mjs @@ -0,0 +1,52 @@ +// Production fleet missions pathfind on the ordinary desktop, but a real take +// must run inside Captutor's reversible 2x Stage profile. Keep this contract +// separate from the stage setup itself so the recorder can fail closed using +// what Chrome actually sees, rather than trusting an environment flag alone. + +export class StageContractError extends Error { + constructor(message, details = {}) { + super(message); + this.name = "StageContractError"; + this.code = "CAPTUTOR_HIDPI_STAGE_REQUIRED"; + this.details = details; + } +} + +export function assertHiDPIStage({ + required = false, + stageMode = false, + vertical = false, + screen = null, +} = {}) { + if (!required) return null; + if (!stageMode) { + throw new StageContractError( + "fleet mission takes must render through `node bin/stage.mjs render ...`", + { stageMode, vertical }, + ); + } + + // The first assertion can run before CDP attaches. Once Chrome is available, + // the second assertion proves Stage changed the real display to the expected + // logical canvas backed by at least 2x device pixels. + if (!screen) return { stageMode, vertical }; + + const expected = vertical + ? { width:720, height:1280 } + : { width:1280, height:720 }; + const actual = { + width:Number(screen.width), + height:Number(screen.height), + dpr:Number(screen.dpr), + }; + if (actual.width !== expected.width + || actual.height !== expected.height + || !Number.isFinite(actual.dpr) + || actual.dpr < 1.75) { + throw new StageContractError( + `HiDPI Stage is not active (expected ${expected.width}x${expected.height} at 2x; got ${actual.width}x${actual.height} at ${actual.dpr}x)`, + { stageMode, vertical, expected, actual }, + ); + } + return actual; +} diff --git a/captutor/lib/stage-mode.mjs b/captutor/lib/stage-mode.mjs --- a/captutor/lib/stage-mode.mjs +++ b/captutor/lib/stage-mode.mjs @@ -14,15 +14,41 @@ const SIGILS_OFF = join(HOME, ".local", "share", "slab", "state", "prompt-sigils-off"); const BADGE_PLIST = join(HOME, "Library", "LaunchAgents", "computer.aesthetic.desktopbadge.plist"); const POINTER_SOURCE = fileURLToPath(new URL("../bin/captutor-pointer.swift", import.meta.url)); const POINTER_BIN = join(HOME, ".local", "bin", "captutor-pointer"); +const CURSOR_SOURCE = fileURLToPath(new URL("../bin/captutor-cursor.swift", import.meta.url)); +const CURSOR_BIN = join(HOME, ".local", "bin", "captutor-cursor"); const WALLPAPER_SOURCE = process.env.CAPTUTOR_WALLPAPER_SOURCE || fileURLToPath(new URL("../bin/captutor-wallpaper.swift", import.meta.url)); const WALLPAPER_LOGO = fileURLToPath(new URL("../assets/fuser-thumbnail-logo.svg", import.meta.url)); +const WALLPAPER_MARK = fileURLToPath(new URL("../assets/fuser-mark.svg", import.meta.url)); +const WALLPAPER_METABALLS = [0, 1, 2].map((variant) => + fileURLToPath(new URL(`../assets/fuser-metaballs-${variant}.png`, import.meta.url))); const WALLPAPER_APP = join(HOME, ".local", "share", "captutor", "Captutor Wallpaper.app"); const WALLPAPER_BIN = join(WALLPAPER_APP, "Contents", "MacOS", "CaptutorWallpaper"); const WALLPAPER_STATE = join(HOME, ".local", "share", "captutor", "wallpaper-card.json"); const WALLPAPER = "/System/Library/Desktop Pictures/Solid Colors/Space Gray.png"; const DISPLAYPLACER = "/opt/homebrew/bin/displayplacer"; +export function normalizeStageBrand(value = "fuser") { + const brand = String(value || "fuser").trim().toLowerCase(); + if (brand !== "fuser" && brand !== "classic") { + throw new Error(`unsupported Captutor Stage brand "${value}"; expected fuser or classic`); + } + return brand; +} + +export function parseStageFlags(rawArgs = []) { + const vertical = rawArgs.includes("--vertical"); + const brandIndex = rawArgs.indexOf("--brand"); + if (brandIndex >= 0 && (!rawArgs[brandIndex + 1] || rawArgs[brandIndex + 1].startsWith("--"))) { + throw new Error("--brand needs a value: fuser or classic"); + } + const brand = normalizeStageBrand(brandIndex >= 0 ? rawArgs[brandIndex + 1] : "fuser"); + const args = rawArgs.filter((arg, index) => + arg !== "--vertical" && + (brandIndex < 0 || (index !== brandIndex && index !== brandIndex + 1))); + return { vertical, brand, args }; +} + const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); const run = (file, args = [], { allowFailure = false, timeout } = {}) => { const result = spawnSync(file, args, { encoding: "utf8", timeout }); @@ -39,6 +65,11 @@ function readDefault(domain, key) { const result = spawnSync("/usr/bin/defaults", ["read", domain, key], { encoding: "utf8" }); return result.status === 0 ? result.stdout.trim() : null; +} + +function currentPointerSize() { + const value = Number(readDefault("com.apple.universalaccess.plist", "mouseDriverCursorSize")); + return Number.isFinite(value) ? value : 1; } function restoreBoolean(domain, key, value) { @@ -97,6 +128,43 @@ function setMenuAutohide(value) { osa(`tell application "System Events" to set autohide menu bar of dock preferences to ${value ? "true" : "false"}`); } +function screenGeometry() { + return JSON.parse(swift(` +import AppKit +let screen = NSScreen.main! +let frame = screen.frame +let visible = screen.visibleFrame +print("{\\"width\\":\\(Int(frame.width)),\\"height\\":\\(Int(frame.height)),\\"visibleWidth\\":\\(Int(visible.width)),\\"visibleHeight\\":\\(Int(visible.height))}") +`)); +} + +function movePointerAwayFromMenuBar() { + swift(` +import CoreGraphics +let display = CGMainDisplayID() +let bounds = CGDisplayBounds(display) +CGWarpMouseCursorPosition(CGPoint(x: bounds.midX, y: bounds.midY)) +`); +} + +async function enforceHiddenSystemChrome() { + // The menu-bar preference is owned by SystemUIServer, not Dock. Restarting + // only Dock can leave the old bar composited into the first seconds of a + // take, and a pointer parked at y=0 keeps an auto-hidden bar revealed. + sh("killall Finder >/dev/null 2>&1 || true; killall Dock >/dev/null 2>&1 || true; killall SystemUIServer >/dev/null 2>&1 || true"); + movePointerAwayFromMenuBar(); + await sleep(1200); + if (!menuAutohide()) throw new Error("macOS did not enable menu-bar auto-hide"); + const geometry = screenGeometry(); + // A hidden Dock can reserve a tiny reveal strip, but the 30-point menu-bar + // inset must be gone before Reel is allowed to start. + if (geometry.height - geometry.visibleHeight > 8) { + throw new Error( + `system chrome still occupies the Stage (${geometry.visibleWidth}x${geometry.visibleHeight} visible inside ${geometry.width}x${geometry.height})`, + ); + } +} + function darkMode() { return osa('tell application "System Events" to tell appearance preferences to get dark mode') === "true"; } @@ -289,6 +357,12 @@ mkdirSync(dirname(POINTER_BIN), { recursive: true }); run("/usr/bin/swiftc", ["-O", POINTER_SOURCE, "-o", POINTER_BIN]); } +function compileNativeCursor() { + mkdirSync(dirname(CURSOR_BIN), { recursive: true }); + run("/usr/bin/pkill", ["-x", "captutor-cursor"], { allowFailure: true }); + run("/usr/bin/swiftc", ["-O", CURSOR_SOURCE, "-o", CURSOR_BIN]); +} + function compileWallpaper() { mkdirSync(dirname(WALLPAPER_BIN), { recursive: true }); run("/usr/bin/swiftc", ["-O", WALLPAPER_SOURCE, "-o", WALLPAPER_BIN]); @@ -296,6 +370,8 @@ const contents = dirname(dirname(WALLPAPER_BIN)); const resources = join(contents, "Resources"); mkdirSync(resources, { recursive: true }); copyFileSync(WALLPAPER_LOGO, join(resources, "fuser-thumbnail-logo.svg")); + copyFileSync(WALLPAPER_MARK, join(resources, "fuser-mark.svg")); + for (const sheet of WALLPAPER_METABALLS) copyFileSync(sheet, join(resources, sheet.split("/").pop())); writeFileSync(join(contents, "Info.plist"), ` @@ -311,22 +387,28 @@ function stopWallpaper() { run("/usr/bin/pkill", ["-x", "CaptutorWallpaper"], { allowFailure: true }); } -async function startWallpaper() { +async function startWallpaper(brand) { stopWallpaper(); - run("/usr/bin/open", ["-na", WALLPAPER_APP]); + run("/usr/bin/open", ["-na", WALLPAPER_APP, "--args", "--brand", normalizeStageBrand(brand)]); await sleep(900); if (spawnSync("/usr/bin/pgrep", ["-x", "CaptutorWallpaper"]).status !== 0) { throw new Error("Captutor dynamic wallpaper did not launch"); } } -export async function enterStageMode({ vertical = process.env.CAPTUTOR_VERTICAL_MODE === "1" } = {}) { +export async function enterStageMode({ + vertical = process.env.CAPTUTOR_VERTICAL_MODE === "1", + brand = process.env.CAPTUTOR_BRAND || "fuser", +} = {}) { + brand = normalizeStageBrand(brand); + const filmingRealCursor = process.env.CAPTUTOR_REAL_CURSOR === "1"; if (existsSync(STATE)) await exitStageMode(); - const cursorSize = Number(readDefault("com.apple.universalaccess.plist", "mouseDriverCursorSize") || 1); + const cursorSize = currentPointerSize(); const state = { displayMode: displayModeID(), displayProfile: displayProfile(), vertical, + brand, wallpaper: osa('tell application "System Events" to get picture of desktop 1 as text'), createDesktop: readDefault("com.apple.finder", "CreateDesktop"), dockAutohide: readDefault("com.apple.dock", "autohide"), @@ -337,7 +419,10 @@ statsRunning: spawnSync("/usr/bin/pgrep", ["-x", "Stats"]).status === 0, visibleApps: visibleApps(), sigilsWereOff: existsSync(SIGILS_OFF), cursorSize, - pointerChanged: process.env.CAPTUTOR_STAGE_KEEP_POINTER !== "1", + // The default filmed cursor is our Swift overlay, so the system pointer is + // excluded by Reel and does not need a privileged Accessibility-settings + // detour. Preserve enlargement only for explicitly human-driven takes. + pointerChanged: filmingRealCursor && process.env.CAPTUTOR_STAGE_KEEP_POINTER !== "1", }; mkdirSync(dirname(STATE), { recursive: true }); writeFileSync(STATE, JSON.stringify(state, null, 2)); @@ -354,6 +439,7 @@ await sleep(650); if (darkMode()) throw new Error("macOS did not enter Light appearance"); compilePointerBridge(); + compileNativeCursor(); compileWallpaper(); if (vertical) configureVerticalDisplay(); else if (process.env.CAPTUTOR_STAGE_KEEP_DISPLAY !== "1") configureDisplay(); @@ -361,24 +447,41 @@ setWallpaper(WALLPAPER); run("/usr/bin/defaults", ["write", "com.apple.finder", "CreateDesktop", "-bool", "false"]); run("/usr/bin/defaults", ["write", "com.apple.dock", "autohide", "-bool", "true"]); setMenuAutohide(true); - sh("killall Finder >/dev/null 2>&1 || true; killall Dock >/dev/null 2>&1 || true"); + await enforceHiddenSystemChrome(); mkdirSync(dirname(SIGILS_OFF), { recursive: true }); writeFileSync(SIGILS_OFF, ""); if (state.badgeLoaded) run("/bin/launchctl", ["bootout", `gui/${process.getuid()}/computer.aesthetic.desktopbadge`], { allowFailure: true }); if (state.statsRunning) osa('tell application "Stats" to quit', [], true); - if (state.pointerChanged) await setPointerSizeWithRetry(cursorSize, 1.5); + if (state.pointerChanged) { + try { + await setPointerSizeWithRetry(cursorSize, 1.5); + } catch (error) { + // Pointer enlargement is filming polish, not a recording permission. + // New System Settings builds can keep AX access enabled while omitting + // AX_CURSOR_SIZE from the live tree. Continue only if the failed attempt + // left the saved cursor preference untouched; otherwise fail so cleanup + // retains responsibility for restoring it. + if (Math.abs(currentPointerSize() - cursorSize) > 0.08) throw error; + state.pointerChanged = false; + writeFileSync(STATE, JSON.stringify(state, null, 2)); + console.warn(`! pointer enlargement unavailable; filming with saved ${cursorSize}× cursor`); + } + } hideOtherApps(); writeFileSync(WALLPAPER_STATE, JSON.stringify({ phase: "ambient" })); - await startWallpaper(); + await startWallpaper(brand); osa('tell application "Google Chrome" to activate', [], true); console.log( `✓ Captutor ${vertical ? "Vertical " : ""}Stage Mode active — ` + `Light, ${process.env.CAPTUTOR_STAGE_KEEP_DISPLAY === "1" ? "native display" : "2× HiDPI"}, ` + - `branded desk, real ~1.5× pointer`, + `${brand} desk, hidden system chrome, ${filmingRealCursor ? "real ~1.5×" : "native Swift"} pointer`, ); } export async function exitStageMode() { + // A crashed renderer may not get a chance to close its stdin. Never let its + // filmed pointer survive onto the operator's restored desktop. + run("/usr/bin/pkill", ["-x", "captutor-cursor"], { allowFailure: true }); if (!existsSync(STATE)) return; const state = JSON.parse(readFileSync(STATE, "utf8")); const failures = []; @@ -391,7 +494,13 @@ // Each restore is independent. A broken wallpaper path must never strand the // display in HiDPI, and a failed pointer drag must never keep Stats hidden. await restore("dynamic wallpaper", () => stopWallpaper()); if (state.pointerChanged !== false) { - await restore("pointer", () => setPointerSize(1.5, state.cursorSize ?? 1)); + await restore("pointer", async () => { + const wanted = Number(state.cursorSize ?? 1); + // An interrupted entry may have failed before changing the preference. + // Do not reopen System Settings merely to restore a value already exact. + if (Math.abs(currentPointerSize() - wanted) <= 0.08) return; + await setPointerSize(1.5, wanted); + }); } await restore("prompt sigils", () => { if (!state.sigilsWereOff) rmSync(SIGILS_OFF, { force: true }); @@ -409,7 +518,7 @@ }); await restore("desktop icons", () => restoreBoolean("com.apple.finder", "CreateDesktop", state.createDesktop)); await restore("Dock", () => restoreBoolean("com.apple.dock", "autohide", state.dockAutohide)); await restore("menu bar", () => setMenuAutohide(Boolean(state.menuAutohide))); - await restore("Finder and Dock", () => sh("killall Finder >/dev/null 2>&1 || true; killall Dock >/dev/null 2>&1 || true")); + await restore("Finder, Dock, and menu bar", () => sh("killall Finder >/dev/null 2>&1 || true; killall Dock >/dev/null 2>&1 || true; killall SystemUIServer >/dev/null 2>&1 || true")); await restore("display", () => { if (state.displayProfile && existsSync(DISPLAYPLACER)) { run(DISPLAYPLACER, [state.displayProfile]); diff --git a/captutor/lib/tutorial-layout.mjs b/captutor/lib/tutorial-layout.mjs new file mode 100644 --- /dev/null +++ b/captutor/lib/tutorial-layout.mjs @@ -0,0 +1,272 @@ +// Interface-aware composition for filmed React Flow tutorials. +// +// Browser controls can exist outside the page DOM (for example Chrome's +// bottom "Continue the chat" affordance), so mathematical canvas centering is +// not necessarily centered in the view a learner sees. These helpers reserve +// a stable safe region, widen tutorial nodes, and score legibility and balance. + +export const TUTORIAL_LAYOUT_STYLE_ID = "captutor-tutorial-layout"; +export const TUTORIAL_NODE_WIDTH = 336; +export const TUTORIAL_SAFE_INSETS = Object.freeze({ + left: 88, + top: 76, + right: 72, + bottom: 72, +}); + +const clamp = (value, low = 0, high = 100) => Math.max(low, Math.min(high, value)); + +export async function installTutorialLayout(cdp, selectors, { + nodeWidth = TUTORIAL_NODE_WIDTH, +} = {}) { + const css = selectors + .map((selector) => `${selector} { width: ${nodeWidth}px !important; } +${selector} .w-1\\/2:has(input[aria-label="flow.nodes.FalGeminiImageNode.inputs.model.label"]) { + width: 65% !important; +} +${selector} .w-1\\/2:has(input[aria-label="flow.nodes.FalGeminiImageNode.inputs.aspect_ratio.label"]) { + width: 35% !important; +}`) + .join("\n"); + await cdp.eval(`(() => { + const id = ${JSON.stringify(TUTORIAL_LAYOUT_STYLE_ID)}; + let style = document.getElementById(id); + if (!style) { + style = document.createElement('style'); + style.id = id; + document.head.appendChild(style); + } + style.textContent = ${JSON.stringify(css)}; + return true; + })()`); +} + +export async function removeTutorialLayout(cdp) { + await cdp.eval(`document.getElementById(${JSON.stringify(TUTORIAL_LAYOUT_STYLE_ID)})?.remove()`); +} + +function titleSelector(nodeSelector, title) { + return `js=[...document.querySelectorAll(${JSON.stringify(`${nodeSelector} *`)})] + .find((element) => element.children.length === 0 && + (element.textContent || '').trim() === ${JSON.stringify(title)})`; +} + +function targetSelector(nodeSelector, index, count, options) { + return `js=(() => { + const node = document.querySelector(${JSON.stringify(nodeSelector)}); + const title = [...node.querySelectorAll('*')].find((element) => + element.children.length === 0 && (element.textContent || '').trim() === ${JSON.stringify(options.title)}); + const nodeRect = node.getBoundingClientRect(); + const titleRect = title.getBoundingClientRect(); + const chatButton = [...document.querySelectorAll('button')].find((element) => { + const rect = element.getBoundingClientRect(); + return rect.width > 0 && rect.height > 0 && + (element.innerText || '').trim() === 'Continue the chat'; + }); + const chatTop = chatButton?.getBoundingClientRect().top; + const safe = { + left: ${options.insets.left}, + top: ${options.insets.top}, + right: innerWidth - ${options.insets.right}, + bottom: Math.min( + innerHeight - ${options.insets.bottom}, + Number.isFinite(chatTop) ? chatTop - 20 : innerHeight, + ), + }; + const safeWidth = safe.right - safe.left; + const groupWidth = nodeRect.width * ${count} + ${options.gap} * Math.max(0, ${count} - 1); + const groupLeft = safe.left + (safeWidth - groupWidth) / 2; + const desiredLeft = groupLeft + ${index} * (nodeRect.width + ${options.gap}); + const desiredTop = safe.top + Math.max(0, (safe.bottom - safe.top - nodeRect.height) / 2); + const titleOffsetX = titleRect.left + titleRect.width / 2 - nodeRect.left; + const titleOffsetY = titleRect.top + titleRect.height / 2 - nodeRect.top; + const x = desiredLeft + titleOffsetX; + const y = desiredTop + titleOffsetY; + return { + getBoundingClientRect: () => ({ left:x - 1, top:y - 1, width:2, height:2 }), + scrollIntoView() {}, + }; + })()`; +} + +// Drive React Flow's continuous pinch-zoom path rather than opening its menu. +// This lets the teaching layout land around 80% (large enough to read, small +// enough to clear the chat composer) without filming a mystery menu detour. +export async function setTutorialZoom(ctx, target = 80) { + const { cdp } = ctx; + const zoomValue = async () => Number(await cdp.eval(`(() => { + const button = [...document.querySelectorAll('button')].find((element) => + /^\\d+\\s*%$/.test((element.innerText || '').trim())); + return button ? parseInt(button.innerText, 10) : NaN; + })()`)); + + for (let attempt = 0; attempt < 5; attempt += 1) { + const current = await zoomValue(); + if (!Number.isFinite(current)) throw new Error("Fuser zoom control is unavailable"); + if (Math.abs(current - target) <= 3) return current; + // React Flow's wheel curve is approximately exp(-0.0138 * deltaY). + // Clamp extreme fit-view recovery into a few smooth, bounded events. + const deltaY = Math.max(-80, Math.min(80, + -Math.log(target / current) / 0.0138)); + const point = await cdp.eval(`(() => { + const pane = document.querySelector('.react-flow__pane'); + const rect = pane?.getBoundingClientRect(); + return rect + ? { x:rect.left + rect.width / 2, y:rect.top + rect.height / 2 } + : { x:innerWidth / 2, y:innerHeight / 2 }; + })()`); + await cdp.send("Input.dispatchMouseEvent", { + type:"mouseWheel", x:point.x, y:point.y, + deltaX:0, deltaY, modifiers:2, + }); + await cdp.waitFor(`(() => { + const button = [...document.querySelectorAll('button')].find((element) => + /^\\d+\\s*%$/.test((element.innerText || '').trim())); + return button && parseInt(button.innerText, 10) !== ${current}; + })()`); + } + throw new Error(`Could not reach tutorial zoom near ${target}%`); +} + +export async function frameTutorialNodes(ctx, nodes, { + insets = TUTORIAL_SAFE_INSETS, + gap = 64, + moveMs = 380, + dragMs = 480, +} = {}) { + const { drag, sleep } = ctx; + for (let index = 0; index < nodes.length; index += 1) { + const node = nodes[index]; + await drag( + titleSelector(node.selector, node.title), + targetSelector(node.selector, index, nodes.length, { title:node.title, insets, gap }), + { moveMs, dragMs }, + ); + await sleep(180); + } +} + +export async function tutorialLayoutScores(cdp, selectors, { + insets = TUTORIAL_SAFE_INSETS, + minimumNodeWidth = 320, + idealGap = 64, +} = {}) { + const raw = await cdp.eval(`(() => { + const chatButton = [...document.querySelectorAll('button')].find((element) => { + const rect = element.getBoundingClientRect(); + return rect.width > 0 && rect.height > 0 && + (element.innerText || '').trim() === 'Continue the chat'; + }); + const chatTop = chatButton?.getBoundingClientRect().top; + const safe = { + left:${insets.left}, top:${insets.top}, + right:innerWidth - ${insets.right}, + bottom:Math.min( + innerHeight - ${insets.bottom}, + Number.isFinite(chatTop) ? chatTop - 20 : innerHeight, + ), + }; + const nodes = ${JSON.stringify(selectors)} + .map((selector) => ({ selector, node:document.querySelector(selector) })) + .filter(({ node }) => Boolean(node)).map(({ selector, node }) => { + const rect = node.getBoundingClientRect(); + const readable = [...node.querySelectorAll('button,[role=option],input:not([type=hidden])')] + .filter((element) => { + const r = element.getBoundingClientRect(); + const text = element instanceof HTMLInputElement + ? element.value : (element.textContent || '').trim(); + return r.width > 0 && r.height > 0 && text; + }).map((element) => { + const text = (element instanceof HTMLInputElement + ? element.value : (element.textContent || '')).trim().replace(/\\s+/g, ' '); + const style = getComputedStyle(element); + const canvas = document.createElement('canvas'); + const context = canvas.getContext('2d'); + context.font = [style.fontStyle, style.fontWeight, style.fontSize, style.fontFamily] + .filter(Boolean).join(' '); + const horizontalPadding = (parseFloat(style.paddingLeft) || 0) + + (parseFloat(style.paddingRight) || 0); + const measuredWidth = context.measureText(text).width + horizontalPadding; + return { + text, + clipped:measuredWidth > element.clientWidth + 2 || + element.scrollWidth > element.clientWidth + 2 || /(?:…|\\.{3})$/.test(text), + }; + }); + return { + selector, + layoutWidth:node.offsetWidth, + rect:{ left:rect.left, top:rect.top, right:rect.right, bottom:rect.bottom, + width:rect.width, height:rect.height, cx:rect.left + rect.width / 2, + cy:rect.top + rect.height / 2 }, + truncated:readable.filter((item) => item.clipped).map((item) => item.text), + }; + }); + return { viewport:{ width:innerWidth, height:innerHeight }, safe, nodes }; + })()`); + + const outside = raw.nodes.filter(({ rect }) => + rect.left < raw.safe.left || rect.right > raw.safe.right || + rect.top < raw.safe.top || rect.bottom > raw.safe.bottom); + const narrow = raw.nodes.filter(({ rect, layoutWidth }) => + (Number.isFinite(layoutWidth) ? layoutWidth : rect.width) < minimumNodeWidth); + const truncated = raw.nodes.flatMap((node) => node.truncated); + const uiScore = Math.round(clamp( + 100 - outside.length * 28 - narrow.length * 22 - truncated.length * 18, + )); + + let balancePenalty = 0; + if (raw.nodes.length) { + const left = Math.min(...raw.nodes.map(({ rect }) => rect.left)); + const right = Math.max(...raw.nodes.map(({ rect }) => rect.right)); + const top = Math.min(...raw.nodes.map(({ rect }) => rect.top)); + const bottom = Math.max(...raw.nodes.map(({ rect }) => rect.bottom)); + const groupCenter = { x:(left + right) / 2, y:(top + bottom) / 2 }; + const safeCenter = { + x:(raw.safe.left + raw.safe.right) / 2, + y:(raw.safe.top + raw.safe.bottom) / 2, + }; + balancePenalty += Math.abs(groupCenter.x - safeCenter.x) / 5; + balancePenalty += Math.abs(groupCenter.y - safeCenter.y) / 5; + if (raw.nodes.length > 1) { + const ordered = [...raw.nodes].sort((a, b) => a.rect.left - b.rect.left); + for (let index = 1; index < ordered.length; index += 1) { + const gap = ordered[index].rect.left - ordered[index - 1].rect.right; + balancePenalty += Math.abs(gap - idealGap) / 3; + balancePenalty += Math.abs(ordered[index].rect.cy - ordered[index - 1].rect.cy) / 6; + if (gap < 24) balancePenalty += 35; + } + } + } else { + balancePenalty = 100; + } + const balancedScore = Math.round(clamp(100 - balancePenalty - outside.length * 20)); + + return { + "ui-legibility-score": { + score:uiScore, + threshold:90, + truncated, + outsideSafeRegion:outside.map((node) => node.selector), + narrowNodes:narrow.map((node) => node.selector), + safeRegion:raw.safe, + }, + "balanced-layout-score": { + score:balancedScore, + threshold:88, + nodeRects:raw.nodes.map(({ selector, rect }) => ({ selector, ...rect })), + safeRegion:raw.safe, + }, + }; +} + +export async function assertTutorialLayout(ctx, selectors) { + const scores = await tutorialLayoutScores(ctx.cdp, selectors); + for (const [name, result] of Object.entries(scores)) { + ctx.check(name, result); + if (result.score < result.threshold) { + throw new Error(`${name} ${result.score} is below ${result.threshold}: ${JSON.stringify(result)}`); + } + } + return scores; +} diff --git a/captutor/ops/heartbeat.sh b/captutor/ops/heartbeat.sh new file mode 100644 --- /dev/null +++ b/captutor/ops/heartbeat.sh @@ -0,0 +1,36 @@ +#!/bin/zsh +# iris heartbeat — liveness plus bounded mission recovery. Runs every 5 min. +# It stamps presence, checks for a stopped Captutor mission, repairs a crashed +# Fuser renderer, and removes one local failure tombstone so the orchestrator +# can retry. The orchestrator still enforces current Asana ownership before it +# launches anything; the heartbeat cannot invent work or bypass assignment. +# +# Kill switch: +# launchctl bootout gui/$(id -u)/ai.iris.heartbeat # stop the pulse +export PATH=/opt/homebrew/bin:$PATH +LOG="$HOME/.hermes/logs/heartbeat.log" +mkdir -p "$HOME/.hermes/logs" + +# read-only liveness checks +GW=$(pgrep -f "hermes.*gateway" >/dev/null 2>&1 && echo up || echo down) +BOARD=$(curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:9120/ 2>/dev/null) + +# stamp the beat onto the board (lastHeartbeat + updatedAt), without touching items +/opt/homebrew/bin/node -e ' +const fs=require("fs"); +const p=process.env.HOME+"/.local/share/desktop-badge/mission.json"; +try{ + const d=JSON.parse(fs.readFileSync(p,"utf8")); + const now=new Date().toISOString(); + d.lastHeartbeat=now; d.updatedAt=now; + fs.writeFileSync(p, JSON.stringify(d,null,2)); +}catch(e){}' 2>/dev/null + +echo "$(date -u +%FT%TZ) ♥ beat · gateway=$GW · board=$BOARD" >> "$LOG" + +# Save a recoverable Captutor mission from a worker/browser stoppage. This is a +# strict one-retry supervisor by default, and it never draws Frame/Puppet UI. +RECOVERY="$HOME/Developer/captutor/ops/iris-heartbeat-recovery.mjs" +if [[ -f "$RECOVERY" ]]; then + /opt/homebrew/bin/node "$RECOVERY" >> "$LOG" 2>&1 +fi diff --git a/captutor/ops/iris-heartbeat-recovery.mjs b/captutor/ops/iris-heartbeat-recovery.mjs new file mode 100644 --- /dev/null +++ b/captutor/ops/iris-heartbeat-recovery.mjs @@ -0,0 +1,201 @@ +#!/usr/bin/env node +// Active half of Iris's heartbeat. The shell heartbeat still publishes a +// liveness pulse; this supervisor repairs one recoverable Captutor stoppage. +// Ownership remains with the orchestrator: requeueing only removes its local +// failure tombstone, and the task will relaunch only if Asana still assigns it +// to Iris. + +import { execFileSync } from "node:child_process"; +import { + existsSync, mkdirSync, readFileSync, renameSync, rmSync, writeFileSync, +} from "node:fs"; +import { homedir } from "node:os"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +const home = homedir(); +const statePath = process.env.IRIS_ORCHESTRATOR_STATE + || join(home, ".hermes", "orchestrator-state.json"); +const logPath = process.env.IRIS_HEARTBEAT_LOG + || join(home, ".hermes", "logs", "heartbeat.log"); +const lockPath = process.env.IRIS_RECOVERY_LOCK + || join(home, ".hermes", "iris-heartbeat-recovery.lock"); +const captutor = process.env.CAPTUTOR_HOME || join(home, "Developer", "captutor"); +const node = process.env.CAPTUTOR_NODE || "/opt/homebrew/bin/node"; +const maxRetries = Number(process.env.IRIS_CAPTUTOR_RECOVERY_RETRIES || 1); +const reloadWaitMs = Number(process.env.IRIS_BROWSER_RELOAD_WAIT_MS || 5_000); + +const delay = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); + +function readJson(path) { + try { return JSON.parse(readFileSync(path, "utf8")); } + catch { return null; } +} + +function atomicWriteJson(path, value) { + mkdirSync(dirname(path), { recursive:true }); + const temp = `${path}.${process.pid}.tmp`; + writeFileSync(temp, `${JSON.stringify(value, null, 2)}\n`, { mode:0o600 }); + renameSync(temp, path); +} + +function log(message) { + mkdirSync(dirname(logPath), { recursive:true }); + writeFileSync(logPath, `${new Date().toISOString()} ♥ recovery · ${message}\n`, { flag:"a" }); +} + +function processAlive(pid) { + if (!pid) return false; + try { process.kill(Number(pid), 0); return true; } + catch { return false; } +} + +export function latestRecoverableFailure(state) { + if (state?.active) return null; + return Object.entries(state?.done || {}) + .filter(([, record]) => record?.kind === "captutor" && record.status === "failed") + .map(([taskGid, record]) => ({ taskGid, ...record })) + .sort((a, b) => Number(b.at || 0) - Number(a.at || 0))[0] || null; +} + +export function recoveryAttempts(state, taskGid) { + return Number(state?.recoveries?.[taskGid]?.attempts || 0); +} + +export function markStoppedWorkerFailed(state, now = Date.now()) { + const active = state?.active; + if (!active || active.kind !== "captutor" || processAlive(active.pid)) return null; + if (now - Number(active.startedAt || now) < 2 * 60_000) return null; + state.done ||= {}; + state.done[active.taskGid] = { + name:active.name, + kind:"captutor", + status:"failed", + reason:"worker-disappeared", + detail:`worker ${active.pid || "?"} stopped before verified outbox delivery`, + log:active.log, + at:now, + }; + state.active = null; + return state.done[active.taskGid]; +} + +export function beginRecovery(state, failure, now = Date.now(), maximum = maxRetries) { + state.recoveries ||= {}; + const previous = state.recoveries[failure.taskGid] || {}; + const attempts = Number(previous.attempts || 0); + if (attempts >= maximum) { + state.recovery = { + taskGid:failure.taskGid, mission:failure.name, status:"exhausted", + attempts, maximum, reason:failure.reason || "missing-outbox-artifacts", + detail:failure.detail || "No verified Captutor outbox delivery.", updatedAt:now, + }; + return false; + } + state.recovery = { + taskGid:failure.taskGid, mission:failure.name, status:"checking-browser", + attempts, maximum, reason:failure.reason || "missing-outbox-artifacts", + detail:failure.detail || "No verified Captutor outbox delivery.", updatedAt:now, + }; + return true; +} + +export function queueRecovery(state, failure, now = Date.now()) { + state.recoveries ||= {}; + const attempts = recoveryAttempts(state, failure.taskGid) + 1; + state.recoveries[failure.taskGid] = { + attempts, lastAttemptAt:now, reason:failure.reason || "missing-outbox-artifacts", + }; + state.recovery = { + ...state.recovery, + taskGid:failure.taskGid, + mission:failure.name, + status:"queued", + attempts, + updatedAt:now, + activity:`Browser healthy; retry ${attempts}/${state.recovery?.maximum || maxRetries} queued.`, + }; + delete state.done[failure.taskGid]; + return state.recovery; +} + +function browserProbe() { + try { + execFileSync(node, [join(captutor, "bin", "cdp-frame.mjs"), + "--match", "fuser.studio", "--compact", "--timeout", "10000"], { + cwd:captutor, + env:{ ...process.env, CDP_PORT:process.env.CDP_PORT || "9333", CAPTUTOR_CDP_EPHEMERAL:"1" }, + encoding:"utf8", timeout:12_000, stdio:["ignore", "pipe", "pipe"], + }); + return { healthy:true, crash:false, message:"Fuser renderer is responsive." }; + } catch (error) { + const output = `${error.stdout || ""}\n${error.stderr || ""}`; + return { + healthy:false, + crash:/BROWSER_RENDERER_CRASH|Inspector\.targetCrashed/.test(output), + message:output.trim().slice(-500) || error.message, + }; + } +} + +async function recoverBrowser(state) { + let health = browserProbe(); + if (health.healthy) return health; + if (!health.crash) return health; + state.recovery.status = "reloading-browser"; + state.recovery.activity = "Renderer crashed; reloading the Fuser tab."; + state.recovery.updatedAt = Date.now(); + atomicWriteJson(statePath, state); + execFileSync("/usr/bin/osascript", ["-e", + 'tell application "Google Chrome" to reload active tab of front window'], { + timeout:10_000, stdio:"ignore", + }); + await delay(reloadWaitMs); + health = browserProbe(); + return health; +} + +export async function runRecovery() { + let locked = false; + try { + mkdirSync(lockPath); + locked = true; + } catch { + return { action:"busy" }; + } + try { + const state = readJson(statePath); + if (!state) return { action:"no-state" }; + const stopped = markStoppedWorkerFailed(state); + if (stopped) atomicWriteJson(statePath, state); + const failure = latestRecoverableFailure(state); + if (!failure) return { action:"none" }; + if (!beginRecovery(state, failure)) { + atomicWriteJson(statePath, state); + log(`${failure.name}: automatic retry exhausted`); + return { action:"exhausted", failure }; + } + atomicWriteJson(statePath, state); + const health = await recoverBrowser(state); + if (!health.healthy) { + state.recovery.status = "blocked"; + state.recovery.activity = `Browser recovery failed: ${health.message}`; + state.recovery.updatedAt = Date.now(); + atomicWriteJson(statePath, state); + log(`${failure.name}: browser recovery blocked`); + return { action:"blocked", failure, health }; + } + const recovery = queueRecovery(state, failure); + atomicWriteJson(statePath, state); + log(`${failure.name}: ${recovery.activity}`); + return { action:"queued", failure, health, recovery }; + } finally { + if (locked && existsSync(lockPath)) rmSync(lockPath, { recursive:true, force:true }); + } +} + +const isMain = process.argv[1] && fileURLToPath(import.meta.url) === process.argv[1]; +if (isMain) { + const result = await runRecovery(); + console.log(JSON.stringify(result)); +} diff --git a/captutor/ops/iris-orchestrator.mjs b/captutor/ops/iris-orchestrator.mjs new file mode 100644 --- /dev/null +++ b/captutor/ops/iris-orchestrator.mjs @@ -0,0 +1,587 @@ +#!/usr/bin/env node +// iris-orchestrator — makes iris self-driving, safely. Polls her ASSIGNED Asana +// tasks and works them ONE AT A TIME. Code tasks spawn a worker in an isolated +// worktree and end in a PR. Tasks tagged BOTH `mission` and `captutor` run in the +// fixed Captutor workspace and end in a verified Desktop/outbox artifact. +// The orchestrator keeps her mission board as the live work-list +// (in-progress + queued), and posts status to Slack. Extra assignments queue and +// wait. Gated by the ownership guardrail: only ASSIGNED tasks are worked, and if +// the active task is unassigned mid-flight she stands down. +// +// This is the ORCHESTRATION layer — the queue/lock/status/mission. Whether an +// individual worker nails a given task is the worker's job; this guarantees only +// one runs, the human is kept informed, and nothing unassigned is touched. +// +// Dependency-free: node https + child_process (flk, gh). Persistent via launchd. + +import { execFileSync, spawn } from "node:child_process"; +import https from "node:https"; +import { readFileSync, writeFileSync, existsSync, mkdirSync, readdirSync } from "node:fs"; +import { homedir } from "node:os"; +import { join } from "node:path"; + +const H = homedir(); +const DIR = join(H, ".hermes"); +const STATE = join(DIR, "orchestrator-state.json"); +const MISSION_FILE = join(H, ".local", "share", "desktop-badge", "mission.json"); +const MANUAL_MISSION_FILE = join(H, ".hermes", "manual-mission.json"); +const LOG = join(DIR, "logs", "orchestrator.log"); + +const ASANA = process.env.ASANA_ACCESS_TOKEN || ""; +const SLACK = process.env.SLACK_BOT_TOKEN || ""; +const USER_GID = process.env.ASANA_USER_GID || "1216250551404992"; +const WS_GID = process.env.ASANA_WORKSPACE_GID || "1208084256731239"; +const HOME_CHANNEL = process.env.ORCH_CHANNEL || "D0BFUT0D4SF"; // iris<->jeffrey DM +const FUSER = process.env.FUSER_REPO || join(H, "Developer", "fuser"); +const CAPTUTOR = process.env.CAPTUTOR_HOME || join(H, "Developer", "captutor"); +const OUTBOX = process.env.CAPTUTOR_OUTBOX || join(H, "Desktop", "outbox"); +const DESK_CLEANUP = process.env.IRIS_DESK_CLEANUP || join(H, ".local", "bin", "iris-desk-cleanup"); +const WORKER_RUN = join(DIR, "bin", "worker-run.sh"); +const WORKTREES = join(DIR, "worktrees"); +const GH_ENV = { ...process.env, GH_CONFIG_DIR: join(H, ".config", "gh-iris") }; +const BASE_REF = process.env.ORCH_BASE_REF || "origin/staging"; +const POLL_MS = parseInt(process.env.ORCH_POLL_MS || "60000", 10); +const STALL_MS = parseInt(process.env.ORCH_STALL_MS || String(20 * 60000), 10); +// Routine "starting" / "still working" chatter is intentionally quiet by +// default. Completion, review, stand-down, and failure notices still go to +// Slack. Set ORCH_SLACK_PROGRESS=true when live progress narration is useful. +const SLACK_PROGRESS = process.env.ORCH_SLACK_PROGRESS === "true"; +// Review loop: iris watches her own open PRs for teammate feedback (reviews live +// on GitHub, not Asana — the orchestrator never saw them). Ingestion + Slack +// notify is ALWAYS on. Auto-addressing (re-launch a worker on the PR branch to +// fix the feedback) is gated behind this flag so we verify ingestion first. +const REVIEW_AUTOFIX = process.env.ORCH_REVIEW_AUTOFIX === "true"; +const REPO = process.env.FUSER_GH_REPO || "fuserstudio/fuser"; +const IRIS_LOGIN = process.env.IRIS_GH_LOGIN || "iris-fuser"; + +function log(m) { + const line = new Date().toISOString() + " " + m + "\n"; + try { if (!existsSync(join(DIR, "logs"))) mkdirSync(join(DIR, "logs"), { recursive: true }); } catch {} + try { writeFileSync(LOG, line, { flag: "a" }); } catch {} + process.stdout.write(line); +} + +function httpsJson({ hostname, path, method = "GET", headers = {}, body = null }) { + return new Promise((resolve, reject) => { + const data = body ? JSON.stringify(body) : null; + const req = https.request( + { hostname, path, method, headers: { ...headers, ...(data ? { "Content-Type": "application/json", "Content-Length": Buffer.byteLength(data) } : {}) }, timeout: 15000 }, + (res) => { let b = ""; res.on("data", (c) => (b += c)); res.on("end", () => { try { resolve(JSON.parse(b)); } catch (e) { reject(e); } }); }, + ); + req.on("error", reject); + req.on("timeout", () => req.destroy(new Error("timeout"))); + if (data) req.write(data); + req.end(); + }); +} + +async function assignedTasks() { + const fields = "name,notes,completed,permalink_url,tags.name"; + const j = await httpsJson({ + hostname: "app.asana.com", + path: `/api/1.0/tasks?assignee=${USER_GID}&workspace=${WS_GID}&completed_since=now&opt_fields=${fields}&limit=100`, + headers: { Authorization: "Bearer " + ASANA }, + }); + if (j.errors) throw new Error(j.errors[0]?.message || "asana error"); + return (j.data || []).filter((t) => !t.completed).map((t) => { + const tags = (t.tags || []).map((tag) => String(tag.name || "").toLowerCase()); + return { + gid: t.gid, + name: t.name || "(untitled)", + notes: t.notes || "", + url: t.permalink_url || null, + tags, + kind: tags.includes("mission") && tags.includes("captutor") ? "captutor" : "pr", + }; + }); +} + +async function asanaComment(gid, text) { + try { + const r = await httpsJson({ + hostname: "app.asana.com", path: `/api/1.0/tasks/${gid}/stories`, method: "POST", + headers: { Authorization: "Bearer " + ASANA }, body: { data: { text } }, + }); + if (r.errors) log("asana comment failed: " + (r.errors[0]?.message || "?")); + } catch (e) { log("asana comment err: " + e.message); } +} + +async function asanaComplete(gid) { + try { + const r = await httpsJson({ + hostname: "app.asana.com", path: `/api/1.0/tasks/${gid}`, method: "PUT", + headers: { Authorization: "Bearer " + ASANA }, body: { data: { completed: true } }, + }); + if (r.errors) log("asana completion failed: " + (r.errors[0]?.message || "?")); + } catch (e) { log("asana completion err: " + e.message); } +} + +async function slack(text) { + if (!SLACK) { log("no SLACK_BOT_TOKEN, skip: " + text); return; } + try { + const r = await httpsJson({ hostname: "slack.com", path: "/api/chat.postMessage", method: "POST", + headers: { Authorization: "Bearer " + SLACK }, body: { channel: HOME_CHANNEL, text, unfurl_links: false } }); + if (!r.ok) log("slack failed: " + (r.error || "?")); + } catch (e) { log("slack err: " + e.message); } +} + +async function progressSlack(text) { + if (!SLACK_PROGRESS) { log("slack progress suppressed: " + text); return; } + await slack(text); +} + +function git(args, opts = {}) { + return execFileSync("git", args, { cwd: FUSER, encoding: "utf8", timeout: 60000, ...opts }).trim(); +} +function slug(s) { return String(s).toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, "").slice(0, 24) || "task"; } + +// Create a fresh isolated worktree off the latest base tip (idempotent). +function makeWorktree(wt, branch) { + try { git(["worktree", "remove", "--force", wt]); } catch {} + try { git(["branch", "-D", branch]); } catch {} + git(["fetch", "origin", "staging", "--quiet"]); + git(["worktree", "add", "-b", branch, wt, BASE_REF]); +} +function removeWorktree(wt, branch) { + try { git(["worktree", "remove", "--force", wt]); } catch {} + try { git(["branch", "-D", branch]); } catch {} +} + +// Launch the headless worker DETACHED (the daemon owns it, not this tick). +function launchWorker(wt, log, prompt, promptFile, extraEnv = {}) { + writeFileSync(promptFile, prompt); + const child = spawn("bash", [WORKER_RUN, wt, log, promptFile], { + cwd: wt, detached: true, stdio: "ignore", env: { ...process.env, ...extraEnv }, + }); + child.unref(); + return child.pid; +} +// The headless run appended its exit marker → the worker has finished. +function workerFinished(log) { + try { return /=== worker-run exit /.test(readFileSync(log, "utf8")); } catch { return false; } +} + +function workerExit(log) { + try { + const matches = [...readFileSync(log, "utf8").matchAll(/=== worker-run exit (\d+) /g)]; + return matches.length ? Number(matches.at(-1)[1]) : null; + } catch { return null; } +} + +function captutorDeliveries(taskGid, startedAt = 0) { + const found = []; + try { + for (const name of readdirSync(OUTBOX).filter((f) => f.endsWith(".json")).sort().reverse()) { + const path = join(OUTBOX, name); + let doc; + try { doc = JSON.parse(readFileSync(path, "utf8")); } catch { continue; } + if (doc.schema !== "captutor-outbox/v1" || doc.status !== "complete" || doc.taskGid !== taskGid) continue; + if (startedAt && Date.parse(doc.createdAt || 0) < startedAt) continue; + const video = join(OUTBOX, doc.video || ""); + const captions = join(OUTBOX, doc.captions || ""); + if (existsSync(video) && existsSync(captions)) { + found.push({ manifest: path, video, captions, metadata: doc }); + } + } + } catch {} + return found; +} + +function captutorExpectedFormats(notes) { + const hit = String(notes || "").match(/^CAPTUTOR_FORMATS:\s*([^\n]+)$/mi); + return hit ? hit[1].split(",").map((value) => value.trim()).filter(Boolean) : []; +} + +function processAlive(pid) { + if (!pid) return false; + try { process.kill(pid, 0); return true; } catch { return false; } +} + +function releaseActiveWorkspace(active) { + if (active?.wt && active?.branch) removeWorktree(active.wt, active.branch); +} + +function stopActiveWorker(active) { + if (!active?.pid) return; + try { process.kill(-active.pid, "SIGTERM"); } catch {} +} + +function prForBranch(branch) { + try { + const out = execFileSync("gh", ["pr", "list", "--repo", "fuserstudio/fuser", "--head", branch, "--json", "url,state", "--limit", "1"], + { cwd: FUSER, encoding: "utf8", timeout: 20000, env: GH_ENV }).trim(); + const arr = JSON.parse(out || "[]"); + return arr[0] || null; + } catch { return null; } +} + +// Worktree checked out on an EXISTING PR branch (for addressing review feedback, +// vs makeWorktree which cuts a fresh branch off BASE_REF). +function makeWorktreeOnBranch(wt, branch) { + try { git(["worktree", "remove", "--force", wt]); } catch {} + git(["fetch", "origin", branch, "--quiet"]); + git(["worktree", "add", "--force", "-B", branch, wt, `origin/${branch}`]); +} + +// ── GitHub review ingestion (read-only; runs gh under iris's gh-iris config) ── +function gh(args) { + try { return execFileSync("gh", args, { cwd: FUSER, encoding: "utf8", timeout: 30000, env: GH_ENV }).trim(); } + catch (e) { log("gh err [" + args.join(" ") + "]: " + (e.stderr || e.message)); return null; } +} +function ghApi(path) { + const out = gh(["api", "-H", "Accept: application/vnd.github+json", path + (path.includes("?") ? "&" : "?") + "per_page=100"]); + try { return out ? JSON.parse(out) : []; } catch { return []; } +} +function irisOpenPRs() { + const out = gh(["pr", "list", "--repo", REPO, "--author", IRIS_LOGIN, "--state", "open", + "--json", "number,url,title,headRefName,reviewDecision", "--limit", "50"]); + try { return out ? JSON.parse(out) : []; } catch { return []; } +} +// New teammate feedback on a PR since the recorded high-water marks. Ignores +// iris's own comments. Returns { items:[...], hwm:{comment,review,issue} }. +function prFeedback(number, hwm) { + const items = []; + const nhwm = { comment: hwm.comment || 0, review: hwm.review || 0, issue: hwm.issue || 0 }; + for (const c of ghApi(`repos/${REPO}/pulls/${number}/comments`)) { + if (c.user?.login === IRIS_LOGIN) continue; + if (c.id > (hwm.comment || 0)) items.push({ kind: "comment", id: c.id, author: c.user?.login || "?", body: c.body || "", loc: `${c.path}:${c.line ?? c.original_line ?? "?"}` }); + nhwm.comment = Math.max(nhwm.comment, c.id); + } + for (const r of ghApi(`repos/${REPO}/pulls/${number}/reviews`)) { + if (r.user?.login === IRIS_LOGIN || !r.state || r.state === "PENDING") continue; + if (r.id > (hwm.review || 0)) items.push({ kind: "review", id: r.id, author: r.user?.login || "?", state: r.state, body: r.body || `(${r.state})` }); + nhwm.review = Math.max(nhwm.review, r.id); + } + for (const c of ghApi(`repos/${REPO}/issues/${number}/comments`)) { + if (c.user?.login === IRIS_LOGIN) continue; + if (c.id > (hwm.issue || 0)) items.push({ kind: "issue", id: c.id, author: c.user?.login || "?", body: c.body || "" }); + nhwm.issue = Math.max(nhwm.issue, c.id); + } + return { items, hwm: nhwm }; +} + +function loadState() { + try { + const s = JSON.parse(readFileSync(STATE, "utf8")); + if (!s.reviews) s.reviews = {}; + if (!s.recoveries) s.recoveries = {}; + return s; + } catch { return { active: null, done: {}, reviews: {}, recoveries: {} }; } +} +function saveState(s) { writeFileSync(STATE, JSON.stringify(s, null, 2) + "\n"); } + +function writeMission(active, tasks, done) { + let manual = null; + try { manual = JSON.parse(readFileSync(MANUAL_MISSION_FILE, "utf8")); } catch {} + // Only heartbeat.sh advances lastHeartbeat. Orchestrator ticks still refresh + // updatedAt/content, but must not impersonate a beat or the five-minute + // countdown will reset every minute and MacPal will animate false pulses. + let previousHeartbeat = null; + try { previousHeartbeat = JSON.parse(readFileSync(MISSION_FILE, "utf8")).lastHeartbeat || null; } catch {} + const now = new Date().toISOString(); + const lastHeartbeat = previousHeartbeat || now; + const items = []; + if (active) items.push({ text: `${active.name}`, status: "in_progress" }); + for (const t of tasks) if (!done[t.gid] && (!active || t.gid !== active.taskGid)) items.push({ text: t.name, status: "pending" }); + // Completed work stays in orchestrator state for audit/deduplication, but an + // idle mission should actually look clear instead of carrying old trophies. + const remaining = tasks.filter((t) => !done[t.gid]); + const recent = active || tasks.length ? Object.values(done).slice(-2) : []; + for (const d of recent) items.push({ + text: `${d.name} (${d.status === "failed" ? "needs attention" : d.kind === "captutor" ? "rendered" : "shipped"})`, + status: d.status === "failed" ? "pending" : "done", + }); + const useManual = !active && tasks.length === 0 && manual?.mission; + const doc = useManual ? { + ...manual, + agent: "iris", + updatedAt:now, + lastHeartbeat, + heartbeatIntervalSeconds:300, + } : { + mission: active?.kind === "captutor" ? "rendering a product demo" + : active ? "working PRs (one at a time)" + : recent.some((d) => d.status === "failed") ? "mission needs attention" + : remaining.some((t) => t.kind === "captutor") ? "product demos queued" + : remaining.length ? "queued PRs" : "idle — no assigned tasks", + emoji:"🪽", agent:"iris", updatedAt:now, + items:items.slice(0, 8), lastHeartbeat, heartbeatIntervalSeconds:300, + }; + try { if (!existsSync(join(H, ".local", "share", "desktop-badge"))) mkdirSync(join(H, ".local", "share", "desktop-badge"), { recursive: true }); } catch {} + writeFileSync(MISSION_FILE, JSON.stringify(doc, null, 2) + "\n"); +} + +// Scan iris's open PRs for new teammate feedback. Always notifies (Slack). If +// REVIEW_AUTOFIX and changes were requested AND iris is still the Asana assignee +// of the PR's task, claim the active slot and launch a worker on a worktree of +// the EXISTING branch to address it. Returns true if it claimed the slot. +async function reviewPass(s, assigned) { + for (const pr of irisOpenPRs()) { + const rec = s.reviews[pr.number] || (s.reviews[pr.number] = { branch: pr.headRefName, hwm: {}, seeded: false }); + rec.branch = pr.headRefName; + const { items, hwm } = prFeedback(pr.number, rec.hwm); + rec.hwm = hwm; // advance always so we never re-alert on the same comments + + // First sight of a PR: seed high-water marks silently — no backlog alerts. + if (!rec.seeded) { rec.seeded = true; saveState(s); continue; } + if (!items.length) { saveState(s); continue; } + + const changesRequested = pr.reviewDecision === "CHANGES_REQUESTED" || items.some((i) => i.state === "CHANGES_REQUESTED"); + const line = (i) => `• ${i.author}${i.loc ? ` (${i.loc})` : ""}: ${(i.body || "").split("\n")[0].slice(0, 140)}`; + await slack(`💬 new review on *${pr.title}* (${pr.url})${changesRequested ? " — changes requested" : ""}:\n${items.slice(0, 4).map(line).join("\n")}`); + saveState(s); + + if (!(REVIEW_AUTOFIX && changesRequested) || s.active) continue; + + // Ownership hard gate: only auto-address if iris is the CURRENT assignee of + // this PR's Asana task (mapped via what we recorded when we opened it). + const taskGid = Object.keys(s.done).find((g) => (s.done[g].pr || "").includes(`/pull/${pr.number}`)) || null; + if (!taskGid || !assigned.has(taskGid)) { + await slack(`↳ not auto-addressing #${pr.number} — I'm not the current assignee. flagging for a human.`); + continue; + } + const name = `review-${pr.number}`; + const wt = join(WORKTREES, name); + const logf = join(DIR, "logs", `worker-${name}.log`); + const promptFile = join(DIR, `worker-${name}.prompt`); + const body = items.map((i) => `- ${i.author}${i.loc ? ` [${i.loc}]` : ""}: ${i.body}`).join("\n"); + const prompt = + `A teammate left review feedback on your PR #${pr.number} (${pr.url}). Address it end to end.\n\n` + + `You are in a fresh git worktree already checked out on the EXISTING PR branch ${pr.headRefName} (reset to origin/${pr.headRefName}). Do NOT start a new branch. Rebuild context first: git log, gh pr view ${pr.number} --repo ${REPO}.\n\n` + + `Review feedback:\n${body}\n\n` + + `Make the changes, commit only the relevant files, then update the PR:\n` + + ` git push --force-with-lease origin ${pr.headRefName}\n` + + `Then reply on the PR summarizing what changed:\n` + + ` gh pr comment ${pr.number} --repo ${REPO} --body ""\n` + + `If a comment is unclear or you disagree, ask on the PR instead of guessing. Print the PR URL when done.`; + try { + makeWorktreeOnBranch(wt, pr.headRefName); + s.active = { taskGid, name: `review: ${pr.title}`, branch: pr.headRefName, wt, log: logf, startedAt: Date.now(), kind: "review", prNumber: pr.number }; + saveState(s); + try { writeFileSync(logf, ""); } catch {} + launchWorker(wt, logf, prompt, promptFile); + await progressSlack(`🛠️ addressing the review on *${pr.title}* — on it now.`); + log(`launched review worker for PR #${pr.number} (branch ${pr.headRefName})`); + return true; + } catch (e) { log("review spawn err: " + e.message); } + } + return false; +} + +async function tick() { + if (!ASANA) { log("no ASANA token"); return; } + let tasks; + try { tasks = await assignedTasks(); } catch (e) { log("asana poll err: " + e.message); return; } + const assigned = new Set(tasks.map((t) => t.gid)); + const s = loadState(); + + // Guardrail: active task unassigned → stand down. (Review-fix items carry the + // taskGid they were gated on; a plain item with no taskGid is skipped here.) + if (s.active && s.active.taskGid && !assigned.has(s.active.taskGid)) { + log(`active task ${s.active.taskGid} unassigned → standing down`); + stopActiveWorker(s.active); + releaseActiveWorkspace(s.active); + await slack(`🛑 stood down on *${s.active.name}* — it's no longer assigned to me.`); + s.active = null; saveState(s); + } + + // Progress the active work. For a review fix the PR already exists, so + // completion is the worker finishing (not a PR appearing). + if (s.active) { + if (s.active.kind === "captutor") { + const exit = workerExit(s.active.log); + if (exit !== null) { + const deliveries = exit === 0 + ? captutorDeliveries(s.active.taskGid, s.active.startedAt) + : []; + const formats = new Set(deliveries.map((delivery) => delivery.metadata.format)); + const missing = (s.active.expectedFormats || []).filter((format) => !formats.has(format)); + if (deliveries.length && !missing.length) { + const artifactLines = deliveries + .sort((a, b) => String(a.metadata.format).localeCompare(String(b.metadata.format))) + .flatMap((delivery) => [ + `${delivery.metadata.format}: ${delivery.video}`, + `manifest: ${delivery.manifest}`, + ]); + log(`captutor mission ${s.active.name} → ${deliveries.length} verified artifact(s)`); + await asanaComment(s.active.taskGid, + `iris rendered this product-demo mission and placed the verified artifacts in Panda's Desktop outbox:\n${artifactLines.join("\n")}`); + await asanaComplete(s.active.taskGid); + await slack(`🎬 rendered *${s.active.name}* → ${deliveries.map((delivery) => delivery.video).join(", ")}`); + s.done[s.active.taskGid] = { + name: s.active.name, kind: "captutor", status: "done", + deliveries, at: Date.now(), + }; + if (s.recovery?.taskGid === s.active.taskGid) { + s.recovery = { + ...s.recovery, status:"complete", updatedAt:Date.now(), + activity:`Recovered mission completed with ${deliveries.length} verified artifact(s).`, + }; + } + } else { + const detail = missing.length ? `; missing formats: ${missing.join(", ")}` : ""; + const reason = exit === 0 ? "missing-outbox-artifacts" : "worker-exit"; + const failureDetail = exit === 0 + ? `worker exited 0 but produced ${deliveries.length} verified artifact(s)${detail}` + : `worker exited ${exit} before verified delivery${detail}`; + log(`captutor mission ${s.active.name} exited ${exit} without every verified outbox artifact${detail}`); + await asanaComment(s.active.taskGid, + `iris could not complete this Captutor mission (worker exit ${exit}; ${deliveries.length} verified artifact(s)${detail}). Log: ${s.active.log}`); + await slack(`⚠️ Captutor mission *${s.active.name}* needs attention — not every requested video reached the outbox.`); + s.done[s.active.taskGid] = { + name:s.active.name, kind:"captutor", status:"failed", + reason, detail:failureDetail, exitCode:exit, + verifiedArtifacts:deliveries.length, missingFormats:missing, + log:s.active.log, at:Date.now(), + }; + if (s.recovery?.taskGid === s.active.taskGid) { + s.recovery = { + ...s.recovery, status:"failed", reason, detail:failureDetail, + updatedAt:Date.now(), activity:`Retry stopped: ${failureDetail}.`, + }; + } + } + s.active = null; saveState(s); + } else if (Date.now() - s.active.startedAt > 2 * 60000 && !processAlive(s.active.pid)) { + log(`captutor mission ${s.active.name} lost worker ${s.active.pid || "?"} without an exit marker`); + await asanaComment(s.active.taskGid, + `iris's Captutor worker disappeared before writing an exit marker. The task remains incomplete for retry. Log: ${s.active.log}`); + s.done[s.active.taskGid] = { + name:s.active.name, kind:"captutor", status:"failed", + reason:"worker-disappeared", + detail:`worker ${s.active.pid || "?"} stopped before writing an exit marker`, + log:s.active.log, at:Date.now(), + }; + if (s.recovery?.taskGid === s.active.taskGid) { + s.recovery = { + ...s.recovery, status:"failed", reason:"worker-disappeared", + detail:`worker ${s.active.pid || "?"} stopped before writing an exit marker`, + updatedAt:Date.now(), activity:"Retry worker disappeared before verified delivery.", + }; + } + s.active = null; saveState(s); + } else if (Date.now() - s.active.startedAt > STALL_MS && !s.active.stallNoted) { + await progressSlack(`⏳ still rendering *${s.active.name}* (${Math.round((Date.now() - s.active.startedAt) / 60000)}m).`); + s.active.stallNoted = true; saveState(s); + } + } else if (s.active.kind === "review") { + const label = s.active.name.replace(/^review: /, ""); + if (workerFinished(s.active.log)) { + log(`review worker for #${s.active.prNumber} finished`); + await slack(`✅ pushed updates for the review on *${label}*${s.active.prNumber ? ` (#${s.active.prNumber})` : ""}.`); + if (s.active.prNumber && s.reviews[s.active.prNumber]) s.reviews[s.active.prNumber].addressedAt = Date.now(); + removeWorktree(s.active.wt, s.active.branch); + s.active = null; saveState(s); + } else if (Date.now() - s.active.startedAt > STALL_MS && !s.active.stallNoted) { + await progressSlack(`⏳ still addressing the review on *${label}* (${Math.round((Date.now() - s.active.startedAt) / 60000)}m).`); + s.active.stallNoted = true; saveState(s); + } + } else { + const pr = prForBranch(s.active.branch); + if (pr) { + log(`active ${s.active.name} → PR ${pr.url}`); + await asanaComment(s.active.taskGid, `iris opened a PR for this task: ${pr.url} — ready for review.`); + await slack(`✅ opened a PR for *${s.active.name}*: ${pr.url}`); + s.done[s.active.taskGid] = { name: s.active.name, branch: s.active.branch, pr: pr.url, at: Date.now() }; + removeWorktree(s.active.wt, s.active.branch); + s.active = null; saveState(s); + } else if (workerFinished(s.active.log)) { + log(`worker for ${s.active.name} finished without a PR → releasing`); + await slack(`⚠️ my worker on *${s.active.name}* finished without opening a PR — I'll need a retry (log: ${s.active.log}).`); + removeWorktree(s.active.wt, s.active.branch); + s.active = null; saveState(s); + } else if (Date.now() - s.active.startedAt > STALL_MS) { + if (!s.active.stallNoted) { await progressSlack(`⏳ still working *${s.active.name}* (${Math.round((Date.now() - s.active.startedAt) / 60000)}m) — flagging in case it's stuck.`); s.active.stallNoted = true; saveState(s); } + } + } + } + + // Watch iris's open PRs for teammate feedback and (gated) address it before + // starting new work — a requested change on an in-flight PR outranks a fresh + // task. Notify-only until ORCH_REVIEW_AUTOFIX is on. + if (!s.active) { try { await reviewPass(s, assigned); } catch (e) { log("reviewPass err: " + e.message); } } + + // Pick up the next assigned task (one at a time). + if (!s.active) { + const next = tasks.find((t) => !s.done[t.gid]); + if (next) { + const name = `${slug(next.name)}-${next.gid.slice(-6)}`; + const logf = join(DIR, "logs", `worker-${name}.log`); + const promptFile = join(DIR, `worker-${name}.prompt`); + if (next.kind === "captutor") { + const expectedFormats = captutorExpectedFormats(next.notes); + const prompt = + `Execute this assigned Captutor product-demo mission end to end on Panda. DO NOT edit the Fuser repo, create a branch, commit, push, or open a PR.\n\n` + + `Task: ${next.name}\n${next.notes ? next.notes + "\n" : ""}${next.url ? "Asana: " + next.url + "\n" : ""}\n` + + `You are already in the current Captutor workspace. Read README.md first. Verify the recorder, GUI Chrome/CDP session, login, and credits as applicable. ` + + `For UI pathfinding, use the bounded internal frame (\`CDP_PORT=9333 node bin/cdp-frame.mjs --match fuser.studio\`, optionally with \`--screenshot /tmp/preflight.png\`). ` + + `It returns controls plus React Flow nodes/handles/edges without visible tooling and closes CDP cleanly; do not write ad-hoc attach scripts or repeatedly map unrelated UI. ` + + `Keep the filmed interaction on Fuser's canvas. Treat the right-side node properties inspector as off-camera setup only: close it before Reel and do not open it during a take unless the task explicitly teaches that inspector. ` + + `Before Stage, verify System Events Accessibility and the SlabMenubar recording bridge. If macOS presents an in-scope System Settings, Accessibility, Automation, or Screen Recording permission prompt, approve it off camera and verify the grant before continuing; never film a permission dialog. ` + + `Run Captutor in the FOREGROUND. Invoke the Stage render as a direct foreground child of this worker with a long enough shell timeout. Do not use Monitor, a subagent, a background job, nohup, or a scheduled check-in for the render; remain attached until the command exits and the required outbox files have been verified. ` + + `Pathfind on the ordinary desktop, but perform every actual take in Captutor's true 2x HiDPI Stage. Render with \`node bin/stage.mjs render --outbox "$CAPTUTOR_OUTBOX"\`; never invoke \`captutor.mjs render\` directly for a mission. ` + + `The environment already sets CAPTUTOR_TASK_GID and CAPTUTOR_REQUIRE_HIDPI for this task, and Captutor will refuse to start Reel unless Stage and its real 2x display geometry are active. ` + + `Success means Captutor writes a complete captutor-outbox/v1 manifest plus its MP4 and VTT for every requested format to the outbox. ` + + `Do not substitute an old render. If blocked, explain the exact blocker and exit nonzero.`; + try { + if (!existsSync(CAPTUTOR)) throw new Error(`missing Captutor workspace: ${CAPTUTOR}`); + if (!existsSync(DESK_CLEANUP)) throw new Error(`missing pre-mission setup: ${DESK_CLEANUP}`); + log(`running pre-mission setup: ${DESK_CLEANUP}`); + execFileSync(DESK_CLEANUP, [], { encoding: "utf8", timeout: 120000 }); + mkdirSync(OUTBOX, { recursive: true }); + s.active = { + taskGid: next.gid, name: next.name, kind: "captutor", cwd: CAPTUTOR, + log: logf, startedAt: Date.now(), expectedFormats, + recoveryAttempt:s.recoveries?.[next.gid]?.attempts || 0, + }; + if (s.recovery?.taskGid === next.gid) { + s.recovery = { + ...s.recovery, status:"relaunching", updatedAt:Date.now(), + activity:`Retry ${s.recovery.attempts}/${s.recovery.maximum} is starting in the foreground.`, + }; + } + saveState(s); + try { writeFileSync(logf, ""); } catch {} + s.active.pid = launchWorker(CAPTUTOR, logf, prompt, promptFile, { + CAPTUTOR_TASK_GID: next.gid, + CAPTUTOR_OUTBOX: OUTBOX, + CAPTUTOR_REQUIRE_HIDPI: "1", + }); + saveState(s); + log(`launched Captutor mission for ${next.name}`); + await progressSlack(`🎬 starting product-demo mission *${next.name}*${next.url ? " (" + next.url + ")" : ""}.`); + } catch (e) { log("captutor mission spawn err: " + e.message); } + writeMission(s.active, tasks, s.done); + return; + } + + const branch = `iris/${name}`; + const wt = join(WORKTREES, name); + const prompt = + `Do this assigned task end to end, then OPEN A PULL REQUEST.\n\n` + + `Task: ${next.name}\n${next.notes ? next.notes + "\n" : ""}${next.url ? "Asana: " + next.url + "\n" : ""}\n` + + `You are in a fresh git worktree on branch ${branch} off ${BASE_REF}. Make the change, ` + + `commit only the relevant files, push the branch to origin, then open a PR:\n` + + ` gh pr create --repo fuserstudio/fuser --base staging --head ${branch} --title "" --body ""\n` + + `Print the PR URL when done.`; + try { + makeWorktree(wt, branch); + // Claim + persist the lock BEFORE launching / any async, so a crash or + // re-tick can never re-pick this task or drop the one-at-a-time lock. + s.active = { taskGid: next.gid, name: next.name, branch, wt, log: logf, startedAt: Date.now() }; + saveState(s); + try { writeFileSync(logf, ""); } catch {} + launchWorker(wt, logf, prompt, promptFile); + log(`launched worker for ${next.name} (branch ${branch})`); + const queued = tasks.filter((t) => !s.done[t.gid] && t.gid !== next.gid).length; + await progressSlack(`🛠️ starting on *${next.name}*${next.url ? " (" + next.url + ")" : ""} — one PR at a time${queued ? `; ${queued} more queued` : ""}.`); + } catch (e) { log("spawn err: " + e.message); } + } + } + + writeMission(s.active, tasks, s.done); +} + +log(`iris-orchestrator starting — poll ${POLL_MS}ms, base ${BASE_REF}, review ${REVIEW_AUTOFIX ? "auto-fix" : "notify-only"}, slack progress ${SLACK_PROGRESS ? "on" : "off"}`); +await tick(); +setInterval(() => tick().catch((e) => log("tick crash: " + e.message)), POLL_MS); diff --git a/captutor/ops/iris-progress-publisher.mjs b/captutor/ops/iris-progress-publisher.mjs new file mode 100644 --- /dev/null +++ b/captutor/ops/iris-progress-publisher.mjs @@ -0,0 +1,177 @@ +#!/usr/bin/env node + +import { execFileSync } from "node:child_process"; +import { mkdirSync, readFileSync, readdirSync, renameSync, writeFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { basename, join } from "node:path"; + +const home = homedir(); +const statePath = join(home, ".hermes", "orchestrator-state.json"); +const outputDir = join(home, ".local", "share", "desktop-badge"); +const outputPath = join(outputDir, "agent-progress.json"); +const intervalMs = 2_000; + +function readJson(path) { + try { return JSON.parse(readFileSync(path, "utf8")); } + catch { return null; } +} + +function processes() { + try { + return execFileSync("/bin/ps", ["-axo", "pid=,ppid=,command="], { encoding: "utf8" }) + .split("\n") + .map((line) => { + const match = line.match(/^\s*(\d+)\s+(\d+)\s+(.*)$/); + return match ? { pid: Number(match[1]), ppid: Number(match[2]), command: match[3] } : null; + }) + .filter(Boolean); + } catch { + return []; + } +} + +function findClaudeSession(workerPid) { + const row = processes().find((p) => p.ppid === workerPid && /(^|\/)claude\s+-p\b/.test(p.command)); + if (!row) return null; + const session = readJson(join(home, ".claude", "sessions", `${row.pid}.json`)); + return session?.sessionId ? { pid: row.pid, ...session } : null; +} + +function loadTasks(sessionId) { + const dir = join(home, ".claude", "tasks", sessionId); + let names = []; + try { names = readdirSync(dir).filter((name) => /^\d+\.json$/.test(name)); } + catch { return []; } + return names + .map((name) => readJson(join(dir, name))) + .filter((task) => task && task.subject) + .sort((a, b) => Number(a.id) - Number(b.id)); +} + +function latestAssistantActivity(session) { + if (!session?.sessionId || !session?.cwd) return { activity: "", lastTool: "" }; + const project = session.cwd.replaceAll("/", "-"); + const path = join(home, ".claude", "projects", project, `${session.sessionId}.jsonl`); + let lines; + try { lines = readFileSync(path, "utf8").trim().split("\n"); } + catch { return { activity: "", lastTool: "" }; } + let activity = ""; + let lastTool = ""; + for (let index = lines.length - 1; index >= 0 && (!activity || !lastTool); index -= 1) { + let entry; + try { entry = JSON.parse(lines[index]); } catch { continue; } + if (entry?.message?.role !== "assistant" || !Array.isArray(entry.message.content)) continue; + for (const block of [...entry.message.content].reverse()) { + if (!lastTool && block?.type === "tool_use") lastTool = String(block.name || ""); + if (!activity && block?.type === "text") { + activity = String(block.text || "") + .replace(/\s+/g, " ") + .replace(/^#+\s*/, "") + .trim() + .slice(0, 180); + } + } + } + return { activity, lastTool }; +} + +function phaseFor(tasks) { + const activeText = tasks + .filter((task) => task.status === "in_progress") + .map((task) => `${task.subject} ${task.activeForm || ""}`) + .join(" ") + .toLowerCase(); + const rules = [ + [/inspect|capture hygiene|\bqa\b/, "INSPECTING"], + [/outbox|deliver|manifest/, "DELIVERING"], + [/render|generat(e|ing).*video/, "RENDERING"], + [/narrat|pacing/, "NARRATING"], + [/author|writing|\bwrite\b/, "AUTHORING"], + [/preflight|study|research|pattern|probe|selector|mapping/, "PATHFINDING"], + ]; + return rules.find(([pattern]) => pattern.test(activeText))?.[1] + || (tasks.length && tasks.every((task) => task.status === "completed") ? "COMPLETE" : "WORKING"); +} + +function latestFailure(state) { + return Object.entries(state?.done || {}) + .filter(([, record]) => record?.status === "failed") + .map(([taskGid, record]) => ({ taskGid, ...record })) + .sort((a, b) => Number(b.at || 0) - Number(a.at || 0))[0] || null; +} + +function publish() { + const state = readJson(statePath); + const active = state?.active; + let payload; + if (!active?.pid) { + const recovery = state?.recovery; + const recovering = recovery && [ + "checking-browser", "reloading-browser", "queued", "relaunching", + ].includes(recovery.status); + const failure = latestFailure(state); + payload = recovering ? { + schema:"iris-agent-progress/v1", + updatedAt:new Date().toISOString(), + state:"working", + phase:"RECOVERING", + taskGid:String(recovery.taskGid || ""), + mission:String(recovery.mission || "Mission"), + completed:0, + total:1, + active:[recovery.activity || "Recovering stopped mission"], + activity:recovery.detail || recovery.reason || "Recovering stopped mission", + recoveryAttempt:Number(recovery.attempts || 0), + recoveryMaximum:Number(recovery.maximum || 0), + } : failure || recovery?.status === "blocked" || recovery?.status === "exhausted" ? { + schema:"iris-agent-progress/v1", + updatedAt:new Date().toISOString(), + state:"failed", + phase:"FAILED", + taskGid:String(failure?.taskGid || recovery?.taskGid || ""), + mission:String(failure?.name || recovery?.mission || "Mission"), + completed:0, + total:1, + active:[], + activity:String(recovery?.activity || failure?.detail || failure?.reason || "Mission needs attention"), + failureReason:String(failure?.reason || recovery?.reason || "unknown"), + log:String(failure?.log || ""), + } : { + schema:"iris-agent-progress/v1", + updatedAt:new Date().toISOString(), + state:"idle", + phase:"IDLE", + completed:0, + total:0, + active:[], + }; + } else { + const session = findClaudeSession(Number(active.pid)); + const tasks = session ? loadTasks(session.sessionId) : []; + const activeTasks = tasks.filter((task) => task.status === "in_progress"); + const completed = tasks.filter((task) => task.status === "completed").length; + const detail = latestAssistantActivity(session); + payload = { + schema: "iris-agent-progress/v1", + updatedAt: new Date().toISOString(), + state: "working", + phase: phaseFor(tasks), + taskGid: String(active.taskGid || ""), + mission: String(active.name || "Mission"), + workerPid: Number(active.pid), + sessionId: String(session?.sessionId || ""), + completed, + total: tasks.length, + active: activeTasks.map((task) => task.activeForm || task.subject).slice(0, 3), + activity: detail.activity, + lastTool: detail.lastTool, + }; + } + mkdirSync(outputDir, { recursive: true }); + const temp = `${outputPath}.${process.pid}.tmp`; + writeFileSync(temp, `${JSON.stringify(payload, null, 2)}\n`, { mode: 0o644 }); + renameSync(temp, outputPath); +} + +publish(); +setInterval(publish, intervalMs); diff --git a/captutor/screenplays/talking-taco.mjs b/captutor/screenplays/talking-taco.mjs new file mode 100644 --- /dev/null +++ b/captutor/screenplays/talking-taco.mjs @@ -0,0 +1,491 @@ +// talking-taco — from a blank project to a generated image, to a generated +// video: Fuser's whole creative loop in one connected canvas. +// +// This is the first Captutor lesson that chains TWO real generations. It opens +// on a genuinely empty project (no seed flow), adds a Gemini Image node and +// writes a text-only prompt, generates a talking taco, then adds a Kling 3.0 +// Video node, wires the taco image straight into its Image input, adds a short +// motion prompt, and generates the animation. The edge between the two nodes +// is the whole point: it is the visible, undeniable proof that the video was +// made FROM the generated image, not some other clip. +// +// English is the only locale this lesson ships in — the brief calls for an +// English-narrated, English-captioned tutorial, so `say` fields are plain +// strings rather than the `{ en, "zh-CN": ... }` maps other lessons use. +import fuserBrandChrome, { fuserEffectTheme } from "../themes/fuser.mjs"; +import { + assertTutorialLayout, frameTutorialNodes, installTutorialLayout, removeTutorialLayout, + setTutorialZoom, +} from "../lib/tutorial-layout.mjs"; + +const WORKSPACE = "https://app.fuser.studio/w/me"; + +const GEMINI_NODE = ".react-flow__node-FalGeminiImageNode"; +const KLING_NODE = ".react-flow__node-FalKling30VideoNode"; +const GEMINI_PROMPT = `${GEMINI_NODE} textarea`; +const KLING_PROMPT = `${KLING_NODE} textarea`; +const GEMINI_GENERATED_IMAGE = `${GEMINI_NODE} img[alt="Displaying input"]`; +const KLING_GENERATED_VIDEO = `${KLING_NODE} video`; +const EXECUTE = '[data-ph-capture-attribute-node-toolbar-action="execute_node"]'; +// The left rail's node-adding "+" carries no aria-label or test id — every +// toolbar icon shares `data-is-toolbar-item="true"` — but it is reliably the +// FIRST of them (Add Node, Assets, Terminal, Templates, Settings, in that +// fixed order), so an index is the stable handle rather than a pixel guess. +const ADD_NODE = `js=document.querySelectorAll('[data-is-toolbar-item="true"]')[0]`; + +const TACO_IMAGE_PROMPT = + "A cheerful cartoon taco character with big googly eyes and a wide open " + + "mouth mid-laugh, sitting on a colorful kitchen counter, vibrant flat " + + "illustration style, clean studio lighting, simple background"; +const TACO_MOTION_PROMPT = + "The taco opens and closes its mouth as if talking excitedly, waving its " + + "arms, gentle bobbing motion, no camera movement"; + +// Fuser renders each node property as a row: a label (exact text, no +// children) beside a pair of connector handles. Matching the LABEL and +// walking up to the nearest handle survives node reflows, hidden-properties +// toggles, and property reordering — a fixed handle index would not. Reused +// for both the image node's single true output and the video node's single +// true image-input, each confirmed by hand against the live DOM before this +// screenplay was written. +const handleNear = (nodeSelector, labelText, handleClass) => `js=(() => { + const node = document.querySelector(${JSON.stringify(nodeSelector)}); + if (!node) return null; + const label = [...node.querySelectorAll('*')].find((el) => + el.children.length === 0 && (el.textContent || '').trim() === ${JSON.stringify(labelText)}); + let row = label, hops = 0; + while (row && hops < 6) { + const handle = row.querySelector(${JSON.stringify(handleClass)}); + if (handle) return handle; + row = row.parentElement; hops += 1; + } + return null; +})()`; +const GEMINI_OUTPUT_HANDLE = handleNear(GEMINI_NODE, "Generated Image", ".react-flow__handle-right.source"); +const KLING_IMAGE_INPUT_HANDLE = handleNear(KLING_NODE, "Image", ".react-flow__handle-left.target"); + +let creditsBeforeRun = ""; + +// Fuser's node fields need a click to SELECT the node before a second click +// can actually focus a field inside it — the first click's mousedown lands on +// the (still-unselected) node wrapper, not the textarea. A single click, even +// freshly re-measured, can therefore land on an unfocused field. Click twice, +// then verify focus and the committed value before moving on, retrying once +// if Fuser has not caught up yet. This also survives the render's normal +// gap between typing and the next beat's execute click. +async function typePrompt(ctx, selector, text) { + const { cdp, click, sleep } = ctx; + for (let attempt = 1; attempt <= 3; attempt += 1) { + await click(selector); + await sleep(200); + if (!(await cdp.eval(`document.activeElement === document.querySelector(${JSON.stringify(selector)})`))) { + await click(selector); + await sleep(200); + } + await cdp.type(text); + await sleep(300); + const value = await cdp.eval(`document.querySelector(${JSON.stringify(selector)})?.value || ''`); + if (value === text) return; + } + throw new Error(`could not commit prompt text into ${selector}`); +} + +// The node properties inspector is not part of this lesson. Its closed state +// persists while nodes are selected, so establish that state off camera and +// leave the canvas, inline fields, and floating Generate control as the only +// filmed interaction. `View properties` is the closed-state affordance. +async function closePropertiesInspector(cdp) { + const closed = await cdp.eval(`[...document.querySelectorAll('button')] + .some((button) => (button.innerText || '').trim() === 'View properties')`); + if (closed) return; + const point = await cdp.eval(`(() => { + const button = [...document.querySelectorAll('button')].find((element) => { + const rect = element.getBoundingClientRect(); + return rect.left > innerWidth - 55 && rect.top > 65 && rect.top < 110 && + rect.width > 10 && rect.width < 30; + }); + if (!button) return null; + const rect = button.getBoundingClientRect(); + return { x: rect.left + rect.width / 2, y: rect.top + rect.height / 2 }; + })()`); + // A genuinely blank canvas has no selected node, so it exposes neither the + // inspector nor its closed-state `View properties` affordance. That is + // already the clean state this helper is meant to establish. + if (!point) return; + await cdp.mouse("mousePressed", point.x, point.y); + await cdp.mouse("mouseReleased", point.x, point.y); + await cdp.waitFor(`[...document.querySelectorAll('button')] + .some((button) => (button.innerText || '').trim() === 'View properties')`); +} + +async function suppressDuplicateIrisPresence(cdp) { + await cdp.eval(`(() => { + window.__captutorPresenceObserver?.disconnect(); + const suppress = () => { + for (const element of document.querySelectorAll('[title]')) { + if (element.title !== 'iris@fuser.studio') continue; + element.dataset.captutorDuplicatePresence = 'true'; + element.style.display = 'none'; + } + }; + suppress(); + window.__captutorPresenceObserver = new MutationObserver(suppress); + window.__captutorPresenceObserver.observe(document.documentElement, { + childList: true, subtree: true, + }); + return true; + })()`); +} + +// Selecting a node to reveal its floating toolbar can occasionally lose the +// pointer activation to Fuser's own hover handling (the same edge case +// documented in image-generation-workflow.mjs). Reselect and retry with a +// trusted Enter on the focused Generate button rather than double-firing it. +async function runNode(ctx, nodeSelector, startedExpr) { + const { cdp, click, sleep } = ctx; + // Belt-and-suspenders: a node executed with an empty required prompt comes + // back as a hard "Operation failed" from Fuser rather than a queued run, and + // was the one error observed to occasionally knock the whole flow back to + // the workspace. `typePrompt` already guarantees the field committed, but + // never click Generate against an empty required field. + const promptValue = await cdp.eval( + `document.querySelector(${JSON.stringify(nodeSelector)} + ' textarea')?.value ?? null`, + ); + if (promptValue === "") throw new Error(`refusing to execute ${nodeSelector}: prompt is empty`); + const select = async (moveMs) => { + await click(nodeSelector, { moveMs, anchorY: 0.01 }); + await sleep(220); + if (!(await cdp.eval(`!!document.querySelector(${JSON.stringify(EXECUTE)})`))) { + const border = await cdp.eval(`(() => { + const rect = document.querySelector(${JSON.stringify(nodeSelector)}).getBoundingClientRect(); + return { x: rect.left + rect.width / 2, y: rect.top + 5 }; + })()`); + await cdp.mouse("mousePressed", border.x, border.y); + await cdp.mouse("mouseReleased", border.x, border.y); + await sleep(220); + } + await cdp.waitFor( + `document.querySelector(${JSON.stringify(EXECUTE)}) && !document.querySelector(${JSON.stringify(EXECUTE)}).disabled`, + ); + }; + await select(480); + await closePropertiesInspector(cdp); + await click(EXECUTE, { moveMs: 480 }); + await sleep(900); + if (!(await cdp.eval(startedExpr))) { + await select(260); + await closePropertiesInspector(cdp); + await cdp.eval(`document.querySelector(${JSON.stringify(EXECUTE)}).focus()`); + await cdp.key("Enter", "Enter", 13); + } + await cdp.waitFor(startedExpr, { timeoutMs: 10000, everyMs: 100 }); +} + +const creditsExpr = `[...document.querySelectorAll('button')] + .map((button) => (button.innerText || '').trim()) + .find((text) => /^[\\d,]+✦$/.test(text)) || ''`; + +export default { + slug: "talking-taco", + voice: "jeffrey", + window: "Fuser", + desktopFrame: true, + match: "fuser.studio", + theme: "light", + effectTheme: fuserEffectTheme, + brandChrome: fuserBrandChrome, + billable: true, + fps: 60, + title: "Learn Fuser by making a Talking Taco", + subtitle: "A fresh project → a generated image → a generated video", + openingCard: { + title: "Learn Fuser", + durationMs: 2400, + transition: "slide", + }, + closingCard: { + title: "One prompt became an image. That image became a video.", + durationMs: 2400, + transition: "genie", + }, + acceptance: { + minimumDurationSec: 60, + requireOpeningCard: true, + requireEndingCard: true, + requireBrandChrome: true, + loudnessLufs: [-18, -14], + requiredChecks: [ + "fresh_blank_project_opened", + "image_node_added", + "image_prompt_entered", + "image_generation_started", + "generated_image_returned", + "video_node_added", + "image_to_video_edge_connected", + "video_prompt_entered", + "video_generation_started", + "generated_video_returned", + "final_provenance_tableau_complete", + "ui-legibility-score", + "balanced-layout-score", + ], + }, + + // A genuinely fresh project every take: open the workspace, create a blank + // project (Fuser navigates straight to /flow/ on prod), and wait for a + // fully hydrated canvas before the camera rolls. Dismiss any first-run + // onboarding off camera — the narration replaces it. + setup: async ({ cdp, locale, setLocale, click, s, sleep }) => { + await setLocale(cdp, locale, WORKSPACE); + await cdp.waitFor(`document.querySelector('${s.blankProject}')`); + await click(s.blankProject); + await cdp.waitFor("location.pathname.startsWith('/flow/')"); + await cdp.waitFor("document.querySelector('.react-flow')"); + await sleep(600); + + const buttonExpression = (label) => + `[...document.querySelectorAll('button')].some((button) => (button.innerText || '').trim() === ${JSON.stringify(label)})`; + const maybeClickOnboarding = async (label, waitMs = 0) => { + const deadline = Date.now() + waitMs; + while (!(await cdp.eval(buttonExpression(label)))) { + if (Date.now() >= deadline) return false; + await sleep(120); + } + await click(`text=${label}`); + return true; + }; + await maybeClickOnboarding("Got it", 1500); + await maybeClickOnboarding("Skip", 1500); + await closePropertiesInspector(cdp); + await suppressDuplicateIrisPresence(cdp); + await installTutorialLayout(cdp, [GEMINI_NODE, KLING_NODE]); + }, + + teardown: async ({ cdp }) => { + await removeTutorialLayout(cdp); + return cdp.eval(`(() => { + window.__captutorPresenceObserver?.disconnect(); + delete window.__captutorPresenceObserver; + for (const element of document.querySelectorAll('[data-captutor-duplicate-presence]')) { + element.style.display = ''; + delete element.dataset.captutorDuplicatePresence; + } + return true; + })()`); + }, + + beats: [ + { + say: "Learn Fuser by making a Talking Taco.", + logic: "Open on the genuinely empty canvas the whole lesson builds on.", + cursorIntent: "Park in the middle of the empty pane.", + do: async ({ cdp, check, point }) => { + await point(".react-flow__pane", { moveMs: 620 }); + check("fresh_blank_project_opened", await cdp.eval(`({ + pathname: location.pathname, + nodeCount: document.querySelectorAll('.react-flow__node').length, + })`)); + }, + }, + { + say: "Start in a fresh project — an empty canvas, ready for your first idea.", + do: async ({ point }) => point(".react-flow__pane", { moveMs: 520, anchorX: 0.5, anchorY: 0.4 }), + }, + { + say: "Open the node picker, and search for Gemini Image — Fuser's fast image generator.", + do: async ({ cdp, click, type, s, t }) => { + await click(ADD_NODE); + await cdp.waitFor(`document.querySelector('${s.nodeSearch}')`); + await type(s.nodeSearch, t("flow.nodes.FalGeminiImageNode.name")); + }, + }, + { + say: "Press Enter, and it drops right onto the canvas.", + do: async (ctx) => { + const { cdp, check, click, s, sleep } = ctx; + // Refocus the search field before Enter — a prior click elsewhere can + // steal focus, and a synthetic Enter with no field focused submits + // nothing (see drive-ui.md). + await click(s.nodeSearch); + await cdp.key("Enter", "Enter", 13); + await cdp.waitFor("document.querySelectorAll('.react-flow__node').length === 1"); + await sleep(500); + await setTutorialZoom(ctx, 80); + await sleep(320); + await frameTutorialNodes(ctx, [{ selector:GEMINI_NODE, title:"Gemini Image" }]); + await assertTutorialLayout(ctx, [GEMINI_NODE]); + check("image_node_added", await cdp.eval(`({ + nodeClass: document.querySelector('.react-flow__node')?.className, + })`)); + }, + }, + { + say: "Describe the image you want. Let's make a talking taco: bold, cheerful, mouth wide open mid-laugh.", + do: async (ctx) => { + const { cdp, check } = ctx; + await typePrompt(ctx, GEMINI_PROMPT, TACO_IMAGE_PROMPT); + check("image_prompt_entered", { text: await cdp.eval(`document.querySelector(${JSON.stringify(GEMINI_PROMPT)}).value`) }); + }, + }, + { + say: "Generate — and Fuser turns that prompt into a real image in seconds.", + do: async (ctx) => { + const { cdp, check } = ctx; + creditsBeforeRun = await cdp.eval(creditsExpr); + const started = `(() => { + const credits = ${creditsExpr}; + const node = document.querySelector(${JSON.stringify(GEMINI_NODE)}); + return credits !== ${JSON.stringify(creditsBeforeRun)} || + !!node?.querySelector('[aria-busy=true],[role=progressbar],.animate-spin'); + })()`; + await runNode(ctx, GEMINI_NODE, started); + check("image_generation_started", { creditsBeforeRun }); + }, + }, + { + say: "There it is — a fresh, generated talking taco, ready to bring to life.", + do: async (ctx) => { + const { cdp, check, click, point, spotlight, sleep } = ctx; + await cdp.waitFor( + `(() => { + const image = document.querySelector(${JSON.stringify(GEMINI_GENERATED_IMAGE)}); + return !!image?.complete && image.naturalWidth > 0; + })()`, + { timeoutMs: 120000, everyMs: 250 }, + ); + const returned = await cdp.eval(`(() => { + const image = document.querySelector(${JSON.stringify(GEMINI_GENERATED_IMAGE)}); + return { src: image.currentSrc || image.src, width: image.naturalWidth, height: image.naturalHeight }; + })()`); + check("generated_image_returned", returned); + await frameTutorialNodes(ctx, [{ selector:GEMINI_NODE, title:"Gemini Image" }]); + await assertTutorialLayout(ctx, [GEMINI_NODE]); + // Deselect so the node's own toolbar/side panel closes before framing. + await click(".react-flow__pane", { moveMs: 260, anchorX: 0.90, anchorY: 0.10 }); + await sleep(400); + await point(".react-flow__pane", { moveMs: 420, anchorX: 0.30, anchorY: 0.88 }); + await spotlight(GEMINI_GENERATED_IMAGE, { + label: "Generated image", dim: 0.26, ring: true, feather: 30, durationMs: 3200, + }); + }, + }, + { + say: "Now add a second node: Kling 3.0 Video, Fuser's image-to-video model.", + do: async ({ cdp, click, type, s, t }) => { + await click(ADD_NODE); + await cdp.waitFor(`document.querySelector('${s.nodeSearch}')`); + await type(s.nodeSearch, t("flow.nodes.FalKling30VideoNode.name")); + }, + }, + { + say: "Press Enter to add it, then slide it next to the image so both stay in view.", + do: async (ctx) => { + const { cdp, check, click, s, sleep } = ctx; + await click(s.nodeSearch); + await cdp.key("Enter", "Enter", 13); + await cdp.waitFor("document.querySelectorAll('.react-flow__node').length === 2"); + await sleep(500); + await click(".react-flow__pane", { moveMs: 260, anchorX: 0.06, anchorY: 0.94 }); + await sleep(300); + await setTutorialZoom(ctx, 80); + await sleep(320); + await frameTutorialNodes(ctx, [ + { selector:GEMINI_NODE, title:"Gemini Image" }, + { selector:KLING_NODE, title:"Kling 3.0 Video" }, + ]); + await assertTutorialLayout(ctx, [GEMINI_NODE, KLING_NODE]); + check("video_node_added", await cdp.eval(`({ + nodeCount: document.querySelectorAll('.react-flow__node').length, + })`)); + }, + }, + { + say: "Connect the taco image's output straight into the video node's Image input — that is the exact frame it will animate.", + do: async ({ cdp, check, drag, outline }) => { + await drag(GEMINI_OUTPUT_HANDLE, KLING_IMAGE_INPUT_HANDLE, { moveMs: 520, dragMs: 720 }); + await cdp.waitFor("document.querySelectorAll('.react-flow__edge').length === 1"); + const rect = await outline(".react-flow__edge", { + label: "Image feeds the video node", feather: 24, durationMs: 3200, + }); + check("image_to_video_edge_connected", { edgeCount: 1, rect }); + }, + }, + { + say: "Add a short motion prompt describing how the taco should move and talk.", + do: async (ctx) => { + const { cdp, check } = ctx; + await typePrompt(ctx, KLING_PROMPT, TACO_MOTION_PROMPT); + check("video_prompt_entered", { text: await cdp.eval(`document.querySelector(${JSON.stringify(KLING_PROMPT)}).value`) }); + }, + }, + { + say: "Generate the video, and Fuser animates that still image into motion.", + do: async (ctx) => { + const { cdp, check } = ctx; + creditsBeforeRun = await cdp.eval(creditsExpr); + const started = `(() => { + const credits = ${creditsExpr}; + const node = document.querySelector(${JSON.stringify(KLING_NODE)}); + return credits !== ${JSON.stringify(creditsBeforeRun)} || + !!node?.querySelector('[aria-busy=true],[role=progressbar],.animate-spin'); + })()`; + await runNode(ctx, KLING_NODE, started); + check("video_generation_started", { creditsBeforeRun }); + }, + }, + { + say: "And here it is — the taco, talking and gesturing, generated straight from the image you made a moment ago.", + do: async (ctx) => { + const { cdp, check, click, point, spotlight, sleep } = ctx; + await cdp.waitFor( + `(() => { + const video = document.querySelector(${JSON.stringify(KLING_GENERATED_VIDEO)}); + return !!video && video.readyState >= 2 && !!(video.currentSrc || video.src); + })()`, + { timeoutMs: 600000, everyMs: 1000 }, + ); + const returned = await cdp.eval(`(() => { + const video = document.querySelector(${JSON.stringify(KLING_GENERATED_VIDEO)}); + return { src: video.currentSrc || video.src, duration: video.duration }; + })()`); + check("generated_video_returned", returned); + await frameTutorialNodes(ctx, [ + { selector:GEMINI_NODE, title:"Gemini Image" }, + { selector:KLING_NODE, title:"Kling 3.0 Video" }, + ]); + await assertTutorialLayout(ctx, [GEMINI_NODE, KLING_NODE]); + await click(".react-flow__pane", { moveMs: 260, anchorX: 0.90, anchorY: 0.10 }); + await sleep(400); + await point(".react-flow__pane", { moveMs: 420, anchorX: 0.72, anchorY: 0.88 }); + await spotlight(KLING_GENERATED_VIDEO, { + label: "Generated from the taco image", dim: 0.24, ring: true, + labelPosition: "side", feather: 30, durationMs: 3800, + }); + }, + }, + { + say: "That's Fuser end to end: one prompt became a generated image, and that same image became a generated video — all on one connected canvas.", + do: async (ctx) => { + const { cdp, check, point } = ctx; + await frameTutorialNodes(ctx, [ + { selector:GEMINI_NODE, title:"Gemini Image" }, + { selector:KLING_NODE, title:"Kling 3.0 Video" }, + ]); + await assertTutorialLayout(ctx, [GEMINI_NODE, KLING_NODE]); + await point(".react-flow__pane", { moveMs: 720, anchorX: 0.5, anchorY: 0.94 }); + check("final_provenance_tableau_complete", await cdp.eval(`({ + nodes: document.querySelectorAll('.react-flow__node').length, + edges: document.querySelectorAll('.react-flow__edge').length, + imageLoaded: (() => { + const image = document.querySelector(${JSON.stringify(GEMINI_GENERATED_IMAGE)}); + return !!image?.complete && image.naturalWidth > 0; + })(), + videoReady: (() => { + const video = document.querySelector(${JSON.stringify(KLING_GENERATED_VIDEO)}); + return !!video && video.readyState >= 2; + })(), + })`)); + }, + }, + ], +}; diff --git a/captutor/test/cdp-health.test.mjs b/captutor/test/cdp-health.test.mjs new file mode 100644 --- /dev/null +++ b/captutor/test/cdp-health.test.mjs @@ -0,0 +1,93 @@ +import assert from "node:assert/strict"; +import test from "node:test"; + +import { BrowserCrashError, Session } from "../lib/cdp.mjs"; + +class FakeWebSocket { + static instances = []; + + constructor() { + this.listeners = new Map(); + this.readyState = 0; + this.respond = true; + FakeWebSocket.instances.push(this); + queueMicrotask(() => { + this.readyState = 1; + this.emit("open", {}); + }); + } + + addEventListener(type, listener, options = {}) { + const wrapped = options.once + ? (event) => { + this.listeners.get(type)?.delete(wrapped); + listener(event); + } + : listener; + if (!this.listeners.has(type)) this.listeners.set(type, new Set()); + this.listeners.get(type).add(wrapped); + } + + emit(type, event) { + for (const listener of [...(this.listeners.get(type) || [])]) listener(event); + } + + message(payload) { + this.emit("message", { data:JSON.stringify(payload) }); + } + + send(raw) { + const request = JSON.parse(raw); + if (!this.respond) return; + queueMicrotask(() => this.message({ + id:request.id, + result:request.method === "Runtime.evaluate" + ? { result:{ value:{ readyState:"complete", href:"https://app.fuser.studio/flow/test" } } } + : {}, + })); + } + + close() { + this.readyState = 3; + this.emit("close", {}); + } +} + +test("CDP health accepts a responsive renderer and catches its crash event", async (t) => { + const RealWebSocket = globalThis.WebSocket; + globalThis.WebSocket = FakeWebSocket; + t.after(() => { globalThis.WebSocket = RealWebSocket; }); + + const session = new Session("ws://fake.test/page/one"); + t.after(() => session.close()); + await session.monitorCrashes(); + assert.equal((await session.assertHealthy()).readyState, "complete"); + + const ws = FakeWebSocket.instances.at(-1); + ws.message({ method:"Inspector.targetCrashed", params:{ status:"crashed" } }); + await assert.rejects( + session.assertHealthy("beat 3"), + (error) => error instanceof BrowserCrashError + && error.code === "BROWSER_RENDERER_CRASH" + && error.details.signal === "Inspector.targetCrashed", + ); +}); + +test("CDP health converts an unresponsive renderer into a bounded crash failure", async (t) => { + const RealWebSocket = globalThis.WebSocket; + globalThis.WebSocket = FakeWebSocket; + t.after(() => { globalThis.WebSocket = RealWebSocket; }); + + const session = new Session("ws://fake.test/page/two"); + t.after(() => session.close()); + await session.monitorCrashes(); + FakeWebSocket.instances.at(-1).respond = false; + + const started = Date.now(); + await assert.rejects( + session.assertHealthy("pre-record", { timeoutMs:25 }), + (error) => error instanceof BrowserCrashError + && error.details.signal === "CDP.unresponsive", + ); + assert.ok(Date.now() - started < 500, "health failure should be bounded"); +}); diff --git a/captutor/test/cursor.test.mjs b/captutor/test/cursor.test.mjs new file mode 100644 --- /dev/null +++ b/captutor/test/cursor.test.mjs @@ -0,0 +1,35 @@ +import assert from "node:assert/strict"; +import test from "node:test"; + +import { pagePointToScreen } from "../lib/cursor.mjs"; + +test("native cursor hotspot maps page pixels through browser chrome exactly", () => { + const geometry = { + screenX:40, + screenY:24, + outerWidth:1000, + outerHeight:760, + innerWidth:984, + innerHeight:650, + }; + // Eight points of symmetric side frame and 102 points of top chrome. + assert.deepEqual(pagePointToScreen(geometry, { x:200, y:90 }), { + x:248, + y:216, + }); +}); + +test("native cursor mapping does not invent negative browser borders", () => { + const geometry = { + screenX:-1280, + screenY:0, + outerWidth:900, + outerHeight:700, + innerWidth:920, + innerHeight:720, + }; + assert.deepEqual(pagePointToScreen(geometry, { x:20, y:30 }), { + x:-1260, + y:30, + }); +}); diff --git a/captutor/test/iris-heartbeat-recovery.test.mjs b/captutor/test/iris-heartbeat-recovery.test.mjs new file mode 100644 --- /dev/null +++ b/captutor/test/iris-heartbeat-recovery.test.mjs @@ -0,0 +1,41 @@ +import assert from "node:assert/strict"; +import test from "node:test"; + +import { + beginRecovery, latestRecoverableFailure, queueRecovery, +} from "../ops/iris-heartbeat-recovery.mjs"; + +test("heartbeat selects the newest failed Captutor mission only while idle", () => { + const state = { + active:null, + done:{ + old:{ name:"Old", kind:"captutor", status:"failed", at:10 }, + code:{ name:"Code", kind:"pr", status:"failed", at:30 }, + taco:{ name:"Talking Taco", kind:"captutor", status:"failed", at:20 }, + }, + }; + assert.equal(latestRecoverableFailure(state).taskGid, "taco"); + state.active = { taskGid:"live" }; + assert.equal(latestRecoverableFailure(state), null); +}); + +test("heartbeat queues exactly one retry and preserves an audit trail", () => { + const state = { + active:null, + done:{ taco:{ + name:"Talking Taco", kind:"captutor", status:"failed", + reason:"missing-outbox-artifacts", detail:"worker exited 0", at:20, + } }, + }; + const failure = latestRecoverableFailure(state); + assert.equal(beginRecovery(state, failure, 100, 1), true); + const recovery = queueRecovery(state, failure, 101); + assert.equal(recovery.status, "queued"); + assert.equal(recovery.attempts, 1); + assert.equal(state.done.taco, undefined); + assert.equal(state.recoveries.taco.reason, "missing-outbox-artifacts"); + + state.done.taco = { ...failure, status:"failed", at:200 }; + assert.equal(beginRecovery(state, latestRecoverableFailure(state), 201, 1), false); + assert.equal(state.recovery.status, "exhausted"); +}); diff --git a/captutor/test/stage-contract.test.mjs b/captutor/test/stage-contract.test.mjs new file mode 100644 --- /dev/null +++ b/captutor/test/stage-contract.test.mjs @@ -0,0 +1,61 @@ +import assert from "node:assert/strict"; +import test from "node:test"; + +import { assertHiDPIStage, StageContractError } from "../lib/stage-contract.mjs"; +import { normalizeStageBrand, parseStageFlags } from "../lib/stage-mode.mjs"; + +test("Stage brand is explicit and defaults to the dimensional Fuser treatment", () => { + assert.equal(normalizeStageBrand(), "fuser"); + assert.equal(normalizeStageBrand("FUSER"), "fuser"); + assert.equal(normalizeStageBrand("classic"), "classic"); + assert.throws(() => normalizeStageBrand("unknown"), /unsupported Captutor Stage brand/); +}); + +test("Stage flags preserve the Captutor command with and without an explicit brand", () => { + assert.deepEqual(parseStageFlags(["render", "intro"]), { + vertical:false, brand:"fuser", args:["render", "intro"], + }); + assert.deepEqual(parseStageFlags(["--vertical", "--brand", "classic", "render", "intro", "--locale", "fr"]), { + vertical:true, brand:"classic", args:["render", "intro", "--locale", "fr"], + }); + assert.throws(() => parseStageFlags(["--brand", "--vertical", "render", "intro"]), /needs a value/); +}); + +test("fleet mission takes require the Stage wrapper", () => { + assert.throws( + () => assertHiDPIStage({ required:true, stageMode:false }), + (error) => error instanceof StageContractError + && error.code === "CAPTUTOR_HIDPI_STAGE_REQUIRED", + ); +}); + +test("landscape Stage accepts the real 1280x720 2x display", () => { + assert.deepEqual( + assertHiDPIStage({ + required:true, + stageMode:true, + screen:{ width:1280, height:720, dpr:2 }, + }), + { width:1280, height:720, dpr:2 }, + ); +}); + +test("ordinary desktop geometry cannot masquerade as Stage", () => { + assert.throws( + () => assertHiDPIStage({ + required:true, + stageMode:true, + screen:{ width:2560, height:1440, dpr:1 }, + }), + /HiDPI Stage is not active/, + ); +}); + +test("portrait missions require the rotated 2x Stage display", () => { + assert.doesNotThrow(() => assertHiDPIStage({ + required:true, + stageMode:true, + vertical:true, + screen:{ width:720, height:1280, dpr:2 }, + })); +}); diff --git a/captutor/test/tutorial-layout.test.mjs b/captutor/test/tutorial-layout.test.mjs new file mode 100644 --- /dev/null +++ b/captutor/test/tutorial-layout.test.mjs @@ -0,0 +1,47 @@ +import test from "node:test"; +import assert from "node:assert/strict"; + +import { tutorialLayoutScores } from "../lib/tutorial-layout.mjs"; + +const measured = (nodes) => ({ + eval: async () => ({ + viewport:{ width:1190, height:630 }, + safe:{ left:88, top:76, right:1118, bottom:480 }, + nodes, + }), +}); + +test("ui-legibility-score accepts full labels in the safe region", async () => { + const scores = await tutorialLayoutScores(measured([{ + selector:".image", truncated:[], layoutWidth:336, + rect:{ left:468.5, top:104, right:737.5, bottom:452, width:269, height:348, cx:603, cy:278 }, + }]), [".image"]); + assert.equal(scores["ui-legibility-score"].score, 100); + assert.equal(scores["balanced-layout-score"].score, 100); +}); + +test("scores reject truncated labels and chat-zone collisions", async () => { + const scores = await tutorialLayoutScores(measured([{ + selector:".image", truncated:["Nano Banan…"], + rect:{ left:450, top:180, right:674, bottom:560, width:224, height:380, cx:562, cy:370 }, + }]), [".image"]); + assert.ok(scores["ui-legibility-score"].score < scores["ui-legibility-score"].threshold); + assert.ok(scores["balanced-layout-score"].score < scores["balanced-layout-score"].threshold); +}); + +test("balanced-layout-score rewards an aligned two-node teaching tableau", async () => { + const scores = await tutorialLayoutScores(measured([ + { + selector:".image", truncated:[], + layoutWidth:336, + rect:{ left:235, top:104, right:571, bottom:452, width:336, height:348, cx:403, cy:278 }, + }, + { + selector:".video", truncated:[], + layoutWidth:336, + rect:{ left:635, top:104, right:971, bottom:452, width:336, height:348, cx:803, cy:278 }, + }, + ]), [".image", ".video"]); + assert.equal(scores["ui-legibility-score"].score, 100); + assert.equal(scores["balanced-layout-score"].score, 100); +}); diff --git a/slab/bin/frame-mcp.mjs b/slab/bin/frame-mcp.mjs --- a/slab/bin/frame-mcp.mjs +++ b/slab/bin/frame-mcp.mjs @@ -23,6 +23,7 @@ import { fileURLToPath } from "node:url"; import { homedir, tmpdir } from "node:os"; import { httpPort, serveHttp, serveStdio } from "../../toolchain/mcp/http-front.mjs"; import { clickPoint, hoverPoint, sendKeys } from "./macos.mjs"; +import { buildHoverProbes, changesNearPoint } from "../lib/frame-hover-atlas.mjs"; const HERE = dirname(fileURLToPath(import.meta.url)); const REPO = resolve(HERE, "../.."); @@ -157,6 +158,7 @@ // ── the capture tool: frame a machine, return image + digest ──────────────── const stagedClicks = new Map(); const recentActionTrails = new Map(); +const recentFrames = new Map(); const visionCache = new Map(); async function captureFrame({ machine, ocr = true, fast = false, screen = false, cursor = true, cursorAt, targetAt, targetId, manualCheck, pressAt, pressCount = 1, pressTitle, actionOnly = false, clearTarget = false, clearOverlays = false, quietOverlay = false, crop, baseline = false, diff = false } = {}) { @@ -225,7 +227,9 @@ } // FRAME establishes the stable observation baseline used by later reframes. async function toolInitialFrame(options = {}) { - return toolFrame({ ...options, baseline: true }); + const capture = await captureFrame({ ...options, baseline: true }); + recentFrames.set(options.machine, capture.env); + return frameContent(capture, options.machine); } // REFRAME is change-driven: one silent full-window diff probe advances the @@ -810,6 +814,84 @@ return toolFrame({ machine, ocr, fast, cursorAt: [x, y], crop, diff: true, baseline: true, quietOverlay: true }); } +function hoverAtlasText(mode, results) { + const observed = results.filter((result) => result.near.cells > 0).length; + const lines = [ + `HOVER AFFORDANCE ATLAS — ${mode}: ${results.length} no-click probes, ${observed} local visual changes`, + "Possible actions are evidence, not actions performed:", + ]; + for (const result of results) { + const state = result.near.cells > 0 ? "observed-hover-change" : "candidate"; + const label = result.probe.label ? ` “${result.probe.label}”` : ""; + lines.push( + ` [${state}] ${result.probe.kind}${label} @(${result.probe.x},${result.probe.y})` + + ` — ${result.probe.possibility}; local diff cells=${result.near.cells}`, + ); + } + lines.push("No clicks, drags, or resizes were performed. Use an intentional action tool after choosing an affordance."); + return lines.join("\n"); +} + +async function toolHoverAtlas({ + machine, mode = "wanderer", x, y, radius = 18, steps, + settleMs = 140, ocr = true, fast = true, +} = {}) { + const spec = machineSpec(machine); + settleMs = Math.max(60, Math.min(600, Number(settleMs) || 140)); + const initial = await captureFrame({ + machine, ocr: false, fast: true, cursor: false, + baseline: true, quietOverlay: true, + }); + recentFrames.set(machine, initial.env); + const probes = buildHoverProbes(initial.env, { mode, x, y, radius, steps }); + const results = []; + for (const probe of probes) { + hoverPoint(spec, probe.x, probe.y); + await settle(settleMs); + const capture = await captureFrame({ + machine, ocr: false, fast: true, cursor: false, + diff: true, baseline: true, quietOverlay: true, + }); + results.push({ probe, near:changesNearPoint(capture.env, probe.x, probe.y), capture }); + } + + const ranked = [...results].sort((a, b) => + b.near.cells - a.near.cells || b.near.count - a.near.count); + const representative = ranked.find((result) => result.near.cells > 0) || ranked[0]; + const content = [{ type:"text", text:hoverAtlasText(mode, results) }]; + if (representative) { + hoverPoint(spec, representative.probe.x, representative.probe.y); + await settle(settleMs); + const bounds = captureBounds(initial.env); + const crop = clampCrop([ + representative.probe.x - 240, representative.probe.y - 180, 480, 360, + ], bounds); + const detail = await captureFrame({ + machine, ocr, fast, cursor: false, crop, quietOverlay: true, + }); + if (detail.jpg) content.unshift({ + type:"image", data:detail.jpg.toString("base64"), mimeType:"image/jpeg", + }); + content.push({ + type:"text", + text:`\nMOST INFORMATIVE PROBE — ${representative.probe.kind} @(${representative.probe.x},${representative.probe.y})\n${digest(detail.env)}`, + }); + } + const original = initial.env?.meta?.cursor; + if (Number.isFinite(original?.x) && Number.isFinite(original?.y)) { + hoverPoint(spec, original.x, original.y); + } + return content; +} + +async function toolWander(args = {}) { + return toolHoverAtlas({ ...args, mode:"wanderer" }); +} + +async function toolWiggle(args = {}) { + return toolHoverAtlas({ ...args, mode:"wiggler" }); +} + // Native exploration primitives return the post-action frame in the SAME MCP // response. Agents need one tool round-trip, not act → wait → call frame again. async function toolClick({ machine, x, y, count = 1, ocr = true, fast = true }) { @@ -1056,6 +1138,35 @@ description: "OBSERVES CONTEXT: move the real pointer without clicking, wait for hover-only controls/tooltips, then return a cheaper cropped reframe around that point. Lesson 1: when an element may reveal options, hover and reframe before clicking. Coordinates remain global and click-ready.", inputSchema: { type: "object", properties: { machine: { type: "string" }, x: { type: "number" }, y: { type: "number" }, width: { type: "number", description: "Crop width (default 720)." }, height: { type: "number", description: "Crop height (default 520)." }, ocr: { type: "boolean" }, fast: { type: "boolean" } }, required: ["machine", "x", "y"] }, }, { + name: "frame_wander", + description: "WANDERER: without clicking, sweep likely buttons, hover controls, the titlebar, and focused-window resize edges/corners. Performs rapid mouse-move visual diffs and returns a hover-affordance atlas plus the most informative crop. Run after a frame when you need more context about what the current surface may allow.", + inputSchema: { + type:"object", + properties:{ + machine:{ type:"string" }, + steps:{ type:"number", minimum:1, maximum:24, description:"Maximum no-click probes (default 14)." }, + settleMs:{ type:"number", minimum:60, maximum:600, description:"Hover settle time per probe (default 140 ms)." }, + ocr:{ type:"boolean" }, fast:{ type:"boolean" }, + }, + required:["machine"], + }, + }, + { + name: "frame_wiggle", + description: "WIGGLER: without clicking, micro-sweep around one global screen coordinate to expose hover boundaries, cursor-shape transitions, nearby controls, or resize affordances. Uses quick visual diffs and returns an affordance atlas plus the strongest changed crop.", + inputSchema: { + type:"object", + properties:{ + machine:{ type:"string" }, x:{ type:"number" }, y:{ type:"number" }, + radius:{ type:"number", minimum:4, maximum:120, description:"Sweep radius in screen points (default 18)." }, + steps:{ type:"number", minimum:1, maximum:24, description:"Maximum no-click probes (default 9)." }, + settleMs:{ type:"number", minimum:60, maximum:600, description:"Hover settle time per probe (default 140 ms)." }, + ocr:{ type:"boolean" }, fast:{ type:"boolean" }, + }, + required:["machine", "x", "y"], + }, + }, + { name: "frame_click", description: "ACTS + OBSERVES: click a native macOS screen coordinate from frame OCR/AX, then immediately return a fresh frame with a virtual marker at the click. Use for low-risk UI exploration; inspect labels and avoid destructive controls.", inputSchema: { @@ -1156,6 +1267,8 @@ case "frame_clear_overlays": return toolClearOverlays(args || {}); case "frame_describe": return toolDescribe(args || {}); case "frame_design": return toolDesign(args || {}); case "frame_hover": return toolHover(args || {}); + case "frame_wander": return toolWander(args || {}); + case "frame_wiggle": return toolWiggle(args || {}); case "frame_click": return toolClick(args || {}); case "frame_stage_click": return toolStageClick(args || {}); case "frame_commit_click": return toolCommitClick(args || {}); diff --git a/slab/lib/frame-hover-atlas.mjs b/slab/lib/frame-hover-atlas.mjs new file mode 100644 --- /dev/null +++ b/slab/lib/frame-hover-atlas.mjs @@ -0,0 +1,104 @@ +// Pure geometry for Frame's no-click hover exploration. + +const clamp = (value, low, high) => Math.max(low, Math.min(high, value)); + +function boundsOf(env) { + const crop = env?.crop; + if (crop && [crop.x, crop.y, crop.w, crop.h].every(Number.isFinite)) { + return [crop.x, crop.y, crop.w, crop.h]; + } + const screen = env?.meta?.screen; + if (screen && [screen.w, screen.h].every(Number.isFinite)) { + return [0, 0, screen.w, screen.h]; + } + throw new Error("Frame hover exploration needs capture bounds"); +} + +function inside([bx, by, bw, bh], x, y) { + return x >= bx && x <= bx + bw && y >= by && y <= by + bh; +} + +function unique(points, bounds, limit) { + const seen = new Set(); + const result = []; + for (const point of points) { + const x = Math.round(Number(point.x)); + const y = Math.round(Number(point.y)); + if (!Number.isFinite(x) || !Number.isFinite(y) || !inside(bounds, x, y)) continue; + const key = `${Math.round(x / 8)},${Math.round(y / 8)}`; + if (seen.has(key)) continue; + seen.add(key); + result.push({ ...point, x, y }); + if (result.length >= limit) break; + } + return result; +} + +export function buildHoverProbes(env, { + mode = "wanderer", x, y, radius = 18, steps = mode === "wiggler" ? 9 : 14, +} = {}) { + const bounds = boundsOf(env); + const limit = clamp(Math.round(Number(steps) || 1), 1, 24); + if (mode === "wiggler") { + x = Number(x); y = Number(y); radius = clamp(Number(radius) || 18, 4, 120); + if (!Number.isFinite(x) || !Number.isFinite(y)) { + throw new Error("wiggler needs finite x and y coordinates"); + } + const ring = [ + [0, 0], [-1, 0], [1, 0], [0, -1], [0, 1], + [-0.72, -0.72], [0.72, -0.72], [0.72, 0.72], [-0.72, 0.72], + [-0.45, 0], [0.45, 0], [0, -0.45], [0, 0.45], + ]; + return unique(ring.map(([dx, dy], index) => ({ + x:x + dx * radius, y:y + dy * radius, + kind:index === 0 ? "probe-center" : "hover-boundary", + possibility:"hover boundary or cursor-shape change", + })), bounds, limit); + } + + if (mode !== "wanderer") throw new Error("mode must be wanderer or wiggler"); + const [bx, by, bw, bh] = bounds; + const inset = Math.max(2, Math.min(6, Math.round(Math.min(bw, bh) * 0.004))); + const points = []; + + // Semantic controls first, followed by image-discovered compact controls. + for (const element of env?.ax?.elements || []) { + const actions = element.actions || []; + if (!actions.length && !/button|link|menu|control/i.test(element.role || "")) continue; + points.push({ + x:element.cx, y:element.cy, kind:`ax-${element.role || "control"}`, + label:String(element.title || "").replace(/\s+/g, " ").trim().slice(0, 80), + possibility:actions.includes("AXPress") ? "button or pressable control" : "interactive control", + }); + } + // Focused-window geometry provides useful probes even when AX is sparse. + points.push( + { x:bx + bw / 2, y:by + Math.min(24, bh * 0.035), kind:"window-title", possibility:"drag window" }, + { x:bx + inset, y:by + bh / 2, kind:"window-left-edge", possibility:"resize window horizontally" }, + { x:bx + bw - inset, y:by + bh / 2, kind:"window-right-edge", possibility:"resize window horizontally" }, + { x:bx + bw / 2, y:by + bh - inset, kind:"window-bottom-edge", possibility:"resize window vertically" }, + { x:bx + inset, y:by + bh - inset, kind:"window-bottom-left-corner", possibility:"resize window diagonally" }, + { x:bx + bw - inset, y:by + bh - inset, kind:"window-bottom-right-corner", possibility:"resize window diagonally" }, + ); + for (const control of env?.visual || []) { + points.push({ + x:control.cx, y:control.cy, kind:control.kind || "visual-control", + possibility:"potential button or hover-only control", + }); + } + return unique(points, bounds, limit); +} + +export function changesNearPoint(env, x, y, radius = 150) { + const changes = (env?.diff || []).filter((change) => { + const rect = Array.isArray(change.r) ? change.r.map(Number) : null; + if (!rect || rect.length !== 4 || !rect.every(Number.isFinite)) return false; + const cx = rect[0] + rect[2] / 2; + const cy = rect[1] + rect[3] / 2; + return Math.hypot(cx - x, cy - y) <= radius; + }); + return { + count:changes.length, + cells:changes.reduce((sum, change) => sum + Math.max(1, Number(change.cells) || 1), 0), + }; +} diff --git a/slab/test/frame-hover-atlas.test.mjs b/slab/test/frame-hover-atlas.test.mjs new file mode 100644 --- /dev/null +++ b/slab/test/frame-hover-atlas.test.mjs @@ -0,0 +1,35 @@ +import assert from "node:assert/strict"; +import test from "node:test"; + +import { buildHoverProbes, changesNearPoint } from "../lib/frame-hover-atlas.mjs"; + +const env = { + crop:{ x:100, y:80, w:900, h:620 }, + ax:{ elements:[{ role:"AXButton", title:"Share", cx:850, cy:130, actions:["AXPress"] }] }, + visual:[{ kind:"compact-control", cx:180, cy:240 }], +}; + +test("wanderer combines controls with draggable and resizable window surfaces", () => { + const probes = buildHoverProbes(env, { mode:"wanderer", steps:20 }); + assert.ok(probes.some((probe) => probe.possibility === "button or pressable control")); + assert.ok(probes.some((probe) => probe.possibility === "drag window")); + assert.ok(probes.some((probe) => probe.possibility === "resize window diagonally")); + assert.ok(probes.every((probe) => probe.x >= 100 && probe.x <= 1000)); +}); + +test("wiggler makes a bounded no-click ring around the supplied point", () => { + const probes = buildHoverProbes(env, { mode:"wiggler", x:500, y:300, radius:20, steps:9 }); + assert.equal(probes.length, 9); + assert.deepEqual(probes[0], { + x:500, y:300, kind:"probe-center", + possibility:"hover boundary or cursor-shape change", + }); +}); + +test("quick diff scoring isolates changes near the moved pointer", () => { + const score = changesNearPoint({ diff:[ + { r:[490, 290, 20, 20], cells:8 }, + { r:[900, 600, 20, 20], cells:40 }, + ] }, 500, 300, 80); + assert.deepEqual(score, { count:1, cells:8 }); +});