From 605a06d4844f02758e0976df09601cd334d4d3eb Mon Sep 17 00:00:00 2001 From: onevcat Date: Sat, 18 Apr 2026 19:53:29 +0900 Subject: [PATCH] Add observability runbook for future diagnostic sessions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The observability reboot (PRs #207, #208, #210, #211, #212) touched seven files across SDK init, per-event instrumentation, release pipeline, and build config. Without a map, picking it up again in six months means re-deriving why each piece exists. This doc is that map. It is deliberately a runbook, not an architecture essay: the first thing it answers is "I'm staring at a weird memory report, where do I look?" — and the playbooks walk through Sentry + PostHog queries in the order that gets to the answer fastest. Organized as: - TL;DR (3 bullets, skim in 10s) - Stack overview (both pipelines, identity, credential flow) - CLI tools (sentry vs sentry-cli, which does what) - Diagnostic playbooks (memory explosion, suspicious Sentry issue, DAU) - Event catalog (super properties + 16 events, what carries what) - Known noise and filters (system hangs, PostHog autocapture, 403s that look like bugs but aren't) - File map + quick reference for common modifications - Open gaps + links to all 5 observability PRs Covers the post-#212 state so the doc is accurate the moment that PR merges — document this PR should land after #212. --- doc-onevcat/observability.md | 205 +++++++++++++++++++++++++++++++++++ 1 file changed, 205 insertions(+) create mode 100644 doc-onevcat/observability.md diff --git a/doc-onevcat/observability.md b/doc-onevcat/observability.md new file mode 100644 index 00000000..7e58f0a3 --- /dev/null +++ b/doc-onevcat/observability.md @@ -0,0 +1,205 @@ +# Observability Runbook + +Quick-start reference for investigating production issues using the Sentry + PostHog pipelines built into Prowl. When you (or an agent) come back to diagnose something, read this first — it's the map. + +## TL;DR + +- **Crashes / errors / App Hangs** → Sentry (`sentry issue view ` or https://onevs-den.sentry.io/) +- **User behavior / product questions / memory trends** → PostHog (dashboard) +- **Long-running memory growth** → PostHog events `app_memory_baseline` and `memory_threshold_mb` (see [Memory explosion playbook](#memory-explosion)) +- **Credentials + scopes** → `~/.sentryclirc` (`event:read`, `project:read`, `project:releases`) +- **Debug builds send nothing** — `#if !DEBUG` gates every SDK call + +## Stack overview + +Two independent pipelines, both initialized in `supacode/App/supacodeApp.swift > bootstrapTelemetry`: + +### Sentry (crash + error + App Hang) + +- `options.releaseName = "prowl@"` — matches the release name that `release.sh` registers via `sentry-cli releases new` +- `options.environment = "tip"` for tip-channel users, `"production"` otherwise +- `options.tracesSampleRate = 0.05` — 5% of transactions get performance traces +- `options.enableAppHangTracking = true` + `appHangTimeoutInterval = 3` — main-thread stalls ≥ 3s get reported +- `options.beforeSend = SentryEventFilter.filterSystemHang` — client-side filter for known system-induced hangs +- Watchdog Termination **not** enabled (unsupported on native macOS) + +### PostHog (product analytics) + +- 16 hand-instrumented events — no autocapture +- `enableSwizzling = false`, `captureApplicationLifecycleEvents = false`, `captureScreenViews = false` — all off, avoids `Application Opened` / `$screen` noise +- Super properties registered once at startup via `AnalyticsContext.superProperties` + +### User identification + +`InstallIdentifier.current` — a random UUID generated on first launch, persisted in `UserDefaults`, cleared when analytics is toggled off. Same ID used for both SDKs so cross-referencing works. + +### Credentials flow + +`Config/Secrets.env` (gitignored) → Makefile passes to `xcodebuild` as build settings → substituted into `Info.plist` `$(VAR)` placeholders → read at startup via `Bundle.main.infoDictionary`. Template at `Config/Secrets.env.template`. + +## CLI tools + +Two separate binaries, same `~/.sentryclirc`: + +| Binary | Homebrew | Purpose | +|---|---|---| +| `sentry` (new) | `getsentry/tools/sentry` | Day-to-day inspection: `sentry issue view `, `sentry issue list`, `sentry issue events ` | +| `sentry-cli` (classic) | `getsentry/tools/sentry-cli` | dSYM uploads + release registration, invoked by `release.sh`. No useful `issue view` subcommand. | + +PostHog: dashboard only, no CLI needed. + +### Auth setup + +```ini +# ~/.sentryclirc +[auth] +token=sntryu_xxxxxxxx + +[defaults] +url=https://sentry.io/ +org=onevs-den +project=prowl-macos +``` + +Token scopes: `event:read`, `project:read`, `project:releases`. Generate at . + +## Diagnostic playbooks + +### Memory explosion + +*"User says Prowl was at 500 MB, now it's at 15 GB after running overnight."* + +1. **Sentry**: `sentry issue list --query "memory_threshold"` → look for `memory_threshold_4096mb` or `_8192mb` events. `sentry issue view ` gives you breadcrumbs (last 50 TCA actions) and device/OS context. The breadcrumb trail right before the spike often hints at what the user was doing. +2. **PostHog**: filter on `memory_threshold_2048mb` events (the earliest signal). Critical queries: + - **Correlation with `terminal_tab_count`** — if tab count scales with resident_mb, the leak is surface/ghostty driven + - **Correlation with `repository_count`** — if repo count scales, repo state / GitHub integration driven + - **Counters flat but memory grows** — leak is inside a fixed-count structure (scrollback, caches, Observable state). This is the hard case: Sentry Profiling can't see it (doesn't run between transactions), you'll need to reproduce locally with Instruments. +3. **PostHog**: `app_memory_baseline` histogram — the distribution of normal-use working sets. If median is already 1.5 GB, the leak starts earlier than the 2 GB threshold catches. +4. **Individual user suspect**: if you have an `install_id` (from Sentry `user.id` tag or PostHog person), go to PostHog Persons → find them → their event timeline shows exactly what they did from launch to explosion. + +### Suspicious Sentry issue — is it real? + +*"New issue showed up, but I have no idea if it's our bug or macOS noise."* + +1. `sentry issue view ` +2. **Check stack trace**: any `[app]` frames? If yes → real app issue, dig in. +3. **All frames are system (AppKit/CGS/mach_msg)?** Compare against `SentryEventFilter.systemHangSignatures` — if none of those substrings appear, it's a novel system pattern. Decide: ignore, or add to the filter. +4. **To add a new filter pattern**: append to `SentryEventFilter.systemHangSignatures` array. No test changes needed for a simple addition. + +### DAU / engagement check + +*"Did the tip channel release tank activation?"* + +1. PostHog Insights → filter by `environment = "tip"` (Sentry equivalent) or by super property `app_version = "prowl@X.Y.Z"` +2. DAU proxy: count unique `distinctId` with ≥ 1 event +3. Session lengths: `app_quit.session_duration_seconds` distribution (dropped events that leak past quit are rare on macOS since we capture at confirm-quit, but keep in mind) +4. Cold-start count: `app_activated` (since PR #208, fires once per cold launch only — not on window focus changes) + +## Event catalog + +### Super properties (every event carries these) + +Set once via `AnalyticsContext.superProperties` at `PostHogSDK.shared.register(...)`. + +| Key | Example | Source | +|---|---|---| +| `app_version` | `2026.4.17` | `CFBundleShortVersionString` | +| `build_number` | `20260417` | `CFBundleVersion` | +| `os_version` | `26.3.1` | `ProcessInfo.operatingSystemVersion` | +| `os_major` / `os_minor` | `26` / `3` | split of above for easy aggregation | +| `device_model` | `Mac14,9` | `sysctlbyname("hw.model")` | +| `cpu_arch` | `arm64` / `x86_64` | `#if arch(...)` | +| `locale` | `en_JP` | `Locale.current.identifier` | + +### Events + +Everything is hand-instrumented via `analyticsClient.capture(...)`. Events marked with ✱ carry additional properties beyond the super properties. + +| Event | Extra properties | Fires when | +|---|---|---| +| `app_activated` ✱ | — | cold launch (`.appLaunched` action) | +| `app_quit` ✱ | `session_duration_seconds` | user confirms quit | +| `worktree_opened` ✱ | `action` (editor/finder/terminal/IDE) | open worktree flow | +| `repository_added` ✱ | `count` | N repos added at once | +| `app_memory_baseline` ✱ | `resident_mb`, `uptime_seconds`, counters† | once, at 3 min uptime | +| `memory_threshold_2048mb` ✱ | `resident_mb`, `baseline_mb`, `growth_ratio`, counters† | first crossing of 2 GB | +| `memory_threshold_4096mb` ✱ | same + also Sentry `capture(message:)` | first crossing of 4 GB | +| `memory_threshold_8192mb` ✱ | same + Sentry capture | first crossing of 8 GB | +| `terminal_tab_created` | — | new tab in worktree | +| `terminal_tab_closed` | — | tab closed | +| `script_run` | — | user-triggered script | +| `worktree_created` / `_deleted` | — | lifecycle | +| `worktree_pinned` / `_unpinned` | — | ordering | +| `branch_renamed` | — | | +| `github_pr_opened` / `github_ci_check_opened` | — | click-through to GitHub | +| `settings_changed` | — | only if analytics still enabled after change | +| `update_checked` | — | manual Check for Updates | +| `repository_removed` | — | | + +† `counters` = `{ repository_count, opened_worktree_count, terminal_tab_count }` assembled by the `contextProvider` closure in `supacodeApp.swift` from the TCA store + `WorktreeTerminalManager`. + +Thresholds are **monotonic per session**: crossing 2 GB → dropping to 1 GB → crossing 2 GB again fires `memory_threshold_2048mb` exactly once. We want the session's envelope, not event storms. + +## Known noise and filters + +### System-induced App Hangs + +Wake-from-sleep, Mission Control space switch, external display (dis)connect all trigger `_NSMenuBarDisplayManagerActiveSpaceChanged` → NSWindow replicant rebuild → `mach_msg` IPC to WindowServer. On a busy main thread this can block > 3s. **Not an app bug.** + +Filtered in `supacode/Support/SentryEventFilter.swift`. Current signatures: + +- `_NSMenuBarDisplayManagerActiveSpaceChanged` +- `NSMenuBarLocalDisplayWindow` +- `NSMenuBarPresentationInstance` +- `NSMenuBarReplicantWindow` + +Filter is conservative: drops only when **(a)** `mechanism.type == "AppHang"` **and** **(b)** zero in-app frames **and** **(c)** at least one frame matches. Any novel pattern still gets through. + +### PostHog autocapture + +Disabled via `captureApplicationLifecycleEvents = false` + `captureScreenViews = false` + `enableSwizzling = false`. Rationale: `Application Opened` / `Backgrounded` fire on every Cmd+Tab and would burn the 1M/mo free tier at ~100 DAU. + +### Things that look like bugs but aren't + +- `sentry org list` returns 403 — missing `org:read` scope. Not needed for our flows (org is in `~/.sentryclirc`). Ignore. +- `sentry auth status` says "Could not verify credentials" but `sentry issue view` works fine — same 403 reason. Token is actually valid. +- `Library: posthog-ios` shown in PostHog for macOS events — the Swift SDK ships as `posthog-ios` for both platforms. Use `device_model` + `cpu_arch` super properties to discriminate. + +## File map + +| Concern | File | +|---|---| +| SDK init + `beforeSend` wiring | `supacode/App/supacodeApp.swift > bootstrapTelemetry` | +| PostHog event wrapper | `supacode/Clients/Analytics/AnalyticsClient.swift` | +| Super properties | `supacode/Support/AnalyticsContext.swift` | +| User identity | `supacode/Support/InstallIdentifier.swift` | +| Memory probe (`phys_footprint`) | `supacode/Support/MemoryProbe.swift` | +| Memory watchdog (baseline + thresholds) | `supacode/Support/MemoryWatchdog.swift` | +| System hang filter | `supacode/Support/SentryEventFilter.swift` | +| Release pipeline (dSYM upload + release tracking) | `doc-onevcat/scripts/release.sh` | +| Credentials template | `Config/Secrets.env.template` | + +## Quick reference: making changes + +- **Add a new PostHog event**: call `analyticsClient.capture("event_name", ["key": value])` in the relevant reducer. Test coverage mandatory per CLAUDE.md. +- **Add a new super property**: one-line addition to `AnalyticsContext.superProperties`. Applies to every event automatically. +- **Raise / lower a memory threshold**: edit `thresholdsMB` default in `MemoryWatchdog.init`. Update `MemoryWatchdogTests` assertions to match. +- **Filter a new Sentry noise pattern**: append to `SentryEventFilter.systemHangSignatures`. One line. +- **Switch Sentry environment logic**: `bootstrapTelemetry` maps `updateChannel` → `environment`. Currently `.tip → "tip"`, else `"production"`. +- **Reset analytics locally**: toggle off in Settings → toggle on. `AnalyticsClient.reset` clears both the PostHog identity and the install UUID. + +## Open gaps (possible future work) + +- `terminal_tab_closed.tab_lifetime_seconds` — needs `TerminalClient.Event` extension (`.tabClosed(lifetime)`) and moving capture from action-trigger to event-receipt. +- `script_run.exit_code` / `duration_ms` — ghostty doesn't expose shell exit status via its public API. Skipped. +- Ghostty surface byte counter (scrollback size) — currently `private var surfaces` on `WorktreeTerminalState`. If the memory-growth signal points to scrollback, this is the next handle to expose. +- Sentry Profiling — intentionally off (`profilesSampleRate` unset). Profiling only samples during transactions, which for a long-lived desktop app means 8-hour memory drift is invisible to it. Revisit if we have a concrete transaction-level question. +- User feedback inline — GitHub Issues for now; if volume grows, consider Sentry User Feedback API. + +## Related PRs + +- [#207](https://github.com/onevcat/Prowl/pull/207) Credentials pipeline (`Secrets.env` → `Info.plist` → `Bundle`) +- [#208](https://github.com/onevcat/Prowl/pull/208) Event quality — super properties, Sentry environment, `session_duration_seconds`, autocapture off, `app_activated` moved to cold start +- [#210](https://github.com/onevcat/Prowl/pull/210) dSYM upload + Sentry release tracking in `release.sh` +- [#211](https://github.com/onevcat/Prowl/pull/211) System hang filter + 3s `appHangTimeoutInterval` +- [#212](https://github.com/onevcat/Prowl/pull/212) Memory watchdog (baseline + threshold crossings + Sentry escalation at 4GB) -- 2.51.2