Observability Runbook #

Living document of entry 020. Migrated from doc-onevcat/observability.md on 2026-07-12; update in place. Known drift (2026-07-12): the App-Hang tracking sections below describe machinery removed in #236/#241 (SentryEventFilter is deleted); pending an update pass — see 001-action.md.

Quick-start reference for investigating production issues using the Sentry + PostHog pipelines built into Prowl. When you (or an agent) come back to diagnose something, read this first — it's the map.

TL;DR #

• Crashes / errors / App Hangs → Sentry (sentry issue view <id> or https://onevs-den.sentry.io/)
• User behavior / product questions / memory trends → PostHog (dashboard)
• Long-running memory growth → PostHog events app_memory_baseline and memory_threshold_<N>mb (see Memory explosion playbook)
• Credentials + scopes → ~/.sentryclirc (event:read, project:read, project:releases)
• Debug builds send nothing — #if !DEBUG gates every SDK call

Stack overview #

Two independent pipelines, both initialized in App/Sources/App/ProwlApp.swift > bootstrapTelemetry:

Sentry (crash + error + App Hang) #

• options.releaseName = "prowl@<VERSION>" — matches the release name that release.sh registers via sentry-cli releases new
• options.environment = "tip" for tip-channel users, "production" otherwise
• options.tracesSampleRate = 0.05 — 5% of transactions get performance traces
• options.enableAppHangTracking = true + appHangTimeoutInterval = 3 — main-thread stalls ≥ 3s get reported
• options.beforeSend = SentryEventFilter.filterSystemHang — client-side filter for known system-induced hangs
• Watchdog Termination not enabled (unsupported on native macOS)

PostHog (product analytics) #

• 16 hand-instrumented events — no autocapture
• enableSwizzling = false, captureApplicationLifecycleEvents = false, captureScreenViews = false — all off, avoids Application Opened / $screen noise
• Super properties registered once at startup via AnalyticsContext.superProperties

User identification #

InstallIdentifier.current — a random UUID generated on first launch, persisted in UserDefaults, cleared when analytics is toggled off. Same ID used for both SDKs so cross-referencing works.

Credentials flow #

Config/Secrets.env (gitignored) → Makefile passes to xcodebuild as build settings → substituted into Info.plist $(VAR) placeholders → read at startup via Bundle.main.infoDictionary. Template at Config/Secrets.env.template.

CLI tools #

Two separate binaries, same ~/.sentryclirc:

Binary	Homebrew	Purpose
sentry (new)	getsentry/tools/sentry	Day-to-day inspection: sentry issue view <id>, sentry issue list, sentry issue events <id>
sentry-cli (classic)	getsentry/tools/sentry-cli	dSYM uploads + release registration, invoked by release.sh. No useful issue view subcommand.

PostHog: dashboard only, no CLI needed.

Auth setup #

# ~/.sentryclirc
[auth]
token=sntryu_xxxxxxxx

[defaults]
url=https://sentry.io/
org=onevs-den
project=prowl-macos

Token scopes: event:read, project:read, project:releases. Generate at https://sentry.io/settings/account/api/auth-tokens/.

Diagnostic playbooks #

Memory explosion #

"User says Prowl was at 500 MB, now it's at 15 GB after running overnight."

1. Sentry: sentry issue list --query "memory_threshold" → look for memory_threshold_4096mb or _8192mb events. sentry issue view <id> gives you breadcrumbs (last 50 TCA actions) and device/OS context. The breadcrumb trail right before the spike often hints at what the user was doing.
2. PostHog: filter on memory_threshold_2048mb events (the earliest signal). Critical queries:
   • Correlation with terminal_tab_count — if tab count scales with resident_mb, the leak is surface/ghostty driven
   • Correlation with repository_count — if repo count scales, repo state / GitHub integration driven
   • Counters flat but memory grows — leak is inside a fixed-count structure (scrollback, caches, Observable state). This is the hard case: Sentry Profiling can't see it (doesn't run between transactions), you'll need to reproduce locally with Instruments.
3. PostHog: app_memory_baseline histogram — the distribution of normal-use working sets. If median is already 1.5 GB, the leak starts earlier than the 2 GB threshold catches.
4. Individual user suspect: if you have an install_id (from Sentry user.id tag or PostHog person), go to PostHog Persons → find them → their event timeline shows exactly what they did from launch to explosion.

Suspicious Sentry issue — is it real? #

"New issue showed up, but I have no idea if it's our bug or macOS noise."

1. sentry issue view <id>
2. Check stack trace: any [app] frames? If yes → real app issue, dig in.
3. All frames are system (AppKit/CGS/mach_msg)? Compare against SentryEventFilter.systemHangSignatures — if none of those substrings appear, it's a novel system pattern. Decide: ignore, or add to the filter.
4. To add a new filter pattern: append to SentryEventFilter.systemHangSignatures array. No test changes needed for a simple addition.

DAU / engagement check #

"Did the tip channel release tank activation?"

1. PostHog Insights → filter by environment = "tip" (Sentry equivalent) or by super property app_version = "prowl@X.Y.Z"
2. DAU proxy: count unique distinctId with ≥ 1 event
3. Session lengths: app_quit.session_duration_seconds distribution (dropped events that leak past quit are rare on macOS since we capture at confirm-quit, but keep in mind)
4. Cold-start count: app_launched (since PR #208, fires once per cold launch only — not on window focus changes)

Event catalog #

Super properties (every event carries these) #

Set once via AnalyticsContext.superProperties at PostHogSDK.shared.register(...).

Key	Example	Source
app_version	2026.4.17	CFBundleShortVersionString
build_number	20260417	CFBundleVersion
os_version	26.3.1	ProcessInfo.operatingSystemVersion
os_major / os_minor	26 / 3	split of above for easy aggregation
device_model	Mac14,9	sysctlbyname("hw.model")
cpu_arch	arm64 / x86_64	#if arch(...)
locale	en_JP	Locale.current.identifier

Events #

Everything is hand-instrumented via analyticsClient.capture(...). Events marked with ✱ carry additional properties beyond the super properties.

Event	Extra properties	Fires when
app_launched ✱	—	cold launch (.appLaunched action)
app_quit ✱	session_duration_seconds	user confirms quit
worktree_opened ✱	action (editor/finder/terminal/IDE)	open worktree flow
repository_added ✱	count	N repos added at once
app_memory_baseline ✱	resident_mb, uptime_seconds, counters†	once, at 3 min uptime
memory_threshold_2048mb ✱	resident_mb, baseline_mb, growth_ratio, counters†	first crossing of 2 GB
memory_threshold_4096mb ✱	same + also Sentry capture(message:)	first crossing of 4 GB
memory_threshold_8192mb ✱	same + Sentry capture	first crossing of 8 GB
terminal_tab_created	—	new tab in worktree
terminal_tab_closed	—	tab closed
script_run	—	user-triggered script
worktree_created / _deleted	—	lifecycle
worktree_pinned / _unpinned	—	ordering
branch_renamed	—
github_pr_opened / github_ci_check_opened	—	click-through to GitHub
settings_changed	—	only if analytics still enabled after change
update_checked	—	manual Check for Updates
repository_removed	—

† counters = { repository_count, opened_worktree_count, terminal_tab_count } assembled by the contextProvider closure in ProwlApp.swift from the TCA store + WorktreeTerminalManager.

Thresholds are monotonic per session: crossing 2 GB → dropping to 1 GB → crossing 2 GB again fires memory_threshold_2048mb exactly once. We want the session's envelope, not event storms.

Known noise and filters #

System-induced App Hangs #

Wake-from-sleep, Mission Control space switch, external display (dis)connect all trigger _NSMenuBarDisplayManagerActiveSpaceChanged → NSWindow replicant rebuild → mach_msg IPC to WindowServer. On a busy main thread this can block > 3s. Not an app bug.

Filtered in App/Sources/Support/SentryEventFilter.swift. Current signatures:

• _NSMenuBarDisplayManagerActiveSpaceChanged
• NSMenuBarLocalDisplayWindow
• NSMenuBarPresentationInstance
• NSMenuBarReplicantWindow

Filter is conservative: drops only when (a) mechanism.type == "AppHang" and (b) zero in-app frames and (c) at least one frame matches. Any novel pattern still gets through.

PostHog autocapture #

Disabled via captureApplicationLifecycleEvents = false + captureScreenViews = false + enableSwizzling = false. Rationale: Application Opened / Backgrounded fire on every Cmd+Tab and would burn the 1M/mo free tier at ~100 DAU.

Things that look like bugs but aren't #

• sentry org list returns 403 — missing org:read scope. Not needed for our flows (org is in ~/.sentryclirc). Ignore.
• sentry auth status says "Could not verify credentials" but sentry issue view works fine — same 403 reason. Token is actually valid.
• Library: posthog-ios shown in PostHog for macOS events — the Swift SDK ships as posthog-ios for both platforms. Use device_model + cpu_arch super properties to discriminate.

File map #

Concern	File
SDK init + beforeSend wiring	App/Sources/App/ProwlApp.swift > bootstrapTelemetry
PostHog event wrapper	App/Sources/Clients/Analytics/AnalyticsClient.swift
Super properties	App/Sources/Support/AnalyticsContext.swift
User identity	App/Sources/Support/InstallIdentifier.swift
Memory probe (phys_footprint)	App/Sources/Support/MemoryProbe.swift
Memory watchdog (baseline + thresholds)	App/Sources/Support/MemoryWatchdog.swift
System hang filter	App/Sources/Support/SentryEventFilter.swift
Release pipeline (dSYM upload + release tracking)	scripts/release.sh
Credentials template	Config/Secrets.env.template

Quick reference: making changes #

• Add a new PostHog event: call analyticsClient.capture("event_name", ["key": value]) in the relevant reducer. Test coverage mandatory per CLAUDE.md.
• Add a new super property: one-line addition to AnalyticsContext.superProperties. Applies to every event automatically.
• Raise / lower a memory threshold: edit thresholdsMB default in MemoryWatchdog.init. Update MemoryWatchdogTests assertions to match.
• Filter a new Sentry noise pattern: append to SentryEventFilter.systemHangSignatures. One line.
• Sentry environment: bootstrapTelemetry always configures "production"; the retired update channel no longer affects telemetry.
• Reset analytics locally: toggle off in Settings → toggle on. AnalyticsClient.reset clears both the PostHog identity and the install UUID.

Open gaps (possible future work) #

• terminal_tab_closed.tab_lifetime_seconds — needs TerminalClient.Event extension (.tabClosed(lifetime)) and moving capture from action-trigger to event-receipt.
• script_run.exit_code / duration_ms — ghostty doesn't expose shell exit status via its public API. Skipped.
• Ghostty surface byte counter (scrollback size) — currently private var surfaces on WorktreeTerminalState. If the memory-growth signal points to scrollback, this is the next handle to expose.
• Sentry Profiling — intentionally off (profilesSampleRate unset). Profiling only samples during transactions, which for a long-lived desktop app means 8-hour memory drift is invisible to it. Revisit if we have a concrete transaction-level question.
• User feedback inline — GitHub Issues for now; if volume grows, consider Sentry User Feedback API.

Related PRs #

• #207 Credentials pipeline (Secrets.env → Info.plist → Bundle)
• #208 Event quality — super properties, Sentry environment, session_duration_seconds, autocapture off, app_launched (renamed from app_activated) moved to cold start
• #210 dSYM upload + Sentry release tracking in release.sh
• #211 System hang filter + 3s appHangTimeoutInterval
• #212 Memory watchdog (baseline + threshold crossings + Sentry escalation at 4GB)