diff --git a/IMPLEMENTATION.md b/IMPLEMENTATION.md deleted file mode 100644 index f53432b..0000000 --- a/IMPLEMENTATION.md +++ /dev/null @@ -1,349 +0,0 @@ -# wicket — Design Document - -This document captures the design decisions for wicket, a webhook-to-SSE -gateway. It's written for a future implementation session. - -## HTTP API - -Two verbs, same path: - -``` -POST / → publish an event -GET / Accept: text/event-stream → subscribe (SSE stream) -``` - -The path is an arbitrary topic name. No registration, no schema. The -path exists as soon as someone POSTs or subscribes to it. - -A GET without `Accept: text/event-stream` returns 404 (or a simple -status page at `/`). - -## Hierarchical topics - -Path separators (`/`) are topic hierarchy levels. Publishing to a path -delivers to subscribers at every prefix level: - -``` -POST /github.com/chrisguidry/docketeer -``` - -delivers to subscribers on: -- `/github.com/chrisguidry/docketeer` (exact match) -- `/github.com/chrisguidry/` (parent prefix) -- `/github.com/` (grandparent) -- `/` (root — everything) - -This is a fan-out from leaf to root, not a glob or pattern match. - -## Event envelope - -Every event delivered to subscribers is wrapped: - -```json -{ - "id": "unique-event-id", - "timestamp": "2026-03-04T12:00:00Z", - "path": "github.com/chrisguidry/docketeer", - "headers": {"X-GitHub-Event": "push", "Content-Type": "application/json"}, - "payload": { ... raw webhook body ... } -} -``` - -SSE format: - -``` -id: unique-event-id -data: {"id":"...","timestamp":"...","path":"...","headers":{...},"payload":{...}} -``` - -The `id` field is a UUID (or ULID — implementer's choice). `headers` is -the subset of request headers from the POST (skip hop-by-hop headers). -`payload` is the raw body, JSON-parsed if Content-Type is JSON, otherwise -base64-encoded. - -## Auth model: open by default, configuration for secrets - -wicket works with zero configuration. A YAML configuration file layers in -authentication only where needed: - -```yaml -paths: - github.com/chrisguidry/docketeer: - # Inbound: verify webhook signatures - verify: hmac-sha256 - secret: "the-github-webhook-secret" - signature_header: X-Hub-Signature-256 - - # Outbound: require Bearer token to subscribe - subscribe_secret: "token-for-docketeer-subscribers" - - github.com/chrisguidry: - # Broader subscribe secret for all repos under this prefix - subscribe_secret: "token-for-all-chris-repos" -``` - -Behavior matrix: - -| Scenario | Result | -|---|---| -| POST to unconfigured path | Accepted, forwarded to subscribers | -| POST to configured path, valid signature | Accepted | -| POST to configured path, invalid/missing signature | 403 Forbidden | -| GET SSE on path with no subscribe_secret | Open | -| GET SSE on path with subscribe_secret | Requires `Authorization: Bearer ` | - -Configuration lookup walks up the path hierarchy. A path inherits the -`subscribe_secret` of its nearest configured ancestor. Verification -(`verify`/`secret`/`signature_header`) only applies to exact path matches -— it doesn't inherit, because different webhook sources have different -secrets. - -Configuration is hot-reloaded via fsnotify file watching or SIGHUP signal. -The server reads configuration on every request through an -`atomic.Pointer` or `sync.RWMutex` — no restart needed. - -There is no REST API for configuration. The file IS the configuration. - -## CORS - -wicket sends CORS headers on all responses so browser apps can publish -and subscribe. `Access-Control-Allow-Origin: *`, -`Access-Control-Allow-Methods: GET, POST, OPTIONS`, -`Access-Control-Allow-Headers: Content-Type`. OPTIONS preflight requests -get a 204. - -Note: the browser's native `EventSource` API doesn't support custom -headers, so there's no way to send `Authorization: Bearer` from a -browser SSE connection. This means browser apps can only subscribe to -paths that don't have a `subscribe_secret`. That's fine — protected -paths are for server-to-server use; browsers subscribe to public topics. - -## SSE reconnection - -The SSE spec supports `Last-Event-ID`. When a subscriber reconnects with -this header, wicket replays any events it missed. - -Storage: a single global ring buffer (configurable size, default 1000 -events). Each event in the buffer has its path, so on replay we filter to -only events matching the subscriber's path (respecting hierarchy). A -per-path ring buffer would waste memory for sparse topics. - -Events older than the buffer are gone. This is real-time delivery, not a -durable queue. - -## Query filters - -Optional query params for filtering on the subscribe side: - -``` -GET /github.com/chrisguidry/docketeer?filter=payload.ref:refs/heads/main -``` - -Parsing: split on `:` to get `field.path` and `value`. The field path -uses dot notation to navigate the JSON envelope. Multiple `filter` params -are AND'd. - -No operators, no regex, no full query language. Just `field.path:value` -string equality. This covers the common case (filter GitHub pushes to a -specific branch) without building a query engine. - -## Project structure - -``` -wicket/ - go.mod - main.go # CLI entry point, flag parsing, signal handling - server.go # HTTP server, route dispatch (POST vs SSE GET) - configuration.go # YAML configuration loading + hot reload (fsnotify) - broker.go # Topic tree, fan-out to subscribers, ring buffer - verify.go # Signature verification (HMAC-SHA256, extensible) - filter.go # Query filter parsing and matching - server_test.go # Integration tests - broker_test.go # Unit tests for topic matching and fan-out - verify_test.go # Unit tests for signature verification - filter_test.go # Unit tests for filter matching - configuration_test.go # Unit tests for configuration loading - README.md - DESIGN.md - Dockerfile - wicket.yaml.example -``` - -Single package (`main`). No internal packages, no pkg directory. This is -a small, focused binary. - -## Component details - -### main.go - -Parse flags: `-configuration` (path to YAML, optional), `-address` (listen -address, default `:8080`), `-buffer-size` (ring buffer size, default 1000). - -Load configuration if provided. Create broker. Start HTTP server. Handle -signals: SIGHUP reloads configuration, SIGTERM/SIGINT triggers graceful shutdown (close -listeners, drain SSE connections). - -### server.go - -Single `http.Handler` implementation. Route based on method + Accept header: - -- `POST` → read body, look up path configuration, verify signature if configured, - publish to broker, return 202 Accepted (or 403 if verification fails) -- `GET` with `Accept: text/event-stream` → check subscribe auth if - configured, subscribe via broker, write SSE stream with flushing, - handle client disconnect via `request.Context()` -- `GET` without SSE accept → 404 - -SSE streaming: set `Content-Type: text/event-stream`, `Cache-Control: no-cache`, -`Connection: keep-alive`. Flush after each event. Use `http.Flusher`. - -Return 202 (not 200) for POSTs to signal "accepted for delivery" since -delivery is async. - -### broker.go - -The core component. A topic tree where each node holds a list of -subscriber channels. - -```go -type Broker struct { - mu sync.RWMutex - subscribers map[string][]chan *Event // path → subscriber channels - buffer *RingBuffer -} -``` - -Key methods: -- `Publish(path string, event *Event)`: Add to ring buffer. Walk up the - path hierarchy (split on `/`), delivering to subscribers at each level. -- `Subscribe(path string, lastEventID string) (<-chan *Event, func())`: - Register a channel at the path. If `lastEventID` is set, replay from - buffer. Return the channel and an unsubscribe function. - -The ring buffer is a fixed-size circular array with a monotonic index for -efficient `Last-Event-ID` lookup. - -### configuration.go - -```go -type Configuration struct { - Paths map[string]PathConfiguration `yaml:"paths"` -} - -type PathConfiguration struct { - Verify string `yaml:"verify"` - Secret string `yaml:"secret"` - SignatureHeader string `yaml:"signature_header"` - SubscribeSecret string `yaml:"subscribe_secret"` -} -``` - -`LoadConfiguration(path string) (*Configuration, error)` reads and parses the file. -`WatchConfiguration(path string, callback func(*Configuration))` uses fsnotify to -detect changes and calls the callback with the new configuration. - -The server holds the configuration in an `atomic.Pointer[Configuration]` for lock-free -reads on every request. - -### verify.go - -Interface-based for extensibility: - -```go -type Verifier interface { - Verify(body []byte, headers http.Header, secret string) error -} -``` - -Implementations: -- `hmacSHA256Verifier`: reads signature from configured header, computes - `HMAC-SHA256(secret, body)`, compares with `hmac.Equal` -- `hmacSHA1Verifier`: same pattern for SHA1 (older GitHub webhooks) - -Factory function: `NewVerifier(method string) (Verifier, error)` returns -the right implementation based on the configuration string. - -### filter.go - -```go -type Filter struct { - Path string // dot-separated path into the event JSON - Value string // expected string value -} -``` - -`ParseFilters(query url.Values) []Filter` extracts `filter` params. -`MatchAll(filters []Filter, event *Event) bool` checks all filters -against the event envelope (navigating nested JSON via the dot path). - -## Dockerfile - -```dockerfile -FROM golang:1.24 AS build -WORKDIR /src -COPY go.mod go.sum ./ -RUN go mod download -COPY . . -RUN CGO_ENABLED=0 go build -o /wicket . - -FROM scratch -COPY --from=build /wicket /wicket -ENTRYPOINT ["/wicket"] -``` - -Two-stage build, scratch base, static binary. - -## Test strategy - -### Unit tests - -**broker_test.go**: -- Publish to exact path, subscriber receives event -- Publish to child path, parent subscriber receives event -- Publish to child path, unrelated subscriber does NOT receive -- Multiple subscribers on same path all receive -- Unsubscribe removes subscriber, no more events -- Ring buffer wraps correctly at capacity -- `Last-Event-ID` replay returns correct events -- `Last-Event-ID` replay respects path hierarchy -- `Last-Event-ID` for an event that fell off the buffer returns only new events - -**verify_test.go**: -- HMAC-SHA256 with known test vectors (GitHub's documented examples) -- Missing signature header → error -- Wrong signature → error -- Unknown verify method → error - -**filter_test.go**: -- Single filter matches -- Single filter doesn't match -- Multiple filters AND'd — all match -- Multiple filters AND'd — one doesn't match -- Nested dot path navigation -- Missing field in payload → no match (not an error) -- Empty filter list → matches everything - -**configuration_test.go**: -- Load valid configuration -- Load configuration with missing file → error -- Empty configuration (no paths) is valid -- Path lookup walks hierarchy for subscribe_secret -- Path lookup does NOT inherit verify/secret - -### Integration tests - -**server_test.go**: -- Start server, POST event, SSE subscriber receives it -- POST with valid HMAC signature → 202 -- POST with invalid signature → 403 -- POST to unconfigured path → 202 (no verification) -- SSE subscribe with valid Bearer token → connected -- SSE subscribe with wrong token → 401 -- SSE subscribe to open path → connected -- Subscribe to prefix, POST to child, subscriber receives event -- `Last-Event-ID` reconnection replays missed events -- Filter query param filters events correctly -- Graceful shutdown closes SSE connections - -All tests use `httptest.NewServer` for the integration tests. No external -dependencies, no test containers, everything runs in-process.