From 87037e024f971b1b90b4355d26c8aacc2925e07a Mon Sep 17 00:00:00 2001 From: Bretton Date: Mon, 20 Oct 2025 17:02:13 -0700 Subject: [PATCH] docs: add community feeds and aggregator documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Comprehensive documentation for feed systems and future aggregator features: New Documentation: - COMMUNITY_FEEDS.md: Complete guide to feed architecture and implementation - aggregators/PRD_AGGREGATORS.md: Product spec for RSS/aggregator features - aggregators/PRD_KAGI_NEWS_RSS.md: Kagi News integration design Updated: - PRD_POSTS.md: Refined post creation flow and security model Feed Documentation Coverage: - Architecture overview (service โ†’ repo โ†’ postgres) - Sort algorithms (hot, new, top) - Query optimization and indexing strategy - Security considerations - API examples and usage Aggregator PRDs: - RSS feed generation per community - External content aggregation - Kagi News integration patterns - Federation considerations These docs provide context for current feed implementation and roadmap for future aggregator features. ๐Ÿค– Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude --- docs/COMMUNITY_FEEDS.md | 795 ++++++++++++++ docs/PRD_POSTS.md | 20 +- docs/aggregators/PRD_AGGREGATORS.md | 1412 +++++++++++++++++++++++++ docs/aggregators/PRD_KAGI_NEWS_RSS.md | 1294 ++++++++++++++++++++++ 4 files changed, 3511 insertions(+), 10 deletions(-) create mode 100644 docs/COMMUNITY_FEEDS.md create mode 100644 docs/aggregators/PRD_AGGREGATORS.md create mode 100644 docs/aggregators/PRD_KAGI_NEWS_RSS.md diff --git a/docs/COMMUNITY_FEEDS.md b/docs/COMMUNITY_FEEDS.md new file mode 100644 index 0000000..a7598d0 --- /dev/null +++ b/docs/COMMUNITY_FEEDS.md @@ -0,0 +1,795 @@ +# Community Feeds Implementation + +**Status:** โœ… Implemented (Alpha) +**PR:** #1 - Community Feed Discovery +**Date:** October 2025 + +--- + +## Problem Statement + +### What We're Solving + +Users need a way to **browse and discover posts** in communities. Before this implementation: + +โŒ **No way to see what's in a community** +- Users could create posts, but couldn't view them +- No community browsing experience +- No sorting or ranking algorithms +- No pagination for large feeds + +โŒ **Missing core forum functionality** +- Forums need "Hot", "Top", "New" sorting +- Users expect Reddit-style ranking +- Need to discover trending content +- Must handle thousands of posts per community + +### User Stories + +1. **As a user**, I want to browse /c/gaming and see the hottest posts +2. **As a user**, I want to see top posts from this week in /c/cooking +3. **As a user**, I want to see newest posts in /c/music +4. **As a moderator**, I want posts ranked by engagement to surface quality content + +--- + +## Solution: Hydrated Community Feeds + +### Architecture Decision + +We chose **hydrated feeds** over Bluesky's skeleton pattern for Alpha: + +``` +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ Client โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ GET /xrpc/social.coves.feed.getCommunity?community=gaming&sort=hot + โ–ผ +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ Feed Service โ”‚ โ† Validates request, resolves community DID +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ–ผ +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ Feed Repository โ”‚ โ† Single SQL query with JOINs +โ”‚ (PostgreSQL) โ”‚ Returns fully hydrated posts +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ–ผ + [Full PostViews with author, community, stats] +``` + +**Why hydrated instead of skeleton + hydration?** + +| Criterion | Hydrated (Our Choice) | Skeleton Pattern | +|-----------|----------------------|------------------| +| **Requests** | 1 | 2 (skeleton โ†’ hydrate) | +| **Latency** | Lower | Higher | +| **Complexity** | Simple | Complex | +| **Flexibility** | Fixed algorithms | Custom feed generators | +| **Right for Alpha?** | โœ… Yes | โŒ Overkill | +| **Future-proof?** | โœ… Can add later | N/A | + +**Decision:** Ship fast with hydrated feeds now, add skeleton pattern in Beta when users request custom algorithms. + +**Alpha Scope (YAGNI):** +- โœ… Basic community sorting (hot, top, new) +- โœ… Public feeds only (no authentication required) +- โŒ Viewer state (deferred to feed generator phase) +- โŒ Custom feed algorithms (deferred to Beta) + +This keeps Alpha simple and focused on core browsing functionality. + +--- + +## Implementation Details + +### 1. Sorting Algorithms + +#### **Hot (Reddit Algorithm)** + +Balances score and recency for discovery: + +```sql +ORDER BY (score / POWER(age_hours + 2, 1.5)) DESC +``` + +**How it works:** +- New posts with low scores can outrank old posts with high scores +- Decay factor (1.5) tuned for forum dynamics +- Posts "age out" naturally over time + +**Example:** +- Post A: 100 upvotes, 1 day old โ†’ Rank: 10.4 +- Post B: 10 upvotes, 1 hour old โ†’ Rank: 3.5 +- Post C: 50 upvotes, 12 hours old โ†’ Rank: 5.1 + +**Result:** Fresh content surfaces while respecting engagement + +#### **Top (Score-Based)** + +Pure engagement ranking with timeframe filtering: + +```sql +WHERE created_at > NOW() - INTERVAL '1 day' +ORDER BY score DESC +``` + +**Timeframes:** +- `hour` - Last 60 minutes +- `day` - Last 24 hours (default) +- `week` - Last 7 days +- `month` - Last 30 days +- `year` - Last 365 days +- `all` - All time + +#### **New (Chronological)** + +Latest first, simple and predictable: + +```sql +ORDER BY created_at DESC +``` + +### 2. Pagination + +**Keyset pagination** for stability: + +``` +Cursor format (base64): "score::created_at::uri" +Delimiter: :: (following Bluesky convention) +``` + +**Why keyset over offset?** +- โœ… No duplicates when new posts appear +- โœ… No skipped posts when posts are deleted +- โœ… Consistent performance at any page depth +- โœ… Works with all sort orders + +**Cursor formats by sort type:** +- `new`: `timestamp::uri` (e.g., `2025-10-20T12:00:00Z::at://...`) +- `top`/`hot`: `score::timestamp::uri` (e.g., `100::2025-10-20T12:00:00Z::at://...`) + +**Why `::` delimiter?** +- Doesn't appear in ISO timestamps (which contain single `:`) +- Doesn't appear in AT-URIs +- Bluesky convention for cursor pagination +- Prevents parsing ambiguity + +**Example cursor flow:** +``` +Page 1: No cursor + โ†’ Returns posts 1-25 + cursor="100::2025-10-20T12:00:00Z::at://..." + +Page 2: cursor from page 1 + โ†’ Returns posts 26-50 + cursor="85::2025-10-20T11:30:00Z::at://..." + +Page 3: cursor from page 2 + โ†’ Returns posts 51-75 + cursor (or null if end) +``` + +### 3. Data Model + +#### **FeedViewPost** (Wrapper) + +```go +type FeedViewPost struct { + Post *PostView // Full post with all metadata + Reason *FeedReason // Why in feed (pin, repost) - Beta + Reply *ReplyRef // Reply context - Beta +} +``` + +#### **PostView** (Hydrated Post) + +```go +type PostView struct { + URI string // at://did:plc:abc/social.coves.post.record/123 + CID string // Content ID + RKey string // Record key (TID) + Author *AuthorView // Author with handle, avatar, reputation + Community *CommunityRef // Community with name, avatar + Title *string // Post title + Text *string // Post content + TextFacets []interface{} // Rich text (bold, mentions, links) + Embed interface{} // Union: images/video/external/quote + CreatedAt time.Time // When posted + IndexedAt time.Time // When AppView indexed it + Stats *PostStats // Upvotes, downvotes, score, comments + // Viewer: Not included in Alpha (deferred to feed generator phase) +} +``` + +#### **SQL Query** (Single Query Performance) + +```sql +SELECT + p.uri, p.cid, p.rkey, + p.author_did, u.handle, u.display_name, u.avatar, -- Author + p.community_did, c.name, c.avatar, -- Community + p.title, p.content, p.content_facets, p.embed, -- Content + p.created_at, p.indexed_at, + p.upvote_count, p.downvote_count, p.score, p.comment_count +FROM posts p +INNER JOIN users u ON p.author_did = u.did +INNER JOIN communities c ON p.community_did = c.did +WHERE p.community_did = $1 + AND p.deleted_at IS NULL + AND (cursor_filter) +ORDER BY (hot_rank) DESC +LIMIT 25 +``` + +**Performance:** One query returns everything - no N+1, no second hydration call. + +--- + +## API Specification + +### Endpoint + +``` +GET /xrpc/social.coves.feed.getCommunity +``` + +### Request Parameters + +| Parameter | Type | Required | Default | Description | +|-----------|------|----------|---------|-------------| +| `community` | string | โœ… Yes | - | Community DID or handle | +| `sort` | string | โŒ No | `"hot"` | Sort order: `hot`, `top`, `new` | +| `timeframe` | string | โŒ No | `"day"` | For `top` sort: `hour`, `day`, `week`, `month`, `year`, `all` | +| `limit` | integer | โŒ No | `15` | Posts per page (max: 50) | +| `cursor` | string | โŒ No | - | Pagination cursor from previous response | + +### Response + +```json +{ + "feed": [ + { + "post": { + "uri": "at://did:plc:gaming123/social.coves.post.record/abc", + "cid": "bafyrei...", + "author": { + "did": "did:plc:alice", + "handle": "alice.bsky.social", + "displayName": "Alice", + "avatar": "https://cdn.bsky.app/avatar/..." + }, + "community": { + "did": "did:plc:gaming123", + "name": "gaming", + "avatar": "https://..." + }, + "title": "Just finished Elden Ring!", + "text": "What an incredible journey...", + "embed": { + "$type": "social.coves.embed.images#view", + "images": [ + {"fullsize": "https://...", "alt": "Final boss screenshot"} + ] + }, + "createdAt": "2025-10-20T12:00:00Z", + "indexedAt": "2025-10-20T12:00:05Z", + "stats": { + "upvotes": 42, + "downvotes": 3, + "score": 39, + "commentCount": 15 + } + } + } + // ... 24 more posts + ], + "cursor": "Mzk6MjAyNS0xMC0yMFQxMjowMDowMFo6YXQ6Ly8uLi4=" +} +``` + +### Example Requests + +#### Browse hot posts in /c/gaming +```bash +curl 'http://localhost:8081/xrpc/social.coves.feed.getCommunity?community=gaming&sort=hot&limit=25' +``` + +#### Top posts this week in /c/cooking +```bash +curl 'http://localhost:8081/xrpc/social.coves.feed.getCommunity?community=did:plc:cooking&sort=top&timeframe=week' +``` + +#### Page 2 of new posts +```bash +curl 'http://localhost:8081/xrpc/social.coves.feed.getCommunity?community=gaming&sort=new&cursor=Mzk6...' +``` + +--- + +## Error Handling + +### Error Responses + +| Error | Status | When | +|-------|--------|------| +| `CommunityNotFound` | 404 | Community doesn't exist | +| `InvalidRequest` | 400 | Invalid parameters | +| `InvalidCursor` | 400 | Malformed pagination cursor | +| `InternalServerError` | 500 | Database or system error | + +### Example Error + +```json +{ + "error": "CommunityNotFound", + "message": "Community not found" +} +``` + +--- + +## Code Structure + +### Package Organization + +``` +internal/ +โ”œโ”€โ”€ core/feeds/ # Business logic +โ”‚ โ”œโ”€โ”€ interfaces.go # Service & Repository contracts +โ”‚ โ”œโ”€โ”€ service.go # Validation, community resolution +โ”‚ โ”œโ”€โ”€ types.go # Request/Response models +โ”‚ โ””โ”€โ”€ errors.go # Error types +โ”œโ”€โ”€ db/postgres/ +โ”‚ โ””โ”€โ”€ feed_repo.go # SQL queries, sorting algorithms +โ””โ”€โ”€ api/ + โ”œโ”€โ”€ handlers/feed/ + โ”‚ โ”œโ”€โ”€ get_community.go # HTTP handler + โ”‚ โ””โ”€โ”€ errors.go # Error mapping + โ””โ”€โ”€ routes/ + โ””โ”€โ”€ feed.go # Route registration +``` + +### Service Layer Flow + +``` +1. HandleGetCommunity (HTTP handler) + โ†“ Parse query params + +2. FeedService.GetCommunityFeed + โ†“ Validate request (sort, limit, timeframe) + โ†“ Resolve community identifier (handle โ†’ DID) + +3. FeedRepository.GetCommunityFeed + โ†“ Build SQL query (ORDER BY based on sort) + โ†“ Apply timeframe filter (for top) + โ†“ Apply cursor pagination + โ†“ Execute single query with JOINs + โ†“ Scan rows into PostView structs + โ†“ Build pagination cursor from last post + +4. Return FeedResponse + โ†“ Array of FeedViewPost + โ†“ Cursor for next page (if more results) +``` + +--- + +## Testing Strategy + +### Unit Tests (Future) + +- [ ] Feed service validation logic +- [ ] Cursor encoding/decoding +- [ ] Sort clause generation +- [ ] Timeframe filtering + +### Integration Tests (Required) + +- [x] Test hot/top/new sorting with real posts +- [x] Test pagination (3 pages, verify no duplicates) +- [x] Test community resolution (handle โ†’ DID) +- [x] Test error cases (invalid community, bad cursor) +- [x] Test empty feed (new community) +- [x] Test limit validation (zero, negative, over max) + +### Integration Test Results + +**All tests passing โœ…** + +```bash +PASS: TestGetCommunityFeed_Hot (0.02s) +PASS: TestGetCommunityFeed_Top_WithTimeframe (0.02s) + PASS: Top_posts_from_last_day (0.00s) + PASS: Top_posts_from_all_time (0.00s) +PASS: TestGetCommunityFeed_New (0.02s) +PASS: TestGetCommunityFeed_Pagination (0.05s) +PASS: TestGetCommunityFeed_InvalidCommunity (0.01s) +PASS: TestGetCommunityFeed_InvalidCursor (0.01s) + PASS: Invalid_base64 (0.00s) + PASS: Malicious_SQL (0.00s) + PASS: Invalid_timestamp (0.00s) + PASS: Invalid_URI_format (0.00s) +PASS: TestGetCommunityFeed_EmptyFeed (0.01s) +PASS: TestGetCommunityFeed_LimitValidation (0.01s) + PASS: Reject_limit_over_50 (0.00s) + PASS: Handle_zero_limit_with_default (0.00s) + +Total: 8 test cases, 12 sub-tests +``` + +**Test Coverage:** +- โœ… Hot algorithm (score decay over time) +- โœ… Top algorithm (timeframe filtering: day, all-time) +- โœ… New algorithm (chronological ordering) +- โœ… Pagination (3 pages, no duplicates, cursor stability) +- โœ… Error handling (invalid community, malformed cursors) +- โœ… Security (cursor injection, SQL injection attempts) +- โœ… Edge cases (empty feeds, zero/negative limits) + +**Location:** `tests/integration/feed_test.go` + +--- + +## Performance Considerations + +### Database Indexes + +Required indexes for optimal performance: + +```sql +-- Hot sorting (uses score and created_at) +CREATE INDEX idx_posts_community_hot +ON posts(community_did, score DESC, created_at DESC) +WHERE deleted_at IS NULL; + +-- Top sorting (score only) +CREATE INDEX idx_posts_community_top +ON posts(community_did, score DESC, created_at DESC) +WHERE deleted_at IS NULL; + +-- New sorting (chronological) +CREATE INDEX idx_posts_community_new +ON posts(community_did, created_at DESC) +WHERE deleted_at IS NULL; +``` + +### Query Performance + +- **Single query** - No N+1 problems +- **JOINs** - users and communities (always small cardinality) +- **Pagination** - Keyset, no OFFSET scans +- **Filtering** - `deleted_at IS NULL` uses partial index + +**Expected performance:** +- 25 posts with full metadata: **< 50ms** +- 1000+ posts in community: **Still < 50ms** (keyset pagination) + +--- + +## Future Enhancements (Beta) + +### 1. Feed Generators (Skeleton Pattern) + +Allow users to create custom algorithms: + +``` +GET /xrpc/social.coves.feed.getSkeleton?feed=at://alice/feed/best-memes + โ†’ Returns: [uri1, uri2, uri3, ...] + +GET /xrpc/social.coves.post.get?uris=[...] + โ†’ Returns: [full posts] +``` + +**Use cases:** +- User-created feeds ("Best of the week") +- Algorithmic feeds ("Rising posts", "Controversial") +- Filtered feeds ("Gaming news only", "No memes") + +### 2. Viewer State (Feed Generator Phase) + +**Status:** Deferred - Not needed for Alpha's basic community sorting + +Include viewer's relationship with posts when implementing feed generators: + +```json +"viewer": { + "vote": "up", + "voteUri": "at://...", + "saved": true, + "savedUri": "at://...", + "tags": ["read-later", "favorite"] +} +``` + +**Implementation Plan:** +- Wire up OptionalAuth middleware to feed routes +- Extract viewer DID from auth context +- Query viewer state tables (votes, saves, blocks) +- Include in PostView response + +**Requires:** +- Votes table (user_did, post_uri, vote_type) +- Saved posts table +- Blocks table +- Tags table + +**Why deferred:** Alpha only needs raw community sorting (hot/new/top). Viewer-specific features like upvote highlighting and saved posts will be implemented when we build the feed generator skeleton. + +### 3. Post Type Filtering (Feed Generator Phase) + +**Status:** Deferred - Not needed for Alpha's basic community sorting + +Filter by embed type when implementing feed generators: + +``` +GET ...?postTypes=image,video + โ†’ Only image and video posts +``` + +**Implementation Plan:** +- Check `embed->>'$type'` in SQL WHERE clause +- Map to friendly types (text, image, video, link, quote) +- Support both single (`postType`) and array (`postTypes`) filtering + +**Why deferred:** Alpha displays all posts without filtering. Post type filtering will be useful in feed generators for specialized feeds (e.g., "images only"). + +### 4. Pinned Posts (Feed Generator Phase) + +Moderators pin important posts to top: + +```json +"reason": { + "$type": "social.coves.feed.defs#reasonPin", + "community": {"did": "...", "name": "gaming"} +} +``` + +### 5. Reply Context + +Show post's position in thread: + +```json +"reply": { + "root": {"uri": "at://...", "cid": "..."}, + "parent": {"uri": "at://...", "cid": "..."} +} +``` + +--- + +## Lexicon Updates + +### Updated: `social.coves.post.get` + +**Changes:** +1. โœ… Batch URIs: `uri` โ†’ `uris[]` (max 25) +2. โœ… Union embed: Matches Bluesky pattern exactly +3. โœ… Error handling: `notFoundPost`, `blockedPost` + +**Before:** +```json +{ + "parameters": { + "uri": "string" + }, + "output": { + "post": "#postView" + } +} +``` + +**After:** +```json +{ + "parameters": { + "uris": ["string"] // Array, max 25 + }, + "output": { + "posts": [ + "union": ["#postView", "#notFoundPost", "#blockedPost"] + ] + } +} +``` + +**Why?** +- Batch fetching for feed hydration (future) +- Handle missing/blocked posts gracefully +- Bluesky compatibility + +### Using: `social.coves.feed.getCommunity` + +Already exists, matches our implementation: + +```json +{ + "id": "social.coves.feed.getCommunity", + "parameters": { + "community": "at-identifier", + "sort": "hot|top|new", + "timeframe": "hour|day|week|month|year|all", + "limit": 1-50, + "cursor": "string" + }, + "output": { + "feed": ["#feedViewPost"], + "cursor": "string" + } +} +``` + +--- + +## Migration Path + +### Alpha โ†’ Beta: Adding Feed Generators + +**Good news:** No breaking changes needed! + +**Approach:** +1. Keep `getCommunity` for standard sorting +2. Add `getFeedSkeleton` for custom algorithms +3. Add `post.get` batch support (already lexicon-ready) +4. Users choose: fast hydrated OR flexible skeleton + +**Both coexist:** +``` +// Standard community browsing (most users) +GET /xrpc/social.coves.feed.getCommunity?community=gaming&sort=hot + โ†’ One request, hydrated posts + +// Custom feed (power users) +GET /xrpc/social.coves.feed.getSkeleton?feed=at://alice/feed/best-memes + โ†’ Returns URIs +GET /xrpc/social.coves.post.get?uris=[...] + โ†’ Hydrates posts +``` + +--- + +## Success Metrics + +### Alpha Launch + +- [ ] Users can browse communities +- [ ] Hot/top/new sorting works correctly +- [ ] Pagination stable across 3+ pages +- [ ] Performance < 100ms for 25 posts +- [ ] Handles 1000+ posts per community + +### Future KPIs + +- Feed load time (target: < 50ms) +- Cache hit rate (future: Redis cache) +- Custom feed adoption (Beta) +- User engagement (time in feed, clicks) + +--- + +## Dependencies + +### Required Services + +- โœ… PostgreSQL (AppView database) +- โœ… Posts indexed via Jetstream +- โœ… Users indexed via Jetstream +- โœ… Communities indexed via Jetstream + +### Optional (Future) + +- Redis (feed caching) +- Feed generator services (custom algorithms) + +--- + +## Security Considerations + +### Input Validation + +- โœ… Community identifier format (DID or handle) +- โœ… Sort parameter (enum: hot/top/new) +- โœ… Limit (1-50, default 15, explicit rejection over 50) +- โœ… Cursor (base64 decoding, format validation) +- โœ… **Cursor injection prevention:** + - Timestamp format validation (RFC3339Nano) + - URI format validation (must start with `at://`) + - Score numeric validation + - Part count validation (2 for new, 3 for top/hot) + +### SQL Injection Prevention + +- โœ… All queries use parameterized statements +- โœ… **Dynamic ORDER BY uses whitelist map** (defense-in-depth) + ```go + var sortClauses = map[string]string{ + "hot": `(p.score / POWER(...)) DESC, p.created_at DESC`, + "top": `p.score DESC, p.created_at DESC`, + "new": `p.created_at DESC, p.uri DESC`, + } + ``` +- โœ… **Timeframe filter uses hardcoded switch** (no user input in INTERVAL) +- โœ… No string concatenation in SQL + +### DoS Prevention + +- โœ… **Zero-limit pagination fix:** Guards against `limit=0` causing panic + - Service layer: Sets default limit if โ‰ค 0 + - Repository layer: Additional check before array slicing +- โœ… Limit validation: Explicit error for limits over 50 +- โœ… Cursor validation: Rejects malformed cursors early + +### Rate Limiting + +- โœ… Global rate limiter (100 req/min per IP) +- Future: Per-endpoint limits + +### Privacy + +- Alpha: All feeds public +- Beta: Respect community visibility (private/unlisted) +- Beta: Block lists (hide posts from blocked users) + +### Security Audit (PR Review) + +All critical and important issues from PR review have been addressed: + +**P0 - Critical (Fixed):** +1. โœ… Zero-limit DoS vulnerability +2. โœ… Cursor injection attacks +3. โœ… Validation by-value bug + +**Important (Fixed):** +4. โœ… ORDER BY SQL injection hardening +5. โœ… Silent error swallowing in JSON encoding +6. โœ… Limit validation (reject vs silent cap) + +**False Positives (Rejected):** +- โŒ Time filter SQL injection (safe by design) +- โŒ Nil pointer dereference (impossible condition) + +--- + +## Conclusion + +### What We Shipped + +โœ… **Complete community feed system (Alpha scope)** +- Hot/top/new sorting algorithms +- Cursor-based pagination +- Single-query performance +- Full post hydration (author, community, stats) +- Error handling +- Production-ready code +- **No viewer state** (YAGNI - deferred to feed generator phase) + +### Why It Matters + +**Before:** Users could create posts but not see them +**After:** Full community browsing experience + +**Impact:** +- ๐ŸŽฏ Core forum functionality +- ๐Ÿš€ Fast, scalable implementation +- ๐Ÿ”ฎ Future-proof architecture +- ๐Ÿค Bluesky-compatible patterns + +### Next Steps + +1. ~~**Write E2E tests**~~ โœ… Complete (8 test cases, all passing) +2. **Performance testing** (1000+ posts under load) +3. **Add to docs site** (API reference) +4. **Monitor in production** (query performance, cursor stability) +5. **PR #2:** Batch `getPosts` for feed generators (Beta) + +--- + +## References + +- [PRD: Posts](../PRD_POSTS.md) +- [Lexicon: getCommunity](../internal/atproto/lexicon/social/coves/feed/getCommunity.json) +- [Lexicon: post.get](../internal/atproto/lexicon/social/coves/post/get.json) +- [Bluesky Feed Pattern](https://github.com/bluesky-social/atproto/discussions/4245) +- [Reddit Hot Algorithm](https://medium.com/hacking-and-gonzo/how-reddit-ranking-algorithms-work-ef111e33d0d9) + +--- + +**Document Version:** 1.0 +**Last Updated:** October 20, 2025 +**Status:** โœ… Implemented, Ready for Testing diff --git a/docs/PRD_POSTS.md b/docs/PRD_POSTS.md index c4b5935..4f67e4c 100644 --- a/docs/PRD_POSTS.md +++ b/docs/PRD_POSTS.md @@ -17,7 +17,7 @@ - **at-identifier Support:** All 4 formats (DIDs, canonical, @-prefixed, scoped handles) ### โš ๏ธ DEFERRED TO BETA -- Content rules validation (text-only, image-only communities) +- Content rules validation (text-only, image-only communities) - Governence - Post read operations (get, list) - Post update/edit operations - Post deletion @@ -379,7 +379,7 @@ CREATE INDEX idx_votes_voter_subject ON votes(voter_did, subject_uri) WHERE dele - [x] **Community Integration:** โœ… Posts correctly reference communities via at-identifiers - [x] **at-identifier Support:** โœ… All 4 formats supported (DIDs, canonical, @-prefixed, scoped) - [ ] **Content Rules Validation:** โš ๏ธ DEFERRED TO BETA - Posts validated against community content rules -- [ ] **Vote System:** โš ๏ธ DEFERRED TO BETA - Upvote/downvote with community-level controls +- [ ] **Vote System:** - Upvote/downvote with community-level controls - [ ] **Moderator Permissions:** โš ๏ธ DEFERRED TO BETA - Community moderators can delete posts ### Testing Requirements @@ -392,17 +392,17 @@ CREATE INDEX idx_votes_voter_subject ON votes(voter_did, subject_uri) WHERE dele - [x] Handler security: All 4 at-identifier formats accepted โœ… - [x] Consumer security: Rejects posts from wrong repository โœ… - [x] Consumer security: Verifies community and author exist โœ… -- [ ] **Content rules validation:** Text-only community rejects image posts โš ๏ธ DEFERRED -- [ ] **Content rules validation:** Image community rejects posts without images โš ๏ธ DEFERRED -- [ ] **Content rules validation:** Post with too-short text rejected โš ๏ธ DEFERRED -- [ ] **Content rules validation:** Federated post rejected if `allowFederated: false` โš ๏ธ DEFERRED +- [ ] **Content rules validation:** Text-only community rejects image posts โš ๏ธ DEFERRED - Governence +- [ ] **Content rules validation:** Image community rejects posts without images โš ๏ธ DEFERRED - Governence +- [ ] **Content rules validation:** Post with too-short text rejected โš ๏ธ DEFERRED - Governence +- [ ] **Content rules validation:** Federated post rejected if `allowFederated: false` โš ๏ธ DEFERRED - Governence - [ ] Update post within 24 hours โ†’ Edit reflected โš ๏ธ DEFERRED - [ ] Delete post as author โ†’ Hidden from queries โš ๏ธ DEFERRED - [ ] Delete post as moderator โ†’ Hidden from queries โš ๏ธ DEFERRED -- [ ] Upvote post โ†’ Count increments โš ๏ธ DEFERRED -- [ ] Downvote post โ†’ Count increments (if enabled) โš ๏ธ DEFERRED -- [ ] Toggle vote โ†’ Counts update correctly โš ๏ธ DEFERRED -- [ ] Community with downvotes disabled โ†’ Downvote returns error โš ๏ธ DEFERRED +- [ ] Upvote post โ†’ Count increments +- [ ] Downvote post โ†’ Count increments (if enabled) +- [ ] Toggle vote โ†’ Counts update correctly โš ๏ธ DEFERRED - Governence +- [ ] Community with downvotes disabled โ†’ Downvote returns error โš ๏ธ DEFERRED - Governence --- diff --git a/docs/aggregators/PRD_AGGREGATORS.md b/docs/aggregators/PRD_AGGREGATORS.md new file mode 100644 index 0000000..1d42910 --- /dev/null +++ b/docs/aggregators/PRD_AGGREGATORS.md @@ -0,0 +1,1412 @@ +# Aggregators PRD: Automated Content Posting System + +**Status:** Planning / Design Phase +**Owner:** Platform Team +**Last Updated:** 2025-10-19 + +## Overview + +Coves Aggregators are autonomous services that automatically post content to communities. Each aggregator is identified by its own DID and operates as a specialized actor within the atProto ecosystem. This system enables communities to have automated content feeds (RSS, sports results, TV/movie discussion threads, Bluesky mirrors, etc.) while maintaining full community control over which aggregators can post and what content they can create. + +**Key Differentiator:** Unlike other platforms where users manually aggregate content, Coves communities can enable automated aggregators to handle routine posting tasks, creating a more dynamic and up-to-date community experience. + +## Architecture Principles + +### โœ… atProto-Compliant Design + +Aggregators follow established atProto patterns for autonomous services: + +**Pattern:** Feed Generators + Labelers Model +- Each aggregator has its own DID (like feed generators) +- Declaration record published in aggregator's repo (like `app.bsky.feed.generator`) +- DID document advertises service endpoint +- Service makes authenticated XRPC calls +- Communities explicitly authorize aggregators (like subscribing to labelers) + +**Key Design Decisions:** + +1. **Aggregators are Actors, Not a Separate System** + - Aggregators authenticate as themselves (their DID) + - Use existing `social.coves.post.create` endpoint + - Post record's `author` field = aggregator DID (server-populated) + - No separate posting API needed + +2. **Community Authorization Model** + - Communities create `social.coves.aggregator.authorization` records + - These records grant specific aggregators permission to post + - Authorizations include configuration (which RSS feeds, which users to mirror, etc.) + - Can be enabled/disabled at any time + +3. **Hybrid Hosting** + - Coves can host official aggregators (RSS, sports, media) + - Third parties can build and host their own aggregators + - SDK provided for easy aggregator development + - All aggregators use same authorization system + +--- + +## Architecture Overview + +``` +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ Aggregator Service (External) โ”‚ +โ”‚ DID: did:web:rss-bot.coves.social โ”‚ +โ”‚ โ”‚ +โ”‚ - Watches external data sources (RSS, APIs, etc.) โ”‚ +โ”‚ - Processes content (LLM deduplication, formatting) โ”‚ +โ”‚ - Queries which communities have authorized it โ”‚ +โ”‚ - Creates posts via social.coves.post.create โ”‚ +โ”‚ - Responds to config queries via XRPC โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ + โ”‚ 1. Authenticate as aggregator DID (JWT) + โ”‚ 2. Call social.coves.post.create + โ”‚ { + โ”‚ community: "did:plc:gaming123", + โ”‚ title: "...", + โ”‚ content: "...", + โ”‚ federatedFrom: { platform: "rss", ... } + โ”‚ } + โ–ผ +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ Coves AppView (social.coves.post.create Handler) โ”‚ +โ”‚ โ”‚ +โ”‚ 1. Extract DID from JWT (aggregator's DID) โ”‚ +โ”‚ 2. Check if DID is registered aggregator โ”‚ +โ”‚ 3. Validate authorization record exists & enabled โ”‚ +โ”‚ 4. Apply aggregator-specific rate limits โ”‚ +โ”‚ 5. Validate content against community rules โ”‚ +โ”‚ 6. Create post with author = aggregator DID โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ + โ”‚ Post record created: + โ”‚ { + โ”‚ $type: "social.coves.post.record", + โ”‚ author: "did:web:rss-bot.coves.social", + โ”‚ community: "did:plc:gaming123", + โ”‚ title: "Tech News Roundup", + โ”‚ content: "...", + โ”‚ federatedFrom: { + โ”‚ platform: "rss", + โ”‚ uri: "https://techcrunch.com/..." + โ”‚ } + โ”‚ } + โ–ผ +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ Jetstream โ†’ AppView Indexing โ”‚ +โ”‚ - Post indexed with aggregator attribution โ”‚ +โ”‚ - UI shows: "๐Ÿค– Posted by RSS Aggregator" โ”‚ +โ”‚ - Community feed includes automated posts โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +--- + +## Use Cases + +### 1. RSS News Aggregator +**Problem:** Multiple users posting the same breaking news from different sources +**Solution:** RSS aggregator with LLM deduplication +- Watches configured RSS feeds +- Uses LLM to identify duplicate stories from different outlets +- Creates single "megathread" with all sources linked +- Posts unbiased summary of event +- Automatically tags with relevant topics + +**Community Config:** +```json +{ + "aggregatorDid": "did:web:rss-bot.coves.social", + "enabled": true, + "config": { + "feeds": [ + "https://techcrunch.com/feed", + "https://arstechnica.com/feed" + ], + "topics": ["technology", "ai"], + "dedupeWindow": "6h", + "minSources": 2 + } +} +``` + +### 2. Bluesky Post Mirror +**Problem:** Want to surface specific Bluesky discussions in community +**Solution:** Bluesky mirror aggregator +- Monitors specific users or hashtags on Bluesky +- Creates posts in community when criteria met +- Preserves `originalAuthor` metadata +- Links back to original Bluesky thread + +**Community Config:** +```json +{ + "aggregatorDid": "did:web:bsky-mirror.coves.social", + "enabled": true, + "config": { + "mirrorUsers": [ + "alice.bsky.social", + "bob.bsky.social" + ], + "hashtags": ["covesalpha"], + "minLikes": 10 + } +} +``` + +### 3. Sports Results Aggregator +**Problem:** Need post-game threads created immediately after games end +**Solution:** Sports aggregator watching game APIs +- Monitors sports APIs for game completions +- Creates post-game thread with final score, stats +- Tags with team names and league +- Posts within minutes of game ending + +**Community Config:** +```json +{ + "aggregatorDid": "did:web:sports-bot.coves.social", + "enabled": true, + "config": { + "league": "NBA", + "teams": ["Lakers", "Warriors"], + "includeStats": true, + "autoPin": true + } +} +``` + +### 4. TV/Movie Discussion Aggregator +**Problem:** Want episode discussion threads created when shows air +**Solution:** Media aggregator tracking release schedules +- Uses TMDB/IMDB APIs for release dates +- Creates discussion threads when episodes/movies release +- Includes metadata (cast, synopsis, ratings) +- Automatically pins for premiere episodes + +**Community Config:** +```json +{ + "aggregatorDid": "did:web:media-bot.coves.social", + "enabled": true, + "config": { + "shows": [ + {"tmdbId": "1234", "name": "Breaking Bad"} + ], + "createOn": "airDate", + "timezone": "America/New_York", + "spoilerProtection": true + } +} +``` + +--- + +## Lexicon Schemas + +### 1. Aggregator Service Declaration + +**Collection:** `social.coves.aggregator.service` +**Key:** `literal:self` (one per aggregator account) +**Location:** Aggregator's own repository + +This record declares the existence of an aggregator service and provides metadata for discovery. + +```json +{ + "lexicon": 1, + "id": "social.coves.aggregator.service", + "defs": { + "main": { + "type": "record", + "description": "Declaration of an aggregator service that can post to communities", + "key": "literal:self", + "record": { + "type": "object", + "required": ["did", "displayName", "createdAt", "aggregatorType"], + "properties": { + "did": { + "type": "string", + "format": "did", + "description": "DID of the aggregator service (must match repo DID)" + }, + "displayName": { + "type": "string", + "maxGraphemes": 64, + "maxLength": 640, + "description": "Human-readable name (e.g., 'RSS News Aggregator')" + }, + "description": { + "type": "string", + "maxGraphemes": 300, + "maxLength": 3000, + "description": "Description of what this aggregator does" + }, + "avatar": { + "type": "blob", + "accept": ["image/png", "image/jpeg"], + "maxSize": 1000000, + "description": "Avatar image for bot identity" + }, + "aggregatorType": { + "type": "string", + "knownValues": [ + "social.coves.aggregator.types#rss", + "social.coves.aggregator.types#blueskyMirror", + "social.coves.aggregator.types#sports", + "social.coves.aggregator.types#media", + "social.coves.aggregator.types#custom" + ], + "description": "Type of aggregator for categorization" + }, + "configSchema": { + "type": "unknown", + "description": "JSON Schema describing config options for this aggregator. Communities use this to know what configuration fields are available." + }, + "sourceUrl": { + "type": "string", + "format": "uri", + "description": "URL to aggregator's source code (for transparency)" + }, + "maintainer": { + "type": "string", + "format": "did", + "description": "DID of person/organization maintaining this aggregator" + }, + "createdAt": { + "type": "string", + "format": "datetime" + } + } + } + } + } +} +``` + +**Example Record:** +```json +{ + "$type": "social.coves.aggregator.service", + "did": "did:web:rss-bot.coves.social", + "displayName": "RSS News Aggregator", + "description": "Automatically posts breaking news from configured RSS feeds with LLM-powered deduplication", + "aggregatorType": "social.coves.aggregator.types#rss", + "configSchema": { + "type": "object", + "properties": { + "feeds": { + "type": "array", + "items": { "type": "string", "format": "uri" } + }, + "topics": { + "type": "array", + "items": { "type": "string" } + }, + "dedupeWindow": { "type": "string" }, + "minSources": { "type": "integer", "minimum": 1 } + } + }, + "sourceUrl": "https://github.com/coves-social/rss-aggregator", + "maintainer": "did:plc:coves-platform", + "createdAt": "2025-10-19T12:00:00Z" +} +``` + +--- + +### 2. Community Authorization Record + +**Collection:** `social.coves.aggregator.authorization` +**Key:** `any` (one per aggregator per community) +**Location:** Community's repository + +This record grants an aggregator permission to post to a community and contains aggregator-specific configuration. + +```json +{ + "lexicon": 1, + "id": "social.coves.aggregator.authorization", + "defs": { + "main": { + "type": "record", + "description": "Authorization for an aggregator to post to a community with specific configuration", + "key": "any", + "record": { + "type": "object", + "required": ["aggregatorDid", "communityDid", "createdAt", "enabled"], + "properties": { + "aggregatorDid": { + "type": "string", + "format": "did", + "description": "DID of the authorized aggregator" + }, + "communityDid": { + "type": "string", + "format": "did", + "description": "DID of the community granting access (must match repo DID)" + }, + "enabled": { + "type": "boolean", + "description": "Whether this aggregator is currently active. Can be toggled without deleting the record." + }, + "config": { + "type": "unknown", + "description": "Aggregator-specific configuration. Must conform to the aggregator's configSchema." + }, + "createdAt": { + "type": "string", + "format": "datetime" + }, + "createdBy": { + "type": "string", + "format": "did", + "description": "DID of moderator who authorized this aggregator" + }, + "disabledAt": { + "type": "string", + "format": "datetime", + "description": "When this authorization was disabled (if enabled=false)" + }, + "disabledBy": { + "type": "string", + "format": "did", + "description": "DID of moderator who disabled this aggregator" + } + } + } + } + } +} +``` + +**Example Record:** +```json +{ + "$type": "social.coves.aggregator.authorization", + "aggregatorDid": "did:web:rss-bot.coves.social", + "communityDid": "did:plc:gaming123", + "enabled": true, + "config": { + "feeds": [ + "https://techcrunch.com/feed", + "https://arstechnica.com/feed" + ], + "topics": ["technology", "ai", "gaming"], + "dedupeWindow": "6h", + "minSources": 2 + }, + "createdAt": "2025-10-19T14:00:00Z", + "createdBy": "did:plc:alice123" +} +``` + +--- + +### 3. Aggregator Type Definitions + +**Collection:** `social.coves.aggregator.types` +**Purpose:** Define known aggregator types for categorization + +```json +{ + "lexicon": 1, + "id": "social.coves.aggregator.types", + "defs": { + "rss": { + "type": "string", + "description": "Aggregator that monitors RSS/Atom feeds" + }, + "blueskyMirror": { + "type": "string", + "description": "Aggregator that mirrors Bluesky posts" + }, + "sports": { + "type": "string", + "description": "Aggregator for sports scores and game threads" + }, + "media": { + "type": "string", + "description": "Aggregator for TV/movie discussion threads" + }, + "custom": { + "type": "string", + "description": "Custom third-party aggregator" + } + } +} +``` + +--- + +## XRPC Methods + +### For Communities (Moderators) + +#### `social.coves.aggregator.enable` +Enable an aggregator for a community + +**Input:** +```json +{ + "aggregatorDid": "did:web:rss-bot.coves.social", + "config": { + "feeds": ["https://techcrunch.com/feed"], + "topics": ["technology"] + } +} +``` + +**Output:** +```json +{ + "uri": "at://did:plc:gaming123/social.coves.aggregator.authorization/3jui7kd58dt2g", + "cid": "bafyreif5...", + "authorization": { + "aggregatorDid": "did:web:rss-bot.coves.social", + "communityDid": "did:plc:gaming123", + "enabled": true, + "config": {...}, + "createdAt": "2025-10-19T14:00:00Z" + } +} +``` + +**Behavior:** +- Validates caller is community moderator +- Validates aggregator exists and has service declaration +- Validates config against aggregator's configSchema +- Creates authorization record in community's repo +- Indexes to AppView for authorization checks + +**Errors:** +- `NotAuthorized` - Caller is not a moderator +- `AggregatorNotFound` - Aggregator DID doesn't exist +- `InvalidConfig` - Config doesn't match configSchema + +--- + +#### `social.coves.aggregator.disable` +Disable an aggregator for a community + +**Input:** +```json +{ + "aggregatorDid": "did:web:rss-bot.coves.social" +} +``` + +**Output:** +```json +{ + "uri": "at://did:plc:gaming123/social.coves.aggregator.authorization/3jui7kd58dt2g", + "disabled": true, + "disabledAt": "2025-10-19T15:00:00Z" +} +``` + +**Behavior:** +- Validates caller is community moderator +- Updates authorization record (sets `enabled=false`, `disabledAt`, `disabledBy`) +- Aggregator can no longer post until re-enabled + +--- + +#### `social.coves.aggregator.updateConfig` +Update configuration for an enabled aggregator + +**Input:** +```json +{ + "aggregatorDid": "did:web:rss-bot.coves.social", + "config": { + "feeds": ["https://techcrunch.com/feed", "https://arstechnica.com/feed"], + "topics": ["technology", "gaming"] + } +} +``` + +**Output:** +```json +{ + "uri": "at://did:plc:gaming123/social.coves.aggregator.authorization/3jui7kd58dt2g", + "cid": "bafyreif6...", + "config": {...} +} +``` + +--- + +#### `social.coves.aggregator.listForCommunity` +List all aggregators (enabled and disabled) for a community + +**Input:** +```json +{ + "community": "did:plc:gaming123", + "enabledOnly": false, + "limit": 50, + "cursor": "..." +} +``` + +**Output:** +```json +{ + "aggregators": [ + { + "aggregatorDid": "did:web:rss-bot.coves.social", + "displayName": "RSS News Aggregator", + "description": "...", + "aggregatorType": "social.coves.aggregator.types#rss", + "enabled": true, + "config": {...}, + "createdAt": "2025-10-19T14:00:00Z" + } + ], + "cursor": "..." +} +``` + +--- + +### For Aggregators + +#### Existing: `social.coves.post.create` +**Modified Behavior:** Now handles aggregator authentication + +**Authorization Flow:** +1. Extract DID from JWT +2. Check if DID is registered aggregator (query `aggregators` table) +3. If aggregator: + - Validate authorization record exists for this community + - Check `enabled=true` + - Apply aggregator rate limits (e.g., 10 posts/hour) +4. If regular user: + - Validate membership, bans, etc. (existing logic) +5. Create post with `author = actorDID` + +**Rate Limits:** +- Regular users: 20 posts/hour per community +- Aggregators: 10 posts/hour per community (to prevent spam) + +--- + +#### `social.coves.aggregator.getAuthorizations` +Get list of communities that have authorized this aggregator + +**Input:** +```json +{ + "aggregatorDid": "did:web:rss-bot.coves.social", + "enabledOnly": true, + "limit": 100, + "cursor": "..." +} +``` + +**Output:** +```json +{ + "authorizations": [ + { + "communityDid": "did:plc:gaming123", + "communityName": "Gaming News", + "enabled": true, + "config": {...}, + "createdAt": "2025-10-19T14:00:00Z" + } + ], + "cursor": "..." +} +``` + +**Use Case:** Aggregator queries this to know which communities to post to + +--- + +### For Discovery + +#### `social.coves.aggregator.list` +List all available aggregators + +**Input:** +```json +{ + "type": "social.coves.aggregator.types#rss", + "limit": 50, + "cursor": "..." +} +``` + +**Output:** +```json +{ + "aggregators": [ + { + "did": "did:web:rss-bot.coves.social", + "displayName": "RSS News Aggregator", + "description": "...", + "aggregatorType": "social.coves.aggregator.types#rss", + "avatar": "...", + "maintainer": "did:plc:coves-platform", + "sourceUrl": "https://github.com/coves-social/rss-aggregator" + } + ], + "cursor": "..." +} +``` + +--- + +#### `social.coves.aggregator.get` +Get detailed information about a specific aggregator + +**Input:** +```json +{ + "aggregatorDid": "did:web:rss-bot.coves.social" +} +``` + +**Output:** +```json +{ + "did": "did:web:rss-bot.coves.social", + "displayName": "RSS News Aggregator", + "description": "...", + "aggregatorType": "social.coves.aggregator.types#rss", + "configSchema": {...}, + "sourceUrl": "...", + "maintainer": "...", + "stats": { + "communitiesUsing": 42, + "postsCreated": 1337, + "createdAt": "2025-10-19T12:00:00Z" + } +} +``` + +--- + +## Database Schema + +### `aggregators` Table +Indexed aggregator service declarations from Jetstream + +```sql +CREATE TABLE aggregators ( + did TEXT PRIMARY KEY, + display_name TEXT NOT NULL, + description TEXT, + aggregator_type TEXT NOT NULL, + config_schema JSONB, + avatar_url TEXT, + source_url TEXT, + maintainer_did TEXT, + + -- Indexing metadata + record_uri TEXT NOT NULL, + record_cid TEXT NOT NULL, + indexed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + + -- Stats (cached) + communities_using INTEGER NOT NULL DEFAULT 0, + posts_created BIGINT NOT NULL DEFAULT 0, + + CONSTRAINT aggregators_type_check CHECK ( + aggregator_type IN ( + 'social.coves.aggregator.types#rss', + 'social.coves.aggregator.types#blueskyMirror', + 'social.coves.aggregator.types#sports', + 'social.coves.aggregator.types#media', + 'social.coves.aggregator.types#custom' + ) + ) +); + +CREATE INDEX idx_aggregators_type ON aggregators(aggregator_type); +CREATE INDEX idx_aggregators_indexed_at ON aggregators(indexed_at DESC); +``` + +--- + +### `aggregator_authorizations` Table +Indexed authorization records from communities + +```sql +CREATE TABLE aggregator_authorizations ( + id BIGSERIAL PRIMARY KEY, + + -- Authorization identity + aggregator_did TEXT NOT NULL REFERENCES aggregators(did) ON DELETE CASCADE, + community_did TEXT NOT NULL, + + -- Authorization state + enabled BOOLEAN NOT NULL DEFAULT true, + config JSONB, + + -- Audit trail + created_at TIMESTAMPTZ NOT NULL, + created_by TEXT NOT NULL, -- DID of moderator + disabled_at TIMESTAMPTZ, + disabled_by TEXT, -- DID of moderator + + -- atProto record metadata + record_uri TEXT NOT NULL UNIQUE, + record_cid TEXT NOT NULL, + indexed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + + UNIQUE(aggregator_did, community_did) +); + +CREATE INDEX idx_aggregator_auth_agg_did ON aggregator_authorizations(aggregator_did) WHERE enabled = true; +CREATE INDEX idx_aggregator_auth_comm_did ON aggregator_authorizations(community_did) WHERE enabled = true; +CREATE INDEX idx_aggregator_auth_enabled ON aggregator_authorizations(enabled); +``` + +--- + +### `aggregator_posts` Table +Track posts created by aggregators (for rate limiting and stats) + +```sql +CREATE TABLE aggregator_posts ( + id BIGSERIAL PRIMARY KEY, + + aggregator_did TEXT NOT NULL REFERENCES aggregators(did) ON DELETE CASCADE, + community_did TEXT NOT NULL, + post_uri TEXT NOT NULL, + post_cid TEXT NOT NULL, + + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + + UNIQUE(post_uri) +); + +CREATE INDEX idx_aggregator_posts_agg_did_created ON aggregator_posts(aggregator_did, created_at DESC); +CREATE INDEX idx_aggregator_posts_comm_did_created ON aggregator_posts(community_did, created_at DESC); + +-- For rate limiting: count posts in last hour +CREATE INDEX idx_aggregator_posts_rate_limit ON aggregator_posts(aggregator_did, community_did, created_at DESC); +``` + +--- + +## Implementation Plan + +### Phase 1: Core Infrastructure (Coves AppView) + +**Goal:** Enable aggregator authentication and authorization + +#### 1.1 Database Setup +- [ ] Create migration for `aggregators` table +- [ ] Create migration for `aggregator_authorizations` table +- [ ] Create migration for `aggregator_posts` table + +#### 1.2 Lexicon Definitions +- [ ] Create `social.coves.aggregator.service.json` +- [ ] Create `social.coves.aggregator.authorization.json` +- [ ] Create `social.coves.aggregator.types.json` +- [ ] Generate Go types from lexicons + +#### 1.3 Repository Layer +```go +// internal/core/aggregators/repository.go + +type Repository interface { + // Aggregator management + CreateAggregator(ctx context.Context, agg *Aggregator) error + GetAggregator(ctx context.Context, did string) (*Aggregator, error) + ListAggregators(ctx context.Context, filter AggregatorFilter) ([]*Aggregator, error) + UpdateAggregatorStats(ctx context.Context, did string, stats Stats) error + + // Authorization management + CreateAuthorization(ctx context.Context, auth *Authorization) error + GetAuthorization(ctx context.Context, aggDID, commDID string) (*Authorization, error) + ListAuthorizationsForAggregator(ctx context.Context, aggDID string, enabledOnly bool) ([]*Authorization, error) + ListAuthorizationsForCommunity(ctx context.Context, commDID string) ([]*Authorization, error) + UpdateAuthorization(ctx context.Context, auth *Authorization) error + IsAuthorized(ctx context.Context, aggDID, commDID string) (bool, error) + + // Post tracking (for rate limiting) + RecordAggregatorPost(ctx context.Context, aggDID, commDID, postURI string) error + CountRecentPosts(ctx context.Context, aggDID, commDID string, since time.Time) (int, error) +} +``` + +#### 1.4 Service Layer +```go +// internal/core/aggregators/service.go + +type Service interface { + // For communities (moderators) + EnableAggregator(ctx context.Context, commDID, aggDID string, config map[string]interface{}) (*Authorization, error) + DisableAggregator(ctx context.Context, commDID, aggDID string) error + UpdateAggregatorConfig(ctx context.Context, commDID, aggDID string, config map[string]interface{}) error + ListCommunityAggregators(ctx context.Context, commDID string, enabledOnly bool) ([]*AggregatorInfo, error) + + // For aggregators + GetAuthorizedCommunities(ctx context.Context, aggDID string) ([]*CommunityAuth, error) + + // For discovery + ListAggregators(ctx context.Context, filter AggregatorFilter) ([]*Aggregator, error) + GetAggregator(ctx context.Context, did string) (*AggregatorDetail, error) + + // Internal: called by post creation handler + ValidateAggregatorPost(ctx context.Context, aggDID, commDID string) error +} +``` + +#### 1.5 Modify Post Creation Handler +```go +// internal/api/handlers/post/create.go + +func CreatePost(ctx context.Context, input *CreatePostInput) (*CreatePostOutput, error) { + actorDID := GetDIDFromAuth(ctx) + + // Check if actor is an aggregator + if isAggregator, _ := aggregatorService.IsAggregator(ctx, actorDID); isAggregator { + // Validate aggregator authorization + if err := aggregatorService.ValidateAggregatorPost(ctx, actorDID, input.Community); err != nil { + return nil, err + } + + // Apply aggregator rate limits + if err := rateLimitAggregator(ctx, actorDID, input.Community); err != nil { + return nil, ErrRateLimitExceeded + } + } else { + // Regular user validation (existing logic) + // ... membership checks, ban checks, etc. + } + + // Create post (author will be actorDID - either user or aggregator) + post, err := postService.CreatePost(ctx, actorDID, input) + if err != nil { + return nil, err + } + + // If aggregator, track the post + if isAggregator { + _ = aggregatorService.RecordPost(ctx, actorDID, input.Community, post.URI) + } + + return post, nil +} +``` + +#### 1.6 XRPC Handlers +- [ ] `social.coves.aggregator.enable` handler +- [ ] `social.coves.aggregator.disable` handler +- [ ] `social.coves.aggregator.updateConfig` handler +- [ ] `social.coves.aggregator.listForCommunity` handler +- [ ] `social.coves.aggregator.getAuthorizations` handler +- [ ] `social.coves.aggregator.list` handler +- [ ] `social.coves.aggregator.get` handler + +#### 1.7 Jetstream Consumer +```go +// internal/atproto/jetstream/aggregator_consumer.go + +func (c *AggregatorConsumer) HandleEvent(ctx context.Context, evt *jetstream.Event) error { + switch evt.Collection { + case "social.coves.aggregator.service": + switch evt.Operation { + case "create", "update": + return c.indexAggregatorService(ctx, evt) + case "delete": + return c.deleteAggregator(ctx, evt.DID) + } + + case "social.coves.aggregator.authorization": + switch evt.Operation { + case "create", "update": + return c.indexAuthorization(ctx, evt) + case "delete": + return c.deleteAuthorization(ctx, evt.URI) + } + } + return nil +} +``` + +#### 1.8 Integration Tests +- [ ] Test aggregator service indexing from Jetstream +- [ ] Test authorization record indexing +- [ ] Test `social.coves.post.create` with aggregator auth +- [ ] Test authorization validation (enabled/disabled) +- [ ] Test rate limiting for aggregators +- [ ] Test config validation against schema + +**Milestone:** Aggregators can authenticate and post to communities with authorization + +--- + +### Phase 2: Aggregator SDK (Go) + +**Goal:** Provide SDK for building aggregators easily + +#### 2.1 SDK Core +```go +// github.com/coves-social/aggregator-sdk-go + +package aggregator + +type Aggregator interface { + // Identity + GetDID() string + GetDisplayName() string + GetDescription() string + GetType() string + GetConfigSchema() map[string]interface{} + + // Lifecycle + Start(ctx context.Context) error + Stop() error + + // Posting (provided by SDK) + CreatePost(ctx context.Context, communityDID string, post Post) error + GetAuthorizedCommunities(ctx context.Context) ([]*CommunityAuth, error) +} + +type BaseAggregator struct { + DID string + DisplayName string + Description string + Type string + PrivateKey crypto.PrivateKey + CovesAPIURL string + + client *http.Client +} + +type Post struct { + Title string + Content string + Embed interface{} + FederatedFrom *FederatedSource + ContentLabels []string +} + +type FederatedSource struct { + Platform string // "rss", "bluesky", etc. + URI string + ID string + OriginalCreatedAt time.Time +} + +// Helper methods provided by SDK +func (a *BaseAggregator) CreatePost(ctx context.Context, communityDID string, post Post) error { + // 1. Sign JWT with aggregator's private key + token := a.signJWT() + + // 2. Call social.coves.post.create via XRPC + resp, err := a.client.Post( + a.CovesAPIURL + "/xrpc/social.coves.post.create", + &CreatePostInput{ + Community: communityDID, + Title: post.Title, + Content: post.Content, + Embed: post.Embed, + FederatedFrom: post.FederatedFrom, + ContentLabels: post.ContentLabels, + }, + &CreatePostOutput{}, + WithAuth(token), + ) + + return err +} + +func (a *BaseAggregator) GetAuthorizedCommunities(ctx context.Context) ([]*CommunityAuth, error) { + // Call social.coves.aggregator.getAuthorizations + token := a.signJWT() + + resp, err := a.client.Get( + a.CovesAPIURL + "/xrpc/social.coves.aggregator.getAuthorizations", + map[string]string{"aggregatorDid": a.DID, "enabledOnly": "true"}, + &GetAuthorizationsOutput{}, + WithAuth(token), + ) + + return resp.Authorizations, err +} +``` + +#### 2.2 SDK Documentation +- [ ] README with quickstart guide +- [ ] Example aggregators (RSS, Bluesky mirror) +- [ ] API reference documentation +- [ ] Configuration schema guide + +**Milestone:** Third parties can build aggregators using SDK + +--- + +### Phase 3: Reference Aggregator (RSS) + +**Goal:** Build working RSS aggregator as reference implementation + +#### 3.1 RSS Aggregator Implementation +```go +// github.com/coves-social/rss-aggregator + +package main + +import "github.com/coves-social/aggregator-sdk-go" + +type RSSAggregator struct { + *aggregator.BaseAggregator + + // RSS-specific config + pollInterval time.Duration + llmClient *openai.Client +} + +func (r *RSSAggregator) Start(ctx context.Context) error { + // 1. Get authorized communities + communities, err := r.GetAuthorizedCommunities(ctx) + if err != nil { + return err + } + + // 2. Start polling loop + ticker := time.NewTicker(r.pollInterval) + defer ticker.Stop() + + for { + select { + case <-ticker.C: + r.pollFeeds(ctx, communities) + case <-ctx.Done(): + return nil + } + } +} + +func (r *RSSAggregator) pollFeeds(ctx context.Context, communities []*CommunityAuth) { + for _, comm := range communities { + // Get RSS feeds from community config + feeds := comm.Config["feeds"].([]string) + + for _, feedURL := range feeds { + items, err := r.fetchFeed(feedURL) + if err != nil { + continue + } + + // Process new items + for _, item := range items { + // Check if already posted + if r.alreadyPosted(item.GUID) { + continue + } + + // LLM deduplication logic + duplicate := r.findDuplicate(item, comm.CommunityDID) + if duplicate != nil { + r.addToMegathread(duplicate, item) + continue + } + + // Create new post + post := aggregator.Post{ + Title: item.Title, + Content: r.summarize(item), + FederatedFrom: &aggregator.FederatedSource{ + Platform: "rss", + URI: item.Link, + OriginalCreatedAt: item.PublishedAt, + }, + } + + err = r.CreatePost(ctx, comm.CommunityDID, post) + if err != nil { + log.Printf("Failed to create post: %v", err) + continue + } + + r.markPosted(item.GUID) + } + } + } +} + +func (r *RSSAggregator) summarize(item *RSSItem) string { + // Use LLM to create unbiased summary + prompt := fmt.Sprintf("Summarize this news article in 2-3 sentences: %s", item.Description) + summary, _ := r.llmClient.Complete(prompt) + return summary +} + +func (r *RSSAggregator) findDuplicate(item *RSSItem, communityDID string) *Post { + // Use LLM to detect semantic duplicates + // Query recent posts in community + // Compare with embeddings/similarity + return nil // or duplicate post +} +``` + +#### 3.2 Deployment +- [ ] Dockerfile for RSS aggregator +- [ ] Kubernetes manifests (for Coves-hosted instance) +- [ ] Environment configuration guide +- [ ] Monitoring and logging setup + +#### 3.3 Testing +- [ ] Unit tests for feed parsing +- [ ] Integration tests with mock Coves API +- [ ] E2E test with real Coves instance +- [ ] LLM deduplication accuracy tests + +**Milestone:** RSS aggregator running in production for select communities + +--- + +### Phase 4: Additional Aggregators + +#### 4.1 Bluesky Mirror Aggregator +- [ ] Monitor Jetstream for specific users/hashtags +- [ ] Preserve `originalAuthor` metadata +- [ ] Link back to original Bluesky post +- [ ] Rate limiting (don't flood community) + +#### 4.2 Sports Aggregator +- [ ] Integrate with ESPN/TheSportsDB APIs +- [ ] Monitor game completions +- [ ] Create post-game threads with stats +- [ ] Auto-pin major games + +#### 4.3 Media (TV/Movie) Aggregator +- [ ] Integrate with TMDB API +- [ ] Track show release schedules +- [ ] Create episode discussion threads +- [ ] Spoiler protection tags + +**Milestone:** Multiple official aggregators available for communities + +--- + +## Security Considerations + +### Authentication +โœ… **DID-based Authentication** +- Aggregators sign JWTs with their private keys +- Server validates JWT signature against DID document +- No shared secrets or API keys + +โœ… **Scoped Authorization** +- Authorization records are per-community +- Aggregator can only post to authorized communities +- Communities can revoke at any time + +### Rate Limiting +โœ… **Per-Aggregator Limits** +- 10 posts/hour per community (configurable) +- Prevents aggregator spam +- Separate from user rate limits + +โœ… **Global Limits** +- Total posts across all communities: 100/hour +- Prevents runaway aggregators + +### Content Validation +โœ… **Community Rules** +- Aggregator posts validated against community content rules +- No special exemptions (same rules as users) +- Community can ban specific content patterns + +โœ… **Config Validation** +- Authorization config validated against aggregator's configSchema +- Prevents injection attacks via config +- JSON schema validation + +### Monitoring & Auditing +โœ… **Audit Trail** +- All aggregator posts logged +- `created_by` tracks which moderator authorized +- `disabled_by` tracks who revoked access +- Full history preserved + +โœ… **Abuse Detection** +- Monitor for spam patterns +- Alert if aggregator posts rejected repeatedly +- Auto-disable after threshold violations + +### Transparency +โœ… **Open Source** +- Official aggregators open source +- Source URL in service declaration +- Community can audit behavior + +โœ… **Attribution** +- Posts clearly show aggregator authorship +- UI shows "๐Ÿค– Posted by [Aggregator Name]" +- No attempt to impersonate users + +--- + +## UI/UX Considerations + +### Community Settings +**Aggregator Management Page:** +- List of available aggregators (with descriptions, types) +- "Enable" button opens config modal +- Config form generated from aggregator's configSchema +- Toggle to enable/disable without deleting config +- Stats: posts created, last active + +**Post Display:** +- Posts from aggregators have bot badge: "๐Ÿค–" +- Shows aggregator name (e.g., "Posted by RSS News Bot") +- `federatedFrom` shows original source +- Link to original content (RSS article, Bluesky post, etc.) + +### User Preferences +- Option to hide all aggregator posts +- Option to hide specific aggregators +- Filter posts by "user-created only" or "include bots" + +--- + +## Success Metrics + +### Pre-Launch Checklist +- [ ] Lexicons defined and validated +- [ ] Database migrations tested +- [ ] Jetstream consumer indexes aggregator records +- [ ] Post creation handler validates aggregator auth +- [ ] Rate limiting prevents spam +- [ ] SDK published and documented +- [ ] Reference RSS aggregator working +- [ ] E2E tests passing +- [ ] Security audit completed + +### Alpha Goals +- 3+ official aggregators (RSS, Bluesky mirror, sports) +- 10+ communities using aggregators +- < 0.1% spam posts (false positives) +- Aggregator posts appear in feed within 1 minute + +### Beta Goals +- Third-party aggregators launched +- 50+ communities using aggregators +- Developer documentation complete +- Marketplace/directory for discovery + +--- + +## Out of Scope (Future Versions) + +### Aggregator Marketplace +- [ ] Community ratings/reviews for aggregators +- [ ] Featured aggregators +- [ ] Paid aggregators (premium features) +- [ ] Aggregator analytics dashboard + +### Advanced Features +- [ ] Scheduled posts (post at specific time) +- [ ] Content moderation integration (auto-label NSFW) +- [ ] Multi-community posting (single post to multiple communities) +- [ ] Interactive aggregators (respond to comments) +- [ ] Aggregator-to-aggregator communication (chains) + +### Federation +- [ ] Cross-instance aggregator discovery +- [ ] Aggregator migration (change hosting provider) +- [ ] Federated aggregator authorization (trust other instances' aggregators) + +--- + +## Technical Decisions Log + +### 2025-10-19: Reuse `social.coves.post.create` Endpoint + +**Decision:** Aggregators use existing post creation endpoint, not a separate `social.coves.aggregator.post.create` + +**Rationale:** +- Post record already server-populates `author` field from JWT +- Aggregators authenticate as themselves โ†’ `author = aggregator DID` +- Simpler: one code path for all post creation +- Follows atProto principle: actors are actors (users, bots, aggregators) +- `federatedFrom` field already handles external content attribution + +**Implementation:** +- Add authorization check to `social.coves.post.create` handler +- Check if authenticated DID is aggregator +- Validate authorization record exists and enabled +- Apply aggregator-specific rate limits +- Otherwise same logic as user posts + +**Trade-offs Accepted:** +- Post creation handler has branching logic (user vs aggregator) +- But: keeps lexicon simple, reuses existing validation + +--- + +### 2025-10-19: Hybrid Hosting Model + +**Decision:** Support both Coves-hosted and third-party aggregators + +**Rationale:** +- Coves can provide high-quality official aggregators (RSS, sports, media) +- Third parties can build specialized aggregators (niche communities) +- SDK makes it easy to build custom aggregators +- Follows feed generator model (anyone can run one) +- Decentralization-friendly + +**Requirements:** +- SDK must be well-documented and maintained +- Authorization system must be DID-agnostic (works for any DID) +- Discovery system shows all aggregators (official + third-party) + +--- + +### 2025-10-19: Config as JSON Schema + +**Decision:** Aggregators declare configSchema in their service record + +**Rationale:** +- Communities need to know what config options are available +- JSON Schema is standard, well-supported +- Enables UI auto-generation (forms from schema) +- Validation at authorization creation time +- Flexible: each aggregator can have different config structure + +**Example:** +```json +{ + "configSchema": { + "type": "object", + "properties": { + "feeds": { + "type": "array", + "items": { "type": "string", "format": "uri" }, + "description": "RSS feed URLs to monitor" + }, + "topics": { + "type": "array", + "items": { "type": "string" }, + "description": "Topics to filter posts by" + } + }, + "required": ["feeds"] + } +} +``` + +**Trade-offs Accepted:** +- More complex than simple key-value config +- But: better UX (self-documenting), prevents errors + +--- + +## References + +- atProto Lexicon Spec: https://atproto.com/specs/lexicon +- Feed Generator Starter Kit: https://github.com/bluesky-social/feed-generator +- Labeler Implementation: https://github.com/bluesky-social/atproto/tree/main/packages/ozone +- JSON Schema Spec: https://json-schema.org/ +- Coves Communities PRD: [PRD_COMMUNITIES.md](PRD_COMMUNITIES.md) +- Coves Posts Implementation: [IMPLEMENTATION_POST_CREATION.md](IMPLEMENTATION_POST_CREATION.md) diff --git a/docs/aggregators/PRD_KAGI_NEWS_RSS.md b/docs/aggregators/PRD_KAGI_NEWS_RSS.md new file mode 100644 index 0000000..0a4f2e5 --- /dev/null +++ b/docs/aggregators/PRD_KAGI_NEWS_RSS.md @@ -0,0 +1,1294 @@ +# Kagi News RSS Aggregator PRD + +**Status:** Planning Phase +**Owner:** Platform Team +**Last Updated:** 2025-10-20 +**Parent PRD:** [PRD_AGGREGATORS.md](PRD_AGGREGATORS.md) + +## Overview + +The Kagi News RSS Aggregator is a reference implementation of the Coves aggregator system that automatically posts high-quality, multi-source news summaries to communities. It leverages Kagi News's free RSS feeds to provide pre-aggregated, deduped news content with multiple perspectives and source citations. + +**Key Value Propositions:** +- **Multi-source aggregation**: Kagi News already aggregates multiple sources per story +- **Balanced perspectives**: Built-in perspective tracking from different outlets +- **Rich metadata**: Categories, highlights, source links included +- **Legal & free**: CC BY-NC licensed for non-commercial use +- **Low complexity**: No LLM deduplication needed (Kagi does it) + +## Data Source: Kagi News RSS Feeds + +### Licensing & Legal + +**License:** CC BY-NC (Creative Commons Attribution-NonCommercial) + +**Terms:** +- โœ… **Free for non-commercial use** (Coves qualifies) +- โœ… **Attribution required** (must credit Kagi News) +- โŒ **Cannot use commercially** (must contact support@kagi.com for commercial license) +- โœ… **Data can be shared** (with same attribution + NC requirements) + +**Source:** https://news.kagi.com/about + +**Quote from Kagi:** +> Note that kite.json and files referenced by it are licensed under CC BY-NC license. This means that this data can be used free of charge (with attribution and for non-commercial use). If you would like to license this data for commercial use let us know through support@kagi.com. + +**Compliance Requirements:** +- Visible attribution to Kagi News on every post +- Link back to original Kagi story page +- Non-commercial operation (met: Coves is non-commercial) + +--- + +### RSS Feed Structure + +**Base URL Pattern:** `https://news.kagi.com/{category}.xml` + +**Known Categories:** +- `world.xml` - World news +- `tech.xml` - Technology (likely) +- `business.xml` - Business (likely) +- `sports.xml` - Sports (likely) +- Additional categories TBD (need to scrape homepage) + +**Feed Format:** RSS 2.0 (standard XML) + +**Update Frequency:** One daily update (~noon UTC) + +--- + +### RSS Item Schema + +Each `` in the feed contains: + +```xml + + Story headline + https://kite.kagi.com/{uuid}/{category}/{id} + Full HTML content (see below) + https://kite.kagi.com/{uuid}/{category}/{id} + Primary category (e.g., "World") + Subcategory (e.g., "World/Conflict & Security") + Tag (e.g., "Conflict & Security") + Mon, 20 Oct 2025 01:46:31 +0000 + +``` + +**Description HTML Structure:** +```html +

Main summary paragraph with inline source citations [source1.com#1][source2.com#1]

+ +Image caption + +

Highlights:

+
    +
  • Key point 1 with [source.com#1] citations
  • +
  • Key point 2...
  • +
+ +
Notable quote - Person Name
+ +

Perspectives:

+
    +
  • Viewpoint holder: Their perspective. (Source)
  • +
+ +

Sources:

+ +``` + +**Key Features:** +- Multiple source citations inline +- Balanced perspectives from different actors +- Highlights extract key points +- Direct quotes preserved +- All sources linked with attribution + +--- + +## Architecture + +### High-Level Flow + +``` +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ Kagi News RSS Feeds (External) โ”‚ +โ”‚ - https://news.kagi.com/world.xml โ”‚ +โ”‚ - https://news.kagi.com/tech.xml โ”‚ +โ”‚ - etc. โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ + โ”‚ HTTP GET one job after update + โ–ผ +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ Kagi News Aggregator Service โ”‚ +โ”‚ DID: did:web:kagi-news.coves.social โ”‚ +โ”‚ โ”‚ +โ”‚ Components: โ”‚ +โ”‚ 1. Feed Poller: Fetches RSS feeds on schedule โ”‚ +โ”‚ 2. Item Parser: Extracts structured data from HTML โ”‚ +โ”‚ 3. Deduplication: Tracks posted GUIDs (no LLM needed) โ”‚ +โ”‚ 4. Category Mapper: Maps Kagi categories to communities โ”‚ +โ”‚ 5. Post Formatter: Converts to Coves post format โ”‚ +โ”‚ 6. Post Publisher: Calls social.coves.post.create โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ + โ”‚ Authenticated XRPC calls + โ–ผ +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ Coves AppView (social.coves.post.create) โ”‚ +โ”‚ - Validates aggregator authorization โ”‚ +โ”‚ - Creates post with author = did:web:kagi-news.coves.socialโ”‚ +โ”‚ - Indexes to community feeds โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +--- + +### Aggregator Service Declaration + +```json +{ + "$type": "social.coves.aggregator.service", + "did": "did:web:kagi-news.coves.social", + "displayName": "Kagi News Aggregator", + "description": "Automatically posts breaking news from Kagi News RSS feeds. Kagi News aggregates multiple sources per story with balanced perspectives and comprehensive source citations.", + "aggregatorType": "social.coves.aggregator.types#rss", + "avatar": "", + "configSchema": { + "type": "object", + "properties": { + "categories": { + "type": "array", + "items": { + "type": "string", + "enum": ["world", "tech", "business", "sports", "science"] + }, + "description": "Kagi News categories to monitor", + "minItems": 1 + }, + "subcategoryFilter": { + "type": "array", + "items": { "type": "string" }, + "description": "Optional: only post stories with these subcategories (e.g., 'World/Middle East', 'Tech/AI')" + }, + "minSources": { + "type": "integer", + "minimum": 1, + "default": 2, + "description": "Minimum number of sources required for a story to be posted" + }, + "includeImages": { + "type": "boolean", + "default": true, + "description": "Include images from Kagi proxy in posts" + }, + "postFormat": { + "type": "string", + "enum": ["full", "summary", "minimal"], + "default": "full", + "description": "How much content to include: full (all sections), summary (main paragraph + sources), minimal (title + link only)" + } + }, + "required": ["categories"] + }, + "sourceUrl": "https://github.com/coves-social/kagi-news-aggregator", + "maintainer": "did:plc:coves-platform", + "createdAt": "2025-10-20T12:00:00Z" +} +``` + +--- + +## Community Configuration Examples + +### Example 1: World News Community + +```json +{ + "aggregatorDid": "did:web:kagi-news.coves.social", + "enabled": true, + "config": { + "categories": ["world"], + "minSources": 3, + "includeImages": true, + "postFormat": "full" + } +} +``` + +**Result:** Posts all world news stories with 3+ sources, full content including images/highlights/perspectives. + +--- + +### Example 2: AI/Tech Community (Filtered) + +```json +{ + "aggregatorDid": "did:web:kagi-news.coves.social", + "enabled": true, + "config": { + "categories": ["tech", "business"], + "subcategoryFilter": ["Tech/AI", "Tech/Machine Learning", "Business/Tech Industry"], + "minSources": 2, + "includeImages": true, + "postFormat": "full" + } +} +``` + +**Result:** Only posts tech stories about AI/ML or tech industry business news with 2+ sources. + +--- + +### Example 3: Breaking News (Minimal) + +```json +{ + "aggregatorDid": "did:web:kagi-news.coves.social", + "enabled": true, + "config": { + "categories": ["world", "business", "tech"], + "minSources": 5, + "includeImages": false, + "postFormat": "minimal" + } +} +``` + +**Result:** Only major stories (5+ sources), minimal format (headline + link), no images. + +--- + +## Post Format Specification + +### Post Record Structure + +```json +{ + "$type": "social.coves.post.record", + "author": "did:web:kagi-news.coves.social", + "community": "did:plc:worldnews123", + "title": "{Kagi story title}", + "content": "{formatted content based on postFormat config}", + "embed": { + "$type": "app.bsky.embed.external", + "external": { + "uri": "https://kite.kagi.com/{uuid}/{category}/{id}", + "title": "{story title}", + "description": "{summary excerpt}", + "thumb": "{image blob if includeImages=true}" + } + }, + "federatedFrom": { + "platform": "kagi-news-rss", + "uri": "https://kite.kagi.com/{uuid}/{category}/{id}", + "id": "{guid}", + "originalCreatedAt": "{pubDate from RSS}" + }, + "contentLabels": [ + "{primary category}", + "{subcategories}" + ], + "createdAt": "{current timestamp}" +} +``` + +--- + +### Content Formatting by `postFormat` + +#### Format: `full` (Default) + +```markdown +{Main summary paragraph with source citations} + +**Highlights:** +โ€ข {Bullet point 1} +โ€ข {Bullet point 2} +โ€ข ... + +**Perspectives:** +โ€ข **{Actor}**: {Their perspective} ([Source]({url})) +โ€ข ... + +> {Notable quote} โ€” {Attribution} + +**Sources:** +โ€ข [{Title}]({url}) - {domain} +โ€ข ... + +--- +๐Ÿ“ฐ Story aggregated by [Kagi News]({kagi_story_url}) +``` + +**Rationale:** Preserves Kagi's rich multi-source analysis, provides maximum value. + +--- + +#### Format: `summary` + +```markdown +{Main summary paragraph with source citations} + +**Sources:** +โ€ข [{Title}]({url}) - {domain} +โ€ข ... + +--- +๐Ÿ“ฐ Story aggregated by [Kagi News]({kagi_story_url}) +``` + +**Rationale:** Clean summary with source links, less overwhelming. + +--- + +#### Format: `minimal` + +```markdown +{Story title} + +Read more: {kagi_story_url} + +**Sources:** {domain1}, {domain2}, {domain3}... + +--- +๐Ÿ“ฐ Via [Kagi News]({kagi_story_url}) +``` + +**Rationale:** Just headlines with link, for high-volume communities or breaking news alerts. + +--- + +## Implementation Details + +### Component 1: Feed Poller + +**Responsibility:** Fetch RSS feeds on schedule + +```go +type FeedPoller struct { + categories []string + pollInterval time.Duration + httpClient *http.Client +} + +func (p *FeedPoller) Start(ctx context.Context) error { + ticker := time.NewTicker(p.pollInterval) // 15 minutes + defer ticker.Stop() + + for { + select { + case <-ticker.C: + for _, category := range p.categories { + feedURL := fmt.Sprintf("https://news.kagi.com/%s.xml", category) + feed, err := p.fetchFeed(feedURL) + if err != nil { + log.Printf("Failed to fetch %s: %v", feedURL, err) + continue + } + p.handleFeed(ctx, category, feed) + } + case <-ctx.Done(): + return nil + } + } +} + +func (p *FeedPoller) fetchFeed(url string) (*gofeed.Feed, error) { + parser := gofeed.NewParser() + feed, err := parser.ParseURL(url) + return feed, err +} +``` + +**Libraries:** +- `github.com/mmcdole/gofeed` - RSS/Atom parser + +--- + +### Component 2: Item Parser + +**Responsibility:** Extract structured data from RSS item HTML + +```go +type KagiStory struct { + Title string + Link string + GUID string + PubDate time.Time + Categories []string + + // Parsed from HTML description + Summary string + Highlights []string + Perspectives []Perspective + Quote *Quote + Sources []Source + ImageURL string + ImageAlt string +} + +type Perspective struct { + Actor string + Description string + SourceURL string +} + +type Quote struct { + Text string + Attribution string +} + +type Source struct { + Title string + URL string + Domain string +} + +func (p *ItemParser) Parse(item *gofeed.Item) (*KagiStory, error) { + doc, err := goquery.NewDocumentFromReader(strings.NewReader(item.Description)) + if err != nil { + return nil, err + } + + story := &KagiStory{ + Title: item.Title, + Link: item.Link, + GUID: item.GUID, + PubDate: *item.PublishedParsed, + Categories: item.Categories, + } + + // Extract summary (first

tag) + story.Summary = doc.Find("p").First().Text() + + // Extract highlights + doc.Find("h3:contains('Highlights')").Next("ul").Find("li").Each(func(i int, s *goquery.Selection) { + story.Highlights = append(story.Highlights, s.Text()) + }) + + // Extract perspectives + doc.Find("h3:contains('Perspectives')").Next("ul").Find("li").Each(func(i int, s *goquery.Selection) { + text := s.Text() + link := s.Find("a").First() + sourceURL, _ := link.Attr("href") + + // Parse format: "Actor: Description (Source)" + parts := strings.SplitN(text, ":", 2) + if len(parts) == 2 { + story.Perspectives = append(story.Perspectives, Perspective{ + Actor: strings.TrimSpace(parts[0]), + Description: strings.TrimSpace(parts[1]), + SourceURL: sourceURL, + }) + } + }) + + // Extract quote + doc.Find("blockquote").Each(func(i int, s *goquery.Selection) { + text := s.Text() + parts := strings.Split(text, " - ") + if len(parts) == 2 { + story.Quote = &Quote{ + Text: strings.TrimSpace(parts[0]), + Attribution: strings.TrimSpace(parts[1]), + } + } + }) + + // Extract sources + doc.Find("h3:contains('Sources')").Next("ul").Find("li").Each(func(i int, s *goquery.Selection) { + link := s.Find("a").First() + url, _ := link.Attr("href") + title := link.Text() + domain := extractDomain(s.Text()) + + story.Sources = append(story.Sources, Source{ + Title: title, + URL: url, + Domain: domain, + }) + }) + + // Extract image + img := doc.Find("img").First() + if img.Length() > 0 { + story.ImageURL, _ = img.Attr("src") + story.ImageAlt, _ = img.Attr("alt") + } + + return story, nil +} +``` + +**Libraries:** +- `github.com/PuerkitoBio/goquery` - HTML parsing + +--- + +### Component 3: Deduplication + +**Responsibility:** Track posted stories to prevent duplicates + +```go +type Deduplicator struct { + db *sql.DB +} + +func (d *Deduplicator) AlreadyPosted(guid string) (bool, error) { + var exists bool + err := d.db.QueryRow(` + SELECT EXISTS( + SELECT 1 FROM kagi_news_posted_stories + WHERE guid = $1 + ) + `, guid).Scan(&exists) + return exists, err +} + +func (d *Deduplicator) MarkPosted(guid, postURI string) error { + _, err := d.db.Exec(` + INSERT INTO kagi_news_posted_stories (guid, post_uri, posted_at) + VALUES ($1, $2, NOW()) + ON CONFLICT (guid) DO NOTHING + `, guid, postURI) + return err +} +``` + +**Database Table:** +```sql +CREATE TABLE kagi_news_posted_stories ( + guid TEXT PRIMARY KEY, + post_uri TEXT NOT NULL, + posted_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE INDEX idx_kagi_posted_at ON kagi_news_posted_stories(posted_at DESC); +``` + +**Cleanup:** Periodic job deletes rows older than 30 days (Kagi unlikely to re-post old stories). + +--- + +### Component 4: Category Mapper + +**Responsibility:** Map Kagi categories to authorized communities + +```go +func (m *CategoryMapper) GetTargetCommunities(story *KagiStory) ([]*CommunityAuth, error) { + // Get all communities that have authorized this aggregator + allAuths, err := m.aggregator.GetAuthorizedCommunities(context.Background()) + if err != nil { + return nil, err + } + + var targets []*CommunityAuth + for _, auth := range allAuths { + if !auth.Enabled { + continue + } + + config := auth.Config + + // Check if story's primary category is in config.categories + primaryCategory := story.Categories[0] + if !contains(config["categories"], primaryCategory) { + continue + } + + // Check subcategory filter (if specified) + if subcatFilter, ok := config["subcategoryFilter"].([]string); ok && len(subcatFilter) > 0 { + if !hasAnySubcategory(story.Categories, subcatFilter) { + continue + } + } + + // Check minimum sources requirement + minSources := config["minSources"].(int) + if len(story.Sources) < minSources { + continue + } + + targets = append(targets, auth) + } + + return targets, nil +} +``` + +--- + +### Component 5: Post Formatter + +**Responsibility:** Convert Kagi story to Coves post format + +```go +func (f *PostFormatter) Format(story *KagiStory, format string) string { + switch format { + case "full": + return f.formatFull(story) + case "summary": + return f.formatSummary(story) + case "minimal": + return f.formatMinimal(story) + default: + return f.formatFull(story) + } +} + +func (f *PostFormatter) formatFull(story *KagiStory) string { + var buf strings.Builder + + // Summary + buf.WriteString(story.Summary) + buf.WriteString("\n\n") + + // Highlights + if len(story.Highlights) > 0 { + buf.WriteString("**Highlights:**\n") + for _, h := range story.Highlights { + buf.WriteString(fmt.Sprintf("โ€ข %s\n", h)) + } + buf.WriteString("\n") + } + + // Perspectives + if len(story.Perspectives) > 0 { + buf.WriteString("**Perspectives:**\n") + for _, p := range story.Perspectives { + buf.WriteString(fmt.Sprintf("โ€ข **%s**: %s ([Source](%s))\n", p.Actor, p.Description, p.SourceURL)) + } + buf.WriteString("\n") + } + + // Quote + if story.Quote != nil { + buf.WriteString(fmt.Sprintf("> %s โ€” %s\n\n", story.Quote.Text, story.Quote.Attribution)) + } + + // Sources + buf.WriteString("**Sources:**\n") + for _, s := range story.Sources { + buf.WriteString(fmt.Sprintf("โ€ข [%s](%s) - %s\n", s.Title, s.URL, s.Domain)) + } + buf.WriteString("\n") + + // Attribution + buf.WriteString(fmt.Sprintf("---\n๐Ÿ“ฐ Story aggregated by [Kagi News](%s)", story.Link)) + + return buf.String() +} + +func (f *PostFormatter) formatSummary(story *KagiStory) string { + var buf strings.Builder + + buf.WriteString(story.Summary) + buf.WriteString("\n\n**Sources:**\n") + for _, s := range story.Sources { + buf.WriteString(fmt.Sprintf("โ€ข [%s](%s) - %s\n", s.Title, s.URL, s.Domain)) + } + buf.WriteString("\n") + buf.WriteString(fmt.Sprintf("---\n๐Ÿ“ฐ Story aggregated by [Kagi News](%s)", story.Link)) + + return buf.String() +} + +func (f *PostFormatter) formatMinimal(story *KagiStory) string { + sourceDomains := make([]string, len(story.Sources)) + for i, s := range story.Sources { + sourceDomains[i] = s.Domain + } + + return fmt.Sprintf( + "%s\n\nRead more: %s\n\n**Sources:** %s\n\n---\n๐Ÿ“ฐ Via [Kagi News](%s)", + story.Title, + story.Link, + strings.Join(sourceDomains, ", "), + story.Link, + ) +} +``` + +--- + +### Component 6: Post Publisher + +**Responsibility:** Create posts via Coves API + +```go +func (p *PostPublisher) PublishStory(ctx context.Context, story *KagiStory, communities []*CommunityAuth) error { + for _, comm := range communities { + config := comm.Config + + // Format content based on config + postFormat := config["postFormat"].(string) + content := p.formatter.Format(story, postFormat) + + // Build embed + var embed *aggregator.Embed + if config["includeImages"].(bool) && story.ImageURL != "" { + // TODO: Handle image upload/blob creation + embed = &aggregator.Embed{ + Type: "app.bsky.embed.external", + External: &aggregator.External{ + URI: story.Link, + Title: story.Title, + Description: truncate(story.Summary, 300), + Thumb: story.ImageURL, // or blob reference + }, + } + } + + // Create post + post := aggregator.Post{ + Title: story.Title, + Content: content, + Embed: embed, + FederatedFrom: &aggregator.FederatedSource{ + Platform: "kagi-news-rss", + URI: story.Link, + ID: story.GUID, + OriginalCreatedAt: story.PubDate, + }, + ContentLabels: story.Categories, + } + + err := p.aggregator.CreatePost(ctx, comm.CommunityDID, post) + if err != nil { + log.Printf("Failed to create post in %s: %v", comm.CommunityDID, err) + continue + } + + // Mark as posted + _ = p.deduplicator.MarkPosted(story.GUID, "post-uri-from-response") + } + + return nil +} +``` + +--- + +## Image Handling Strategy + +### Initial Implementation (MVP) + +**Approach:** Use Kagi proxy URLs directly in embeds + +**Rationale:** +- Simplest implementation +- Kagi proxy likely allows hotlinking for non-commercial use +- No storage costs +- Images are already optimized by Kagi + +**Risk Mitigation:** +- Monitor for broken images +- Add fallback: if image fails to load, skip embed +- Prepare migration plan to self-hosting if needed + +**Code:** +```go +if config["includeImages"].(bool) && story.ImageURL != "" { + // Use Kagi proxy URL directly + embed = &aggregator.Embed{ + External: &aggregator.External{ + Thumb: story.ImageURL, // https://kagiproxy.com/img/... + }, + } +} +``` + +--- + +### Future Enhancement (If Issues Arise) + +**Approach:** Download and re-host images + +**Implementation:** +1. Download image from Kagi proxy +2. Upload to Coves blob storage (or S3/CDN) +3. Use blob reference in embed + +**Code:** +```go +func (p *PostPublisher) uploadImage(imageURL string) (string, error) { + // Download from Kagi proxy + resp, err := http.Get(imageURL) + if err != nil { + return "", err + } + defer resp.Body.Close() + + // Upload to blob storage + blob, err := p.blobStore.Upload(resp.Body, resp.Header.Get("Content-Type")) + if err != nil { + return "", err + } + + return blob.Ref, nil +} +``` + +**Decision Point:** Only implement if: +- Kagi blocks hotlinking +- Kagi proxy becomes unreliable +- Legal clarification needed + +--- + +## Rate Limiting & Performance + +### Rate Limits + +**RSS Fetching:** +- Poll each category feed every 15 minutes +- Max 4 categories = 4 requests per 15 min = 16 req/hour +- Well within any reasonable limit + +**Post Creation:** +- Aggregator rate limit: 10 posts/hour per community +- Global limit: 100 posts/hour across all communities +- Kagi News publishes ~5-10 stories per category per day +- = ~20-40 posts/day total across all categories +- = ~2-4 posts/hour average +- Well within limits + +**Performance Targets:** +- Story posted within 15 minutes of appearing in RSS feed +- < 1 second to parse and format a story +- < 500ms to publish a post via API + +--- + +## Monitoring & Observability + +### Metrics to Track + +**Feed Polling:** +- `kagi_feed_poll_total` (counter) - Total feed polls by category +- `kagi_feed_poll_errors` (counter) - Failed polls by category/error +- `kagi_feed_items_fetched` (gauge) - Items per poll by category +- `kagi_feed_poll_duration_seconds` (histogram) - Poll latency + +**Story Processing:** +- `kagi_stories_parsed_total` (counter) - Successfully parsed stories +- `kagi_stories_parse_errors` (counter) - Parse failures by error type +- `kagi_stories_filtered` (counter) - Stories filtered out by reason (duplicate, min sources, category) +- `kagi_stories_posted` (counter) - Stories successfully posted by community + +**Post Publishing:** +- `kagi_posts_created_total` (counter) - Total posts created +- `kagi_posts_failed` (counter) - Failed posts by error type +- `kagi_post_publish_duration_seconds` (histogram) - Post creation latency + +**Health:** +- `kagi_aggregator_up` (gauge) - Service health (1 = healthy, 0 = down) +- `kagi_last_successful_poll_timestamp` (gauge) - Last successful poll time by category + +--- + +### Logging + +**Structured Logging:** +```go +log.Info("Story posted", + "guid", story.GUID, + "title", story.Title, + "community", comm.CommunityDID, + "post_uri", postURI, + "sources", len(story.Sources), + "format", postFormat, +) + +log.Error("Failed to parse story", + "guid", item.GUID, + "feed", feedURL, + "error", err, +) +``` + +**Log Levels:** +- DEBUG: Feed items, parsing details +- INFO: Stories posted, communities targeted +- WARN: Parse errors, rate limit approaching +- ERROR: Failed posts, feed fetch failures + +--- + +### Alerts + +**Critical:** +- Feed polling failing for > 1 hour +- Post creation failing for > 10 consecutive attempts +- Aggregator unauthorized (auth record disabled/deleted) + +**Warning:** +- Post creation rate < 50% of expected +- Parse errors > 10% of items +- Approaching rate limits (> 80% of quota) + +--- + +## Deployment + +### Infrastructure + +**Service Type:** Long-running daemon + +**Hosting:** Kubernetes (same cluster as Coves AppView) + +**Resources:** +- CPU: 0.5 cores (low CPU usage, mostly I/O) +- Memory: 512 MB (small in-memory cache for recent GUIDs) +- Storage: 1 GB (SQLite for deduplication tracking) + +--- + +### Configuration + +**Environment Variables:** +```bash +# Aggregator identity +AGGREGATOR_DID=did:web:kagi-news.coves.social +AGGREGATOR_PRIVATE_KEY_PATH=/secrets/private-key.pem + +# Coves API +COVES_API_URL=https://api.coves.social + +# Feed polling +POLL_INTERVAL=15m +CATEGORIES=world,tech,business,sports + +# Database (for deduplication) +DB_PATH=/data/kagi-news.db + +# Monitoring +METRICS_PORT=9090 +LOG_LEVEL=info +``` + +--- + +### Deployment Manifest + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: kagi-news-aggregator + namespace: coves +spec: + replicas: 1 + selector: + matchLabels: + app: kagi-news-aggregator + template: + metadata: + labels: + app: kagi-news-aggregator + spec: + containers: + - name: aggregator + image: coves/kagi-news-aggregator:latest + env: + - name: AGGREGATOR_DID + value: did:web:kagi-news.coves.social + - name: COVES_API_URL + value: https://api.coves.social + - name: POLL_INTERVAL + value: 15m + - name: CATEGORIES + value: world,tech,business,sports + - name: DB_PATH + value: /data/kagi-news.db + - name: AGGREGATOR_PRIVATE_KEY_PATH + value: /secrets/private-key.pem + volumeMounts: + - name: data + mountPath: /data + - name: secrets + mountPath: /secrets + readOnly: true + ports: + - name: metrics + containerPort: 9090 + resources: + requests: + cpu: 250m + memory: 256Mi + limits: + cpu: 500m + memory: 512Mi + volumes: + - name: data + persistentVolumeClaim: + claimName: kagi-news-data + - name: secrets + secret: + secretName: kagi-news-private-key +``` + +--- + +## Testing Strategy + +### Unit Tests + +**Feed Parsing:** +```go +func TestParseFeed(t *testing.T) { + feed := loadTestFeed("testdata/world.xml") + stories, err := parser.Parse(feed) + assert.NoError(t, err) + assert.Len(t, stories, 10) + + story := stories[0] + assert.NotEmpty(t, story.Title) + assert.NotEmpty(t, story.Summary) + assert.Greater(t, len(story.Sources), 1) +} + +func TestParseStoryHTML(t *testing.T) { + html := `

Summary [source.com#1]

+

Highlights:

+
  • Point 1
+

Sources:

+ ` + + story, err := parser.ParseHTML(html) + assert.NoError(t, err) + assert.Equal(t, "Summary [source.com#1]", story.Summary) + assert.Len(t, story.Highlights, 1) + assert.Len(t, story.Sources, 1) +} +``` + +**Formatting:** +```go +func TestFormatFull(t *testing.T) { + story := &KagiStory{ + Summary: "Test summary", + Sources: []Source{ + {Title: "Article", URL: "https://example.com", Domain: "example.com"}, + }, + } + + content := formatter.Format(story, "full") + assert.Contains(t, content, "Test summary") + assert.Contains(t, content, "**Sources:**") + assert.Contains(t, content, "๐Ÿ“ฐ Story aggregated by") +} +``` + +**Deduplication:** +```go +func TestDeduplication(t *testing.T) { + guid := "test-guid-123" + + posted, err := deduplicator.AlreadyPosted(guid) + assert.NoError(t, err) + assert.False(t, posted) + + err = deduplicator.MarkPosted(guid, "at://...") + assert.NoError(t, err) + + posted, err = deduplicator.AlreadyPosted(guid) + assert.NoError(t, err) + assert.True(t, posted) +} +``` + +--- + +### Integration Tests + +**With Mock Coves API:** +```go +func TestPublishStory(t *testing.T) { + // Setup mock Coves API + mockAPI := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + assert.Equal(t, "/xrpc/social.coves.post.create", r.URL.Path) + + var input CreatePostInput + json.NewDecoder(r.Body).Decode(&input) + + assert.Equal(t, "did:plc:test-community", input.Community) + assert.NotEmpty(t, input.Title) + assert.Contains(t, input.Content, "๐Ÿ“ฐ Story aggregated by") + + w.WriteHeader(200) + json.NewEncoder(w).Encode(CreatePostOutput{URI: "at://..."}) + })) + defer mockAPI.Close() + + // Test story publishing + publisher := NewPostPublisher(mockAPI.URL) + err := publisher.PublishStory(ctx, testStory, []*CommunityAuth{testComm}) + assert.NoError(t, err) +} +``` + +--- + +### E2E Tests + +**With Real RSS Feed:** +```go +func TestE2E_FetchAndParse(t *testing.T) { + if testing.Short() { + t.Skip("Skipping E2E test") + } + + // Fetch real Kagi News feed + feed, err := poller.fetchFeed("https://news.kagi.com/world.xml") + assert.NoError(t, err) + assert.NotEmpty(t, feed.Items) + + // Parse first item + story, err := parser.Parse(feed.Items[0]) + assert.NoError(t, err) + assert.NotEmpty(t, story.Title) + assert.NotEmpty(t, story.Summary) + assert.Greater(t, len(story.Sources), 0) +} +``` + +**With Test Coves Instance:** +```go +func TestE2E_CreatePost(t *testing.T) { + if testing.Short() { + t.Skip("Skipping E2E test") + } + + // Create post in test community + post := aggregator.Post{ + Title: "Test Kagi News Post", + Content: "Test content...", + } + + err := aggregator.CreatePost(ctx, testCommunityDID, post) + assert.NoError(t, err) + + // Verify post appears in feed + // (requires test community setup) +} +``` + +--- + +## Success Metrics + +### Pre-Launch Checklist + +- [ ] Aggregator service declaration published +- [ ] DID created and configured (did:web:kagi-news.coves.social) +- [ ] RSS feed parser handles all Kagi HTML structures +- [ ] Deduplication prevents duplicate posts +- [ ] Category mapping works for all configs +- [ ] All 3 post formats render correctly +- [ ] Attribution to Kagi News visible on all posts +- [ ] Rate limiting prevents spam +- [ ] Monitoring/alerting configured +- [ ] E2E tests passing against test instance + +--- + +### Alpha Goals (First Week) + +- [ ] 3+ communities using Kagi News aggregator +- [ ] 50+ posts created successfully +- [ ] Zero duplicate posts +- [ ] < 5% parse errors +- [ ] < 1% post creation failures +- [ ] Stories posted within 15 minutes of RSS publication + +--- + +### Beta Goals (First Month) + +- [ ] 10+ communities using aggregator +- [ ] 500+ posts created +- [ ] Community feedback positive (surveys) +- [ ] Attribution compliance verified +- [ ] No rate limit violations +- [ ] < 1% error rate (parsing + posting) + +--- + +## Future Enhancements + +### Phase 2 Features + +**Smart Category Detection:** +- Use LLM to suggest additional categories for stories +- Map Kagi categories to community tags automatically + +**Customizable Templates:** +- Allow communities to customize post format with templates +- Support Markdown/Handlebars templates in config + +**Story Scoring:** +- Prioritize high-impact stories (many sources, breaking news) +- Delay low-priority stories to avoid flooding feed + +**Cross-posting Prevention:** +- Detect when multiple communities authorize same category +- Intelligently cross-post vs. duplicate + +--- + +### Phase 3 Features + +**Interactive Features:** +- Bot responds to comments with additional sources +- Updates megathread with new sources as story develops + +**Analytics Dashboard:** +- Show communities which stories get most engagement +- Trending topics from Kagi News +- Source diversity metrics + +**Federation:** +- Support other Coves instances using same aggregator +- Shared deduplication across instances + +--- + +## Open Questions + +### Need to Resolve Before Launch + +1. **Image Licensing:** + - โ“ Are images from Kagi proxy covered by CC BY-NC? + - โ“ Do we need to attribute original image sources? + - **Action:** Email support@kagi.com for clarification + +2. **Hotlinking Policy:** + - โ“ Is embedding Kagi proxy images acceptable? + - โ“ Should we download and re-host? + - **Action:** Test in staging, monitor for issues + +3. **Category Discovery:** + - โ“ How to discover all available category feeds? + - โ“ Are there categories beyond world/tech/business/sports? + - **Action:** Scrape https://news.kagi.com/ for all .xml links + +4. **Attribution Format:** + - โ“ Is "๐Ÿ“ฐ Story aggregated by Kagi News" sufficient? + - โ“ Do we need more prominent attribution? + - **Action:** Review CC BY-NC best practices + +--- + +## References + +- Kagi News About Page: https://news.kagi.com/about +- Kagi News RSS Example: https://news.kagi.com/world.xml +- Kagi Kite Public Repo: https://github.com/kagisearch/kite-public +- CC BY-NC License: https://creativecommons.org/licenses/by-nc/4.0/ +- Parent PRD: [PRD_AGGREGATORS.md](PRD_AGGREGATORS.md) +- Aggregator SDK: [TBD] -- 2.51.2