From c8834287bd78e9ba531094d59594a6c43d5e24ca Mon Sep 17 00:00:00 2001 From: Bretton Date: Mon, 17 Nov 2025 19:34:25 -0800 Subject: [PATCH] feat(aggregator): add setup scripts for aggregator registration MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add comprehensive setup scripts and documentation for aggregator registration. Scripts: - 1-create-pds-account.sh: Create PDS account for aggregator - 2-setup-wellknown.sh: Generate .well-known/atproto-did file - 3-register-with-coves.sh: Register with Coves instance via XRPC - 4-create-service-declaration.sh: Create service declaration record Documentation: - Detailed README with step-by-step instructions - Troubleshooting guide - Configuration examples (nginx/Apache) - Security best practices These scripts automate the 4-step process of: 1. Creating a DID for the aggregator 2. Proving domain ownership 3. Registering with Coves 4. Publishing service metadata 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude --- docs/aggregators/SETUP_GUIDE.md | 591 ++++++++++++++++++ .../aggregator-setup/1-create-pds-account.sh | 95 +++ scripts/aggregator-setup/2-setup-wellknown.sh | 93 +++ .../aggregator-setup/3-register-with-coves.sh | 103 +++ .../4-create-service-declaration.sh | 125 ++++ scripts/aggregator-setup/README.md | 252 ++++++++ 6 files changed, 1259 insertions(+) create mode 100644 docs/aggregators/SETUP_GUIDE.md create mode 100755 scripts/aggregator-setup/1-create-pds-account.sh create mode 100755 scripts/aggregator-setup/2-setup-wellknown.sh create mode 100755 scripts/aggregator-setup/3-register-with-coves.sh create mode 100755 scripts/aggregator-setup/4-create-service-declaration.sh create mode 100644 scripts/aggregator-setup/README.md diff --git a/docs/aggregators/SETUP_GUIDE.md b/docs/aggregators/SETUP_GUIDE.md new file mode 100644 index 0000000..b256a97 --- /dev/null +++ b/docs/aggregators/SETUP_GUIDE.md @@ -0,0 +1,591 @@ +# Aggregator Setup Guide + +This guide explains how to set up and register an aggregator with Coves instances. + +## Table of Contents + +- [Overview](#overview) +- [Architecture](#architecture) +- [Prerequisites](#prerequisites) +- [Quick Start](#quick-start) +- [Detailed Setup Steps](#detailed-setup-steps) +- [Authorization Process](#authorization-process) +- [Posting to Communities](#posting-to-communities) +- [Rate Limits](#rate-limits) +- [Security Best Practices](#security-best-practices) +- [Troubleshooting](#troubleshooting) +- [API Reference](#api-reference) + +## Overview + +**Aggregators** are automated services that post content to Coves communities. They are similar to Bluesky's feed generators and labelers - self-managed external services that integrate with the platform. + +**Key characteristics**: +- Self-owned: You create and manage your own PDS account +- Domain-verified: Prove ownership via `.well-known/atproto-did` +- Community-authorized: Moderators grant posting permission per-community +- Rate-limited: 10 posts per hour per community + +**Example use cases**: +- RSS feed aggregators (tech news, blog posts) +- Social media cross-posters (Twitter → Coves) +- Event notifications (GitHub releases, weather alerts) +- Content curation bots (daily links, summaries) + +## Architecture + +### Data Flow + +``` +┌──────────────────────────────────────────────────────────┐ +│ 1. One-Time Setup │ +├──────────────────────────────────────────────────────────┤ +│ Aggregator creates PDS account │ +│ ↓ │ +│ Proves domain ownership (.well-known) │ +│ ↓ │ +│ Registers with Coves (enters users table) │ +│ ↓ │ +│ Writes service declaration │ +│ ↓ │ +│ Jetstream indexes into aggregators table │ +└──────────────────────────────────────────────────────────┘ + +┌──────────────────────────────────────────────────────────┐ +│ 2. Per-Community Authorization │ +├──────────────────────────────────────────────────────────┤ +│ Moderator writes authorization record │ +│ ↓ │ +│ Jetstream indexes into aggregator_authorizations │ +└──────────────────────────────────────────────────────────┘ + +┌──────────────────────────────────────────────────────────┐ +│ 3. Posting (Ongoing) │ +├──────────────────────────────────────────────────────────┤ +│ Aggregator calls post creation endpoint │ +│ ↓ │ +│ Handler validates: │ +│ - Author in users table ✓ │ +│ - Author in aggregators table ✓ │ +│ - Authorization exists ✓ │ +│ - Rate limit not exceeded ✓ │ +│ ↓ │ +│ Post written to community's PDS │ +│ ↓ │ +│ Jetstream indexes post │ +└──────────────────────────────────────────────────────────┘ +``` + +### Database Tables + +**users** - All actors (users, communities, aggregators) +```sql +CREATE TABLE users ( + did TEXT PRIMARY KEY, + handle TEXT NOT NULL, + pds_url TEXT, + indexed_at TIMESTAMPTZ +); +``` + +**aggregators** - Aggregator-specific metadata +```sql +CREATE TABLE aggregators ( + did TEXT PRIMARY KEY, + display_name TEXT NOT NULL, + description TEXT, + avatar_url TEXT, + config_schema JSONB, + source_url TEXT, + maintainer_did TEXT, + record_uri TEXT NOT NULL UNIQUE, + record_cid TEXT NOT NULL, + created_at TIMESTAMPTZ, + indexed_at TIMESTAMPTZ +); +``` + +**aggregator_authorizations** - Community authorizations +```sql +CREATE TABLE aggregator_authorizations ( + id BIGSERIAL PRIMARY KEY, + aggregator_did TEXT NOT NULL, + community_did TEXT NOT NULL, + enabled BOOLEAN NOT NULL DEFAULT true, + config JSONB, + created_by TEXT, + record_uri TEXT NOT NULL UNIQUE, + record_cid TEXT NOT NULL, + UNIQUE(aggregator_did, community_did) +); +``` + +## Prerequisites + +1. **Domain ownership**: You must own a domain where you can host static files over HTTPS +2. **Web server**: Ability to serve the `.well-known/atproto-did` file +3. **Development tools**: `curl`, `jq`, basic shell scripting knowledge +4. **Email address**: For creating the PDS account + +**Optional**: +- Custom avatar image (PNG/JPEG/WebP, max 1MB) +- GitHub repository for source code transparency + +## Quick Start + +We provide automated setup scripts: + +```bash +cd scripts/aggregator-setup + +# Make scripts executable +chmod +x *.sh + +# Run setup scripts in order +./1-create-pds-account.sh +./2-setup-wellknown.sh +# (Upload .well-known to your web server) +./3-register-with-coves.sh +./4-create-service-declaration.sh +``` + +See [scripts/aggregator-setup/README.md](../../scripts/aggregator-setup/README.md) for detailed script documentation. + +## Detailed Setup Steps + +### Step 1: Create PDS Account + +Your aggregator needs its own atProto identity (DID). The easiest way is to create an account on an existing PDS. + +**Using an existing PDS (recommended)**: + +```bash +curl -X POST https://bsky.social/xrpc/com.atproto.server.createAccount \ + -H "Content-Type: application/json" \ + -d '{ + "handle": "mynewsbot.bsky.social", + "email": "bot@example.com", + "password": "secure-password-here" + }' +``` + +**Response**: +```json +{ + "accessJwt": "eyJ...", + "refreshJwt": "eyJ...", + "handle": "mynewsbot.bsky.social", + "did": "did:plc:abc123...", + "didDoc": {...} +} +``` + +**Save these credentials securely!** You'll need the DID and access token for all subsequent operations. + +**Alternative**: Run your own PDS or use `did:web` (advanced). + +### Step 2: Prove Domain Ownership + +To register with Coves, you must prove you own a domain by serving your DID at `https://yourdomain.com/.well-known/atproto-did`. + +**Create the file**: + +```bash +mkdir -p .well-known +echo "did:plc:abc123..." > .well-known/atproto-did +``` + +**Upload to your web server** so it's accessible at: +``` +https://rss-bot.example.com/.well-known/atproto-did +``` + +**Verify it works**: +```bash +curl https://rss-bot.example.com/.well-known/atproto-did +# Should return: did:plc:abc123... +``` + +**Nginx configuration example**: +```nginx +location /.well-known/atproto-did { + alias /var/www/.well-known/atproto-did; + default_type text/plain; + add_header Access-Control-Allow-Origin *; +} +``` + +### Step 3: Register with Coves + +Call the registration endpoint to register your aggregator DID with the Coves instance. + +**Endpoint**: `POST /xrpc/social.coves.aggregator.register` + +**Request**: +```bash +curl -X POST https://api.coves.social/xrpc/social.coves.aggregator.register \ + -H "Content-Type: application/json" \ + -d '{ + "did": "did:plc:abc123...", + "domain": "rss-bot.example.com" + }' +``` + +**Response** (Success): +```json +{ + "did": "did:plc:abc123...", + "handle": "mynewsbot.bsky.social", + "message": "Aggregator registered successfully. Next step: create a service declaration record at at://did:plc:abc123.../social.coves.aggregator.service/self" +} +``` + +**What happens**: +1. Coves fetches `https://rss-bot.example.com/.well-known/atproto-did` +2. Verifies it contains your DID +3. Resolves your DID to get handle and PDS URL +4. Inserts you into the `users` table + +**You're now registered!** But you need to create a service declaration next. + +### Step 4: Create Service Declaration + +Write a `social.coves.aggregator.service` record to your repository. This contains metadata about your aggregator and gets indexed by Coves' Jetstream consumer. + +**Endpoint**: `POST https://your-pds.com/xrpc/com.atproto.repo.createRecord` + +**Request**: +```bash +curl -X POST https://bsky.social/xrpc/com.atproto.repo.createRecord \ + -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "repo": "did:plc:abc123...", + "collection": "social.coves.aggregator.service", + "rkey": "self", + "record": { + "$type": "social.coves.aggregator.service", + "did": "did:plc:abc123...", + "displayName": "RSS News Aggregator", + "description": "Aggregates tech news from various RSS feeds", + "sourceUrl": "https://github.com/yourname/rss-aggregator", + "maintainer": "did:plc:your-personal-did", + "createdAt": "2024-01-15T12:00:00Z" + } + }' +``` + +**Response**: +```json +{ + "uri": "at://did:plc:abc123.../social.coves.aggregator.service/self", + "cid": "bafyrei..." +} +``` + +**Optional fields**: +- `avatar`: Blob reference to avatar image +- `configSchema`: JSON Schema for community-specific configuration + +**Wait 5-10 seconds** for Jetstream to index your service declaration into the `aggregators` table. + +## Authorization Process + +Before you can post to a community, a moderator must authorize your aggregator. + +### How Authorization Works + +1. **Moderator decision**: Community moderator evaluates your aggregator +2. **Authorization record**: Moderator writes `social.coves.aggregator.authorization` to community's repo +3. **Jetstream indexing**: Record gets indexed into `aggregator_authorizations` table +4. **Posting enabled**: You can now post to that community + +### Authorization Record Structure + +**Location**: `at://{community_did}/social.coves.aggregator.authorization/{rkey}` + +**Example**: +```json +{ + "$type": "social.coves.aggregator.authorization", + "aggregatorDid": "did:plc:abc123...", + "communityDid": "did:plc:community123...", + "enabled": true, + "createdBy": "did:plc:moderator...", + "createdAt": "2024-01-15T12:00:00Z", + "config": { + "maxPostsPerHour": 5, + "allowedCategories": ["tech", "news"] + } +} +``` + +### Checking Your Authorizations + +**Endpoint**: `GET /xrpc/social.coves.aggregator.getAuthorizations` + +```bash +curl "https://api.coves.social/xrpc/social.coves.aggregator.getAuthorizations?aggregatorDid=did:plc:abc123...&enabledOnly=true" +``` + +**Response**: +```json +{ + "authorizations": [ + { + "aggregatorDid": "did:plc:abc123...", + "communityDid": "did:plc:community123...", + "communityHandle": "~tech@coves.social", + "enabled": true, + "createdAt": "2024-01-15T12:00:00Z", + "config": {...} + } + ] +} +``` + +## Posting to Communities + +Once authorized, you can post to communities using the standard post creation endpoint. + +### Create Post + +**Endpoint**: `POST /xrpc/social.coves.community.post.create` + +**Request**: +```bash +curl -X POST https://api.coves.social/xrpc/social.coves.community.post.create \ + -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "communityDid": "did:plc:community123...", + "post": { + "text": "New blog post: Understanding atProto Identity\nhttps://example.com/post", + "createdAt": "2024-01-15T12:00:00Z", + "facets": [ + { + "index": { "byteStart": 50, "byteEnd": 75 }, + "features": [ + { + "$type": "social.coves.richtext.facet#link", + "uri": "https://example.com/post" + } + ] + } + ] + } + }' +``` + +**Response**: +```json +{ + "uri": "at://did:plc:abc123.../social.coves.community.post/3k...", + "cid": "bafyrei..." +} +``` + +### Post Validation + +The handler validates: +1. **Authentication**: Valid JWT token +2. **Author exists**: DID in `users` table +3. **Is aggregator**: DID in `aggregators` table +4. **Authorization**: Active authorization for (aggregator, community) +5. **Rate limit**: Less than 10 posts/hour to this community +6. **Content**: Valid post structure per lexicon + +### Rate Limits + +**Per-community rate limit**: 10 posts per hour + +This is tracked in the `aggregator_posts` table and enforced at the handler level. + +**Why?**: Prevents spam while allowing useful bot activity. + +**Best practices**: +- Batch similar content +- Post only high-quality content +- Respect community guidelines +- Monitor your posting rate + +## Security Best Practices + +### Credential Management + +✅ **DO**: +- Store credentials in environment variables or secret management +- Use HTTPS for all API calls +- Rotate access tokens regularly (use refresh tokens) +- Keep `aggregator-config.env` out of version control + +❌ **DON'T**: +- Hardcode credentials in source code +- Commit credentials to Git +- Share access tokens publicly +- Reuse personal credentials for bots + +### Domain Security + +✅ **DO**: +- Use HTTPS for `.well-known` endpoint +- Keep domain under your control +- Monitor for unauthorized changes +- Use DNSSEC if possible + +❌ **DON'T**: +- Use HTTP (will fail verification) +- Use shared/untrusted hosting +- Allow others to modify `.well-known` files +- Use expired SSL certificates + +### Content Security + +✅ **DO**: +- Validate all external content before posting +- Sanitize URLs and text +- Rate-limit your own posting +- Implement circuit breakers for failures + +❌ **DON'T**: +- Post unvalidated user input +- Include malicious links +- Spam communities +- Bypass rate limits + +## Troubleshooting + +### Registration Errors + +#### Error: "DomainVerificationFailed" + +**Cause**: `.well-known/atproto-did` not accessible or contains wrong DID + +**Solutions**: +1. Verify file is accessible: `curl https://yourdomain.com/.well-known/atproto-did` +2. Check content matches your DID exactly (no extra whitespace) +3. Ensure HTTPS is working (not HTTP) +4. Check web server logs for access errors +5. Verify firewall rules allow HTTPS traffic + +#### Error: "AlreadyRegistered" + +**Cause**: This DID is already registered with this Coves instance + +**Solutions**: +- This is safe to ignore if you're re-running setup +- If you need to update info, just create a new service declaration +- Contact instance admin if you need to remove registration + +#### Error: "DIDResolutionFailed" + +**Cause**: Could not resolve DID document from PLC directory + +**Solutions**: +1. Verify DID exists: `curl https://plc.directory/{your-did}` +2. Wait 30 seconds and retry (PLC propagation delay) +3. Check PDS is accessible +4. Verify DID format is correct (must start with `did:plc:` or `did:web:`) + +### Posting Errors + +#### Error: "NotAuthorized" + +**Cause**: No active authorization for this (aggregator, community) pair + +**Solutions**: +1. Check authorizations: `GET /xrpc/social.coves.aggregator.getAuthorizations` +2. Contact community moderator to request authorization +3. Verify authorization wasn't disabled +4. Wait for Jetstream to index authorization (5-10 seconds) + +#### Error: "RateLimitExceeded" + +**Cause**: Exceeded 10 posts/hour to this community + +**Solutions**: +1. Wait for the rate limit window to reset +2. Batch posts to stay under limit +3. Distribute posts across multiple communities +4. Implement posting queue in your aggregator + +### Service Declaration Not Appearing + +**Symptoms**: Service declaration created but not in `aggregators` table + +**Solutions**: +1. Wait 5-10 seconds for Jetstream to index +2. Check Jetstream consumer logs for errors +3. Verify record was created: Check PDS at `at://your-did/social.coves.aggregator.service/self` +4. Verify `$type` field is exactly `"social.coves.aggregator.service"` +5. Check `displayName` is not empty (required field) + +## API Reference + +### Registration Endpoint + +**`POST /xrpc/social.coves.aggregator.register`** + +**Input**: +```typescript +{ + did: string // DID of aggregator (did:plc or did:web) + domain: string // Domain serving .well-known/atproto-did +} +``` + +**Output**: +```typescript +{ + did: string // Registered DID + handle: string // Handle from DID document + message: string // Next steps message +} +``` + +**Errors**: +- `InvalidDID`: DID format invalid +- `DomainVerificationFailed`: .well-known verification failed +- `AlreadyRegistered`: DID already registered +- `DIDResolutionFailed`: Could not resolve DID + +### Query Endpoints + +**`GET /xrpc/social.coves.aggregator.getServices`** + +Get aggregator service details. + +**Parameters**: +- `dids`: Array of DIDs (comma-separated) + +**`GET /xrpc/social.coves.aggregator.getAuthorizations`** + +List communities that authorized an aggregator. + +**Parameters**: +- `aggregatorDid`: Aggregator DID +- `enabledOnly`: Filter to enabled only (default: false) + +**`GET /xrpc/social.coves.aggregator.listForCommunity`** + +List aggregators authorized by a community. + +**Parameters**: +- `communityDid`: Community DID +- `enabledOnly`: Filter to enabled only (default: false) + +## Further Reading + +- [Aggregator PRD](PRD_AGGREGATORS.md) - Architecture and design decisions +- [atProto Guide](../../ATPROTO_GUIDE.md) - atProto fundamentals +- [Communities PRD](../PRD_COMMUNITIES.md) - Community system overview +- [Setup Scripts README](../../scripts/aggregator-setup/README.md) - Script documentation + +## Support + +For issues or questions: + +1. Check this guide's troubleshooting section +2. Review the PRD and architecture docs +3. Check Coves GitHub issues +4. Ask in Coves developer community diff --git a/scripts/aggregator-setup/1-create-pds-account.sh b/scripts/aggregator-setup/1-create-pds-account.sh new file mode 100755 index 0000000..4a76f28 --- /dev/null +++ b/scripts/aggregator-setup/1-create-pds-account.sh @@ -0,0 +1,95 @@ +#!/bin/bash + +# Script: 1-create-pds-account.sh +# Purpose: Create a PDS account for your aggregator +# +# This script helps you create an account on a PDS (Personal Data Server). +# The PDS will automatically create a DID:PLC for you. + +set -e + +echo "================================================" +echo "Step 1: Create PDS Account for Your Aggregator" +echo "================================================" +echo "" + +# Get PDS URL +read -p "Enter PDS URL (default: https://bsky.social): " PDS_URL +PDS_URL=${PDS_URL:-https://bsky.social} + +# Get credentials +read -p "Enter desired handle (e.g., mynewsbot.bsky.social): " HANDLE +read -p "Enter email: " EMAIL +read -sp "Enter password: " PASSWORD +echo "" + +# Validate inputs +if [ -z "$HANDLE" ] || [ -z "$EMAIL" ] || [ -z "$PASSWORD" ]; then + echo "Error: All fields are required" + exit 1 +fi + +echo "" +echo "Creating account on $PDS_URL..." + +# Create account via com.atproto.server.createAccount +RESPONSE=$(curl -s -X POST "$PDS_URL/xrpc/com.atproto.server.createAccount" \ + -H "Content-Type: application/json" \ + -d "{ + \"handle\": \"$HANDLE\", + \"email\": \"$EMAIL\", + \"password\": \"$PASSWORD\" + }") + +# Check if successful +if echo "$RESPONSE" | jq -e '.error' > /dev/null 2>&1; then + echo "Error creating account:" + echo "$RESPONSE" | jq '.' + exit 1 +fi + +# Extract DID and access token +DID=$(echo "$RESPONSE" | jq -r '.did') +ACCESS_JWT=$(echo "$RESPONSE" | jq -r '.accessJwt') +REFRESH_JWT=$(echo "$RESPONSE" | jq -r '.refreshJwt') + +if [ -z "$DID" ] || [ "$DID" = "null" ]; then + echo "Error: Failed to extract DID from response" + echo "$RESPONSE" | jq '.' + exit 1 +fi + +echo "" +echo "✓ Account created successfully!" +echo "" +echo "=== Save these credentials ====" +echo "DID: $DID" +echo "Handle: $HANDLE" +echo "PDS URL: $PDS_URL" +echo "Email: $EMAIL" +echo "Password: [hidden]" +echo "Access JWT: $ACCESS_JWT" +echo "Refresh JWT: $REFRESH_JWT" +echo "===============================" +echo "" + +# Save to config file +CONFIG_FILE="aggregator-config.env" +cat > "$CONFIG_FILE" <> aggregator-config.env +echo "AGGREGATOR_DOMAIN=\"$DOMAIN\"" >> aggregator-config.env + +echo "" +echo "Creating .well-known directory..." +mkdir -p .well-known + +# Create the atproto-did file +echo "$AGGREGATOR_DID" > .well-known/atproto-did + +echo "✓ Created .well-known/atproto-did with content: $AGGREGATOR_DID" +echo "" + +echo "================================================" +echo "Next Steps:" +echo "================================================" +echo "" +echo "1. Upload the .well-known directory to your web server" +echo " The file must be accessible at:" +echo " https://$DOMAIN/.well-known/atproto-did" +echo "" +echo "2. Verify it's working by running:" +echo " curl https://$DOMAIN/.well-known/atproto-did" +echo " (Should return: $AGGREGATOR_DID)" +echo "" +echo "3. Once verified, run: ./3-register-with-coves.sh" +echo "" + +# Create nginx example +cat > nginx-example.conf < apache-example.conf < + Options None + AllowOverride None + Require all granted + Header set Access-Control-Allow-Origin "*" + +EOF + +echo "✓ Created apache-example.conf for reference" diff --git a/scripts/aggregator-setup/3-register-with-coves.sh b/scripts/aggregator-setup/3-register-with-coves.sh new file mode 100755 index 0000000..0dfae0e --- /dev/null +++ b/scripts/aggregator-setup/3-register-with-coves.sh @@ -0,0 +1,103 @@ +#!/bin/bash + +# Script: 3-register-with-coves.sh +# Purpose: Register your aggregator with a Coves instance +# +# This script calls the social.coves.aggregator.register XRPC endpoint +# to register your aggregator DID with the Coves instance. + +set -e + +echo "================================================" +echo "Step 3: Register with Coves Instance" +echo "================================================" +echo "" + +# Load config if available +if [ -f "aggregator-config.env" ]; then + source aggregator-config.env + echo "✓ Loaded configuration from aggregator-config.env" + echo " DID: $AGGREGATOR_DID" + echo " Domain: $AGGREGATOR_DOMAIN" + echo "" +else + echo "Configuration file not found. Please run previous scripts first." + exit 1 +fi + +# Validate domain is set +if [ -z "$AGGREGATOR_DOMAIN" ]; then + echo "Error: AGGREGATOR_DOMAIN not set. Please run 2-setup-wellknown.sh first." + exit 1 +fi + +# Get Coves instance URL +read -p "Enter Coves instance URL (default: https://api.coves.social): " COVES_URL +COVES_URL=${COVES_URL:-https://api.coves.social} + +echo "" +echo "Verifying .well-known/atproto-did is accessible..." + +# Verify .well-known is accessible +WELLKNOWN_URL="https://$AGGREGATOR_DOMAIN/.well-known/atproto-did" +WELLKNOWN_CONTENT=$(curl -s "$WELLKNOWN_URL" || echo "ERROR") + +if [ "$WELLKNOWN_CONTENT" = "ERROR" ]; then + echo "✗ Error: Could not access $WELLKNOWN_URL" + echo " Please ensure the file is uploaded and accessible." + exit 1 +elif [ "$WELLKNOWN_CONTENT" != "$AGGREGATOR_DID" ]; then + echo "✗ Error: .well-known/atproto-did contains wrong DID" + echo " Expected: $AGGREGATOR_DID" + echo " Got: $WELLKNOWN_CONTENT" + exit 1 +fi + +echo "✓ .well-known/atproto-did is correctly configured" +echo "" + +echo "Registering with $COVES_URL..." + +# Call registration endpoint +RESPONSE=$(curl -s -X POST "$COVES_URL/xrpc/social.coves.aggregator.register" \ + -H "Content-Type: application/json" \ + -d "{ + \"did\": \"$AGGREGATOR_DID\", + \"domain\": \"$AGGREGATOR_DOMAIN\" + }") + +# Check if successful +if echo "$RESPONSE" | jq -e '.error' > /dev/null 2>&1; then + echo "✗ Registration failed:" + echo "$RESPONSE" | jq '.' + exit 1 +fi + +# Extract response +REGISTERED_DID=$(echo "$RESPONSE" | jq -r '.did') +REGISTERED_HANDLE=$(echo "$RESPONSE" | jq -r '.handle') +MESSAGE=$(echo "$RESPONSE" | jq -r '.message') + +if [ -z "$REGISTERED_DID" ] || [ "$REGISTERED_DID" = "null" ]; then + echo "✗ Error: Unexpected response format" + echo "$RESPONSE" | jq '.' + exit 1 +fi + +echo "" +echo "✓ Registration successful!" +echo "" +echo "=== Registration Details ====" +echo "DID: $REGISTERED_DID" +echo "Handle: $REGISTERED_HANDLE" +echo "Message: $MESSAGE" +echo "=============================" +echo "" + +# Save Coves URL to config +echo "" >> aggregator-config.env +echo "COVES_INSTANCE_URL=\"$COVES_URL\"" >> aggregator-config.env + +echo "✓ Updated aggregator-config.env with Coves instance URL" +echo "" +echo "Next step: Run ./4-create-service-declaration.sh" diff --git a/scripts/aggregator-setup/4-create-service-declaration.sh b/scripts/aggregator-setup/4-create-service-declaration.sh new file mode 100755 index 0000000..1807e44 --- /dev/null +++ b/scripts/aggregator-setup/4-create-service-declaration.sh @@ -0,0 +1,125 @@ +#!/bin/bash + +# Script: 4-create-service-declaration.sh +# Purpose: Create aggregator service declaration record +# +# This script writes a social.coves.aggregator.service record to your aggregator's repository. +# This record contains metadata about your aggregator (name, description, etc.) and will be +# indexed by Coves' Jetstream consumer into the aggregators table. + +set -e + +echo "================================================" +echo "Step 4: Create Service Declaration" +echo "================================================" +echo "" + +# Load config if available +if [ -f "aggregator-config.env" ]; then + source aggregator-config.env + echo "✓ Loaded configuration from aggregator-config.env" + echo " DID: $AGGREGATOR_DID" + echo " PDS URL: $AGGREGATOR_PDS_URL" + echo "" +else + echo "Configuration file not found. Please run previous scripts first." + exit 1 +fi + +# Validate required fields +if [ -z "$AGGREGATOR_ACCESS_JWT" ]; then + echo "Error: AGGREGATOR_ACCESS_JWT not set. Please run 1-create-pds-account.sh first." + exit 1 +fi + +echo "Enter aggregator metadata:" +echo "" + +# Get metadata from user +read -p "Display Name (e.g., 'RSS News Aggregator'): " DISPLAY_NAME +read -p "Description: " DESCRIPTION +read -p "Source URL (e.g., 'https://github.com/yourname/aggregator'): " SOURCE_URL +read -p "Maintainer DID (your personal DID, optional): " MAINTAINER_DID + +if [ -z "$DISPLAY_NAME" ]; then + echo "Error: Display name is required" + exit 1 +fi + +echo "" +echo "Creating service declaration record..." + +# Build the service record +SERVICE_RECORD=$(cat < /dev/null 2>&1; then + echo "✗ Failed to create service declaration:" + echo "$RESPONSE" | jq '.' + exit 1 +fi + +# Extract response +RECORD_URI=$(echo "$RESPONSE" | jq -r '.uri') +RECORD_CID=$(echo "$RESPONSE" | jq -r '.cid') + +if [ -z "$RECORD_URI" ] || [ "$RECORD_URI" = "null" ]; then + echo "✗ Error: Unexpected response format" + echo "$RESPONSE" | jq '.' + exit 1 +fi + +echo "" +echo "✓ Service declaration created successfully!" +echo "" +echo "=== Record Details ====" +echo "URI: $RECORD_URI" +echo "CID: $RECORD_CID" +echo "=======================" +echo "" + +# Save to config +echo "" >> aggregator-config.env +echo "SERVICE_DECLARATION_URI=\"$RECORD_URI\"" >> aggregator-config.env +echo "SERVICE_DECLARATION_CID=\"$RECORD_CID\"" >> aggregator-config.env + +echo "✓ Updated aggregator-config.env" +echo "" +echo "================================================" +echo "Setup Complete!" +echo "================================================" +echo "" +echo "Your aggregator is now registered with Coves!" +echo "" +echo "Next steps:" +echo "1. Wait a few seconds for Jetstream to index your service declaration" +echo "2. Verify your aggregator appears in the aggregators list" +echo "3. Community moderators can now authorize your aggregator" +echo "4. Once authorized, you can start posting to communities" +echo "" +echo "To test posting, use the Coves XRPC endpoint:" +echo " POST $COVES_INSTANCE_URL/xrpc/social.coves.community.post.create" +echo "" +echo "See docs/aggregators/SETUP_GUIDE.md for more information" diff --git a/scripts/aggregator-setup/README.md b/scripts/aggregator-setup/README.md new file mode 100644 index 0000000..e78bab0 --- /dev/null +++ b/scripts/aggregator-setup/README.md @@ -0,0 +1,252 @@ +# Aggregator Setup Scripts + +This directory contains scripts to help you set up and register your aggregator with Coves instances. + +## Overview + +Aggregators are automated services that post content to Coves communities. They are similar to Bluesky's feed generators and labelers. To use aggregators with Coves, you need to: + +1. Create a PDS account for your aggregator (gets you a DID) +2. Prove you own a domain via `.well-known/atproto-did` +3. Register with a Coves instance +4. Create a service declaration record + +These scripts automate this process for you. + +## Prerequisites + +- **Domain ownership**: You must own a domain where you can host the `.well-known/atproto-did` file +- **Web server**: Ability to serve static files over HTTPS +- **Tools**: `curl`, `jq` (for JSON processing) +- **Account**: Email address for creating the PDS account + +## Quick Start + +### Interactive Setup (Recommended) + +Run the scripts in order: + +```bash +# Make scripts executable +chmod +x *.sh + +# Step 1: Create PDS account +./1-create-pds-account.sh + +# Step 2: Generate .well-known file +./2-setup-wellknown.sh + +# Step 3: Register with Coves (after uploading .well-known) +./3-register-with-coves.sh + +# Step 4: Create service declaration +./4-create-service-declaration.sh +``` + +### Automated Setup Example + +For a reference implementation of automated setup, see the Kagi News aggregator at [aggregators/kagi-news/scripts/setup.sh](../../aggregators/kagi-news/scripts/setup.sh). + +The Kagi script shows how to automate all 4 steps (with the manual .well-known upload step in between). + +## Script Reference + +### 1-create-pds-account.sh + +**Purpose**: Creates a PDS account for your aggregator + +**Prompts for**: +- PDS URL (default: https://bsky.social) +- Handle (e.g., mynewsbot.bsky.social) +- Email +- Password + +**Outputs**: +- `aggregator-config.env` - Configuration file with DID and credentials +- Prints your DID and access tokens + +**Notes**: +- Keep the config file secure! It contains your credentials +- The PDS automatically generates a DID:PLC for you +- You can use any PDS service, not just bsky.social + +### 2-setup-wellknown.sh + +**Purpose**: Generates the `.well-known/atproto-did` file for domain verification + +**Prompts for**: +- Your domain (e.g., rss-bot.example.com) + +**Outputs**: +- `.well-known/atproto-did` - File containing your DID +- `nginx-example.conf` - Example nginx configuration +- `apache-example.conf` - Example Apache configuration + +**Manual step required**: +Upload the `.well-known` directory to your web server. The file must be accessible at: +``` +https://yourdomain.com/.well-known/atproto-did +``` + +**Verify it works**: +```bash +curl https://yourdomain.com/.well-known/atproto-did +# Should return your DID (e.g., did:plc:abc123...) +``` + +### 3-register-with-coves.sh + +**Purpose**: Registers your aggregator with a Coves instance + +**Prompts for**: +- Coves instance URL (default: https://api.coves.social) + +**Prerequisites**: +- `.well-known/atproto-did` must be accessible from your domain +- Scripts 1 and 2 must be completed + +**What it does**: +1. Verifies your `.well-known/atproto-did` is accessible +2. Calls `social.coves.aggregator.register` XRPC endpoint +3. Coves verifies domain ownership +4. Inserts your aggregator into the `users` table + +**Outputs**: +- Updates `aggregator-config.env` with Coves instance URL +- Prints registration confirmation + +### 4-create-service-declaration.sh + +**Purpose**: Creates the service declaration record in your repository + +**Prompts for**: +- Display name (e.g., "RSS News Aggregator") +- Description +- Source URL (GitHub repo, etc.) +- Maintainer DID (optional) + +**What it does**: +1. Creates a `social.coves.aggregator.service` record at `at://your-did/social.coves.aggregator.service/self` +2. Jetstream consumer will index this into the `aggregators` table +3. Communities can now discover and authorize your aggregator + +**Outputs**: +- Updates `aggregator-config.env` with record URI and CID +- Prints record details + +## Configuration File + +After running the scripts, you'll have an `aggregator-config.env` file with: + +```bash +AGGREGATOR_DID="did:plc:..." +AGGREGATOR_HANDLE="mynewsbot.bsky.social" +AGGREGATOR_PDS_URL="https://bsky.social" +AGGREGATOR_EMAIL="bot@example.com" +AGGREGATOR_PASSWORD="..." +AGGREGATOR_ACCESS_JWT="..." +AGGREGATOR_REFRESH_JWT="..." +AGGREGATOR_DOMAIN="rss-bot.example.com" +COVES_INSTANCE_URL="https://api.coves.social" +SERVICE_DECLARATION_URI="at://did:plc:.../social.coves.aggregator.service/self" +SERVICE_DECLARATION_CID="..." +``` + +**Use this in your aggregator code** to authenticate and post. + +## What Happens Next? + +After completing all 4 steps: + +1. **Your aggregator is registered** in the Coves instance's `users` table +2. **Your service declaration is indexed** in the `aggregators` table (takes a few seconds) +3. **Community moderators can now authorize** your aggregator for their communities +4. **Once authorized**, your aggregator can post to those communities + +## Creating an Authorization + +Authorizations are created by community moderators, not by aggregators. The moderator writes a `social.coves.aggregator.authorization` record to their community's repository. + +See `docs/aggregators/SETUP_GUIDE.md` for more information on the authorization process. + +## Posting to Communities + +Once authorized, your aggregator can post using: + +```bash +curl -X POST https://api.coves.social/xrpc/social.coves.community.post.create \ + -H "Authorization: Bearer $AGGREGATOR_ACCESS_JWT" \ + -H "Content-Type: application/json" \ + -d '{ + "communityDid": "did:plc:...", + "post": { + "text": "Your post content", + "createdAt": "2024-01-15T12:00:00Z" + } + }' +``` + +## Troubleshooting + +### Error: "DomainVerificationFailed" + +- Verify `.well-known/atproto-did` is accessible: `curl https://yourdomain.com/.well-known/atproto-did` +- Check the content matches your DID exactly (no extra whitespace) +- Ensure HTTPS is working (not HTTP) +- Check CORS headers if accessing from browser + +### Error: "AlreadyRegistered" + +- You've already registered this DID with this Coves instance +- This is safe to ignore if you're re-running the setup + +### Error: "DIDResolutionFailed" + +- Your DID might be invalid or not found in the PLC directory +- Verify your DID exists: `curl https://plc.directory/` +- Wait a few seconds and try again (PLC directory might be propagating) + +### Service declaration not appearing + +- Wait 5-10 seconds for Jetstream consumer to index it +- Check the Jetstream logs for errors +- Verify the record was created: Check your PDS at `at://your-did/social.coves.aggregator.service/self` + +## Example: Kagi News Aggregator + +For a complete reference implementation, see the Kagi News aggregator at `aggregators/kagi-news/`. + +The Kagi aggregator includes an automated setup script at [aggregators/kagi-news/scripts/setup.sh](../../aggregators/kagi-news/scripts/setup.sh) that demonstrates how to: + +- Automate the entire registration process +- Use environment variables for configuration +- Handle errors gracefully +- Integrate the setup into your aggregator project + +This shows how you can package scripts 1-4 into a single automated flow for your specific aggregator. + +## Security Notes + +- **Never commit `aggregator-config.env`** to version control +- Store credentials securely (use environment variables or secret management) +- Rotate access tokens regularly +- Use HTTPS for all API calls +- Validate community authorization before posting + +## More Information + +- [Aggregator Setup Guide](../../docs/aggregators/SETUP_GUIDE.md) +- [Aggregator PRD](../../docs/aggregators/PRD_AGGREGATORS.md) +- [atProto Identity Guide](../../ATPROTO_GUIDE.md) +- [Coves Communities PRD](../../docs/PRD_COMMUNITIES.md) + +## Support + +If you encounter issues: + +1. Check the troubleshooting section above +2. Review the full documentation in `docs/aggregators/` +3. Open an issue on GitHub with: + - Which script failed + - Error message + - Your domain (without credentials) -- 2.51.2