A Discord bot that watches channels for links and builds a searchable archive with AI-generated summaries, auto-tags, web archive snapshots, and PDF captures.

Features #

Link detection — processes every URL posted in watched channels; Discord invites, already-archived URLs, and rate-limited users are handled gracefully
Content extraction — fetches pages, extracts readable text via Mozilla Readability + strip-tags; YouTube URLs use the transcript API instead
Image understanding — Gemini multimodal: describes images and transcribes visible text from attachments or linked image URLs
AI summaries — Gemini produces a one-line gloss and a 2–3 sentence genre-aware summary (fiction, technical, essay, social media, etc.)
Auto-tagging — Gemini suggests tags from a curated pool; new suggestions enter a proposal queue and are promoted once they exceed a use threshold (or immediately with !promote-tag)
Web archival — submits to archive.is and Wayback Machine, uses whichever succeeds first
PDF snapshots — full-page Playwright captures stored locally and optionally uploaded to the channel
Dual search — !search tries semantic vector search (pgvector) first, falls back to full-text (tsvector + GIN index) if no embedding is available
Daily digest — sends a Markdown summary of the day's archived links to the owner as a DM; windowed by last send time so no overlap or gap
SSRF protection — every outbound fetch validates the target IP before connecting; manual redirect loop re-validates every hop
Rate limiting — per-user token bucket (configurable, owner exempt); quiet 🐢 reaction on limit, no noisy error messages
Per-message flags — inline control over summary, tags, archive, and privacy
Role-based overrides — per-server, per-role, and per-channel config hierarchy; ignore: true on a role makes Archivist completely invisible to that user
Homelab-ready — uv-managed dependencies, K3s deployment.yaml, secrets never committed

Quick Start #

Prerequisites #

Python 3.12+ (managed by uv)
PostgreSQL 14+ with the pgvector extension available
(the bot installs it on first connect, but the server must permit CREATE EXTENSION vector)
Google Gemini API key (Google AI Studio)
Discord bot token with two Privileged Gateway Intents enabled in the
Discord Developer Portal → Bot → Privileged Gateway Intents:
- Message Content — required to read message text
- Server Members — required for role-based overrides (including the "Archivist ignore" role)
Permissions: Send Messages, Embed Links, Attach Files, Read Message History

Setup #

# Install dependencies
uv sync

# Configure
cp config/default.yaml config/config.yaml
# Edit config/config.yaml — set discord.token, gemini.api_key, and database.*

# (Optional) create the database if it doesn't exist
psql -h localhost -U postgres -c "CREATE DATABASE archivist;"

# Run
uv run python bot.py

The bot creates all tables and installs the vector extension on first connect.

Configuration #

All settings live in config/config.yaml (copy from config/default.yaml). Key fields:

Field	Description
`discord.token`	Bot token
`discord.watched_channels`	List of channel snowflakes to watch (empty = all channels)
`discord.owner_id`	Snowflake of the user who receives digests and can use owner commands
`discord.digest_time`	`HH:MM` time to send the daily digest
`discord.timezone`	IANA timezone name (e.g. `America/New_York`); blank = UTC
`gemini.api_key`	Gemini API key
`gemini.model`	Model name (default: `gemini-2.0-flash`)
`rate_limit.per_minute`	Max URLs processed per user per minute (default: 5)
`rate_limit.per_hour`	Max URLs processed per user per hour (default: 30)
`archive.snapshots.enabled`	Whether to take Playwright PDF snapshots

Environment variable overrides: DISCORD_TOKEN, GEMINI_API_KEY, DB_PASSWORD, DB_HOST.

Role-based overrides #

The overrides section in config.yaml lets you tune behavior per server, per role, and per channel. The hierarchy is: defaults < server < role < channel (channel wins).

overrides:
  servers:
    "123456789012345678":         # guild snowflake
      roles:
        "987654321098765432":     # "Archivist-ignore" role — completely silent
          ignore: true
        "111222333444555666":     # power-user role — always archive, no summary
          summary: false
          archive: true
  channels:
    "555666777888999000":         # never tag anything in this channel
      tags: false

Role overrides require the Server Members privileged intent.

Usage #

Post any link in a watched channel — Archivist handles the rest:

https://simonwillison.net/2024/Jan/26/strip-tags/

Per-message flags #

Add flags anywhere in your message to control this specific post:

Flag	Short	Effect
`-nosummary`	`-ns`	Skip AI summary
`-notags`	`-nt`	Skip auto-tagging
`-noarchive`	`-na`	Skip web archival
`-privacy`	`-p`	Skip summary AND tags

"Archivist, no" anywhere in the message skips all processing (legacy opt-out).

Owner commands #

Command	Effect
`!search <query>`	Search the archive (semantic → full-text fallback)
`!promote-tag <name>`	Promote a pending tag proposal to the curated pool

Architecture #

Discord message
    │
    ├─ parser.py      Extract URLs, flags, commentary
    │
    ├─ net.py         SSRF guard — validate every URL and redirect hop
    │
    ├─ fetcher.py     Fetch page → readability-lxml → strip-tags
    │   youtube.py    YouTube URL → transcript API
    │
    ├─ (concurrent)
    │   archiver.py   archive.is → Wayback Machine fallback
    │   ai.py         Gemini: classify+tag → summary → embedding
    │   snapshot.py   Playwright PDF capture
    │
    ├─ formatter.py   Build Discord message (2000-char safe)
    └─ db.py          PostgreSQL: save link, tags, embedding, meta

Database tables #

Table	Purpose
`archived_links`	One row per archived URL; includes embedding vector and generated `fts` column
`tags`	Curated tag pool fed back to Gemini
`tag_proposals`	LLM-suggested new tags with use counts; promoted to `tags` at threshold
`link_tags`	Many-to-many join between links and curated tags
`meta`	Key-value store for operational bookmarks (e.g. `last_digest_sent_at`)

Deployment #

See DEPLOYMENT.md for full instructions and DATABASE.md for database setup, pgvector installation, and embedding internals. Short version:

# 1. Create the secret (never commit real values)
kubectl create secret generic archivist-secrets \
  --namespace archivist \
  --from-literal=DISCORD_TOKEN='...' \
  --from-literal=GEMINI_API_KEY='...' \
  --from-literal=DB_PASSWORD='...'

# 2. Build and push the image
docker build -t your-registry/archivist:latest .
docker push your-registry/archivist:latest

# 3. Apply
kubectl apply -f deployment.yaml

The deployment.yaml uses the pgvector/pgvector:pg16 image for PostgreSQL so the vector extension is available without any manual setup.

Testing #

uv run pytest

61 tests covering the SSRF guard, message parser, formatter length limits, and YouTube URL patterns.

Clone this repository

Archivist 📚 #