qcscrawl #
qcscrawl is a read-only, local-first ContentAPI archive crawler for QCS. It stores provider-visible users, content, activities, and messages in SQLite for offline browsing and FTS search.
Quick start #
# Build or run through Nix.
nix develop -c go build ./cmd/qcscrawl
# Create a private local config, then make the first complete archive.
./qcscrawl init
./qcscrawl sync --full
# Search and browse without network access.
./qcscrawl search 'sprite loading'
./qcscrawl messages --content 123
Install #
make check
make install
make install builds with Nix and installs qcscrawl to ~/.local/bin/qcscrawl.
Configuration #
qcscrawl init writes a config at the platform-specific default path. Override it with --config PATH or QCSCRAWL_CONFIG.
Anonymous access is the default. To use a bearer token, set the configured environment variable before running a command:
export QCSCRAWL_TOKEN='...'
qcscrawl sync
Command index #
All commands accept --config PATH before the command. Commands that support --json emit machine-readable JSON.
| Command | Purpose |
|---|---|
qcscrawl init |
Create private configuration and runtime directories. Does not crawl history. |
qcscrawl doctor [--json] |
Check config, provider status/about endpoints, and optional token validity. Read-only remotely. |
qcscrawl metadata [--json] |
Print CrawlKit control metadata for discovery. |
qcscrawl status [--json] |
Report local archive/config readiness. Does not create files. |
qcscrawl sync [--full | --repair] [--json] |
Archive or reconcile provider data. |
qcscrawl search QUERY [--include-deleted] [--limit N] [--json] |
Full-text search archived messages. |
qcscrawl messages [--content ID] [--user ID] [--include-deleted] [--limit N] [--json] |
Browse archived messages. |
qcscrawl content ID [--json] |
Show an archived content record. |
qcscrawl users [QUERY] [--json] |
List users, optionally filtering username text. |
qcscrawl sql QUERY [--json] |
Run one local read-only SQL query. |
Sync modes #
# Normal incremental sync: new IDs plus mutation/activity/user refresh passes.
qcscrawl sync
# Complete keyset scan from zero. It upserts; it does not erase the archive.
qcscrawl sync --full
# Complete reconciliation backstop for old edits/deletions. It never deletes
# a local row solely because a later provider response omits it.
qcscrawl sync --repair
Search an existing archive offline #
The browsing commands make no provider requests. Point them at a known config explicitly when working outside your usual machine/profile:
qcscrawl --config /path/to/config.toml search 'graphics' --limit 20 --json
qcscrawl --config /path/to/config.toml messages --user 42 --include-deleted
qcscrawl --config /path/to/config.toml content 123 --json
qcscrawl --config /path/to/config.toml users alice
qcscrawl --config /path/to/config.toml sql 'SELECT id, text FROM messages WHERE deleted = 0 LIMIT 10'
Development #
All Go work uses the included Nix flake:
nix develop -c gofmt -w $(find . -name '*.go')
nix develop -c go vet ./...
nix develop -c go test ./...
nix develop -c go test -race ./...
nix develop -c go build ./cmd/qcscrawl
The flake selects Go 1.27 when packaged by the selected nixpkgs revision, otherwise its default Go package. CrawlKit is pinned to the released v0.14.8 module.