diff --git a/docs/configuration.md b/docs/configuration.md index 000c62a..b6381d8 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -13,13 +13,15 @@ for what to set for a given workload, see [tuning](tuning.md). - everything is read once at startup, so changing a value means a restart. the ones [fixed when the database is created](#fixed-when-the-database-is-created) also need a fresh database. - the real environment wins over `.env`. `.env` lines are `KEY=value`, quotes optional, and the keys still need the `HYDRANT_` prefix. - embedded hydrant (`hydrant_start`, see [embedding](embedding.md)) takes the same keys from its json settings, so all of this applies there too. `ENABLE_DEBUG` and `DEBUG_PORT` are binary only. -- **a value that doesn't parse is ignored and the default is used, with no error and no log line.** that goes for numbers, bools, durations and the fixed-choice ones like `DATA_COMPRESSION` or `BACKFILL_STRATEGY`. hydrant logs the config it ended up with at startup (`hydrant configuration:`), so check that after changing something. -- numbers are plain decimal integers. `4GiB`, `1_000` or `1e9` fall back to the default. values aren't trimmed either, so `" 8"` does too. -- bools only take exactly `true` or `false`. `1`, `yes` and `TRUE` fall back to the default, so `ENABLE_FIREHOSE=0` leaves the firehose **on**. -- durations use [humantime](https://docs.rs/humantime/latest/humantime/fn.parse_duration.html) and need a unit: `3s`, `5min`, `2h`, `3d`. a bare `300` falls back to the default, only a bare `0` works without one. -- lists are comma separated, entries trimmed, empty entries dropped. a bad entry in `RELAY_HOSTS`, `SEED_HOSTS`, `CRAWLER_URLS`, `RATE_TIERS` or `TIER_RULES` is dropped with a warning in the log, and the rest of the list is used. -- a few are strict and stop startup with an error instead: `PLC_URL`, `HISTORY_TTL`, `FILTER_EXCLUDES` (every entry has to be a did), `API_BIND`, `API_SOCKET_MODE`, `API_TCP_MANAGEMENT`, and any non-empty `BACKFILL_PROXIES` in an indexer. -- a few crash hydrant at startup rather than erroring: `FIREHOSE_WORKERS=0`, `DB_WORKER_THREADS=0` and `DB_MAX_JOURNALING_SIZE_MB` below `64`. +- **a value that doesn't parse stops startup with an error** naming the variable and the value, like `invalid HYDRANT_FIREHOSE_WORKERS `4GiB`: want a whole number`. values are trimmed first, and an empty value counts as unset, so it gets the default. hydrant logs the config it ended up with at startup (`hydrant configuration:`). +- numbers are plain decimal integers, so `4GiB`, `1_000` or `1e9` are errors. +- bools take `true`, `false`, `1` or `0`, lowercase. `yes`, `on` or `TRUE` are errors. +- durations use [humantime](https://docs.rs/humantime/latest/humantime/fn.parse_duration.html) and need a unit: `3s`, `5min`, `2h`, `3d`. a bare `300` is an error, only a bare `0` works without one. +- the fixed-choice ones (`DATA_COMPRESSION`, `VERIFY_SIGNATURES`, `BACKFILL_STRATEGY`, ...) only take the values listed for them, lowercase. +- the ones that take `none` take it in any case. +- lists are comma separated, entries trimmed, empty entries dropped. a bad entry in `RELAY_HOSTS`, `SEED_HOSTS`, `CRAWLER_URLS`, `RATE_TIERS` or `TIER_RULES` is dropped with a warning in the log, and the rest of the list is used. `PLC_URL` and `FILTER_EXCLUDES` are stricter, a bad entry there is an error. +- values that parse but would crash hydrant or spin a loop are errors too: `FIREHOSE_WORKERS`, `DB_WORKER_THREADS`, `BACKFILL_CONCURRENCY_LIMIT` (indexer), `DB_EVENTS_MEMTABLE_SIZE_MB` and `DB_RECORDS_MEMTABLE_SIZE_MB` of `0`, a zero `CURSOR_SAVE_INTERVAL`, `SEED_POLL_INTERVAL` or `OFFLINE_HOST_RETRY_INTERVAL`, `DB_MAX_JOURNALING_SIZE_MB` below `64` and `DB_REPOS_MEMTABLE_SIZE_MB` below `2`. +- `BACKFILL_PROXIES` with anything in it is an error in an indexer. ## fixed when the database is created @@ -45,7 +47,7 @@ and one is sticky in a different way: `FULL_NETWORK=true` writes full mode into | `RUST_LOG` | `info,fjall=warn,lsm_tree=warn` | log filter directives (e.g., `debug`, `hydrant=trace`). [tracing env-filter syntax](https://docs.rs/tracing-subscriber/latest/tracing_subscriber/filter/struct.EnvFilter.html), bad directives are skipped. at `info` every firehose message already opens a log span, so `warn` is cheaper on a busy indexer. embedded: an empty string isn't treated as unset, so `""` means plain `info` with the storage engine's logs back on, and only the first `hydrant_start` in a process sets the filter, later ones keep it | | `API_BIND` | `0.0.0.0:3000,[::]:3000` | comma-separated list of `:` socket addresses and `unix:` sockets to bind the API server to. literal IPs only (hostnames not resolved). when both an ipv4 and ipv6 entry share the same port, the v6 listener is set to v6-only to avoid bind collision, and a lone `[::]:` listens dual-stack. unix sockets get `API_SOCKET_MODE`. a stale socket left by a crashed hydrant is replaced, but one that is still being listened on is not. the socket is set up in a short-lived `.hydrant-/` directory next to it and renamed into place, so its directory has to be writable and its path about 20 characters shorter than the os limit (104 on macOS, 108 on linux). `none` runs without the API, and an empty value is a startup error. embedded hydrant has no default and requires it | | `API_SOCKET_MODE` | `0600` | octal permissions for `unix:` sockets in `API_BIND`. owner-only by default since the API has no auth, `0660` also lets the socket's group in. anything that isn't octal up to `0777` is a startup error | -| `API_TCP_MANAGEMENT` | `false` | also serve the [management endpoints](getting-started.md#management-endpoints-keep-private) on tcp binds. by default they're only served on `unix:` binds, since they have no auth and the socket's permissions are what guards them, so with only tcp binds there is no management API. unlike most bools here, a value other than `true` or `false` is a startup error | +| `API_TCP_MANAGEMENT` | `false` | also serve the [management endpoints](getting-started.md#management-endpoints-keep-private) on tcp binds. by default they're only served on `unix:` binds, since they have no auth and the socket's permissions are what guards them, so with only tcp binds there is no management API. | | `GET_REPO_CONCURRENCY_LIMIT` | `2` | how many public `com.atproto.sync.getRepo` requests can build a repo at once. each one reads every record of the repo and rebuilds its tree in memory. a request past the limit gets a rate limited error straight away, it doesn't wait. the slot is released once the repo is built, before it's streamed to the client, so slow clients can keep more repos than this in memory. getRepo only works on databases that store every collection's record bodies (no `FILTER_COLLECTIONS`, `EPHEMERAL` or `ONLY_INDEX_LINKS`) | | `ENABLE_DEBUG` | `false` | enable debug endpoints. binary only | | `DEBUG_PORT` | first tcp `API_BIND` port + 1, or `3001` | port for debug endpoints (if enabled), on `127.0.0.1`. binary only | @@ -77,12 +79,12 @@ these set the database's stored filter, the same one `PATCH /filter` edits (see | `RELAY_HOST` | `wss://relay.fire.hose.cam/` (indexer), empty (relay) | URL of a single firehose source. takes the `pds::` prefix too. ignored whenever `RELAY_HOSTS` is set | | `RELAY_HOSTS` | | comma-separated list of firehose sources. if unset, falls back to `RELAY_HOST`, and if set to an empty string there are no configured sources at all, even with `RELAY_HOST` set. prefix a URL with `pds::` to mark it as a direct PDS connection (e.g. `pds::wss://pds.example.com`), which turns on host authority checks and `RATE_TIERS` pacing for it. bare URLs are treated as relays. each source is one websocket and one task, and its cursor is saved every `CURSOR_SAVE_INTERVAL`. configured sources may point at private or local addresses, unlike seeded ones. defaults to empty in relay mode, PDSes are expected to be seeded via `SEED_HOSTS` or the firehose management API | | `SEED_HOSTS` | `https://bsky.network` (relay) | comma-separated list of base URLs to call `com.atproto.sync.listHosts` on at startup and periodically. hydrant adds every non-banned host as a PDS firehose source. the first pass runs before any source connects, so a slow seed host delays startup. seeded hosts don't count toward `NEW_HOST_LIMIT`, and each one costs a websocket, a task and a cursor write per save, so memory and writes grow with the host count | -| `SEED_POLL_INTERVAL` | `30min` | humantime interval between completed `listHosts` polls. set to `none` (any case) to seed once at startup. `0s` doesn't disable it, it falls back to the default | -| `ENABLE_FIREHOSE` | `true` | whether to ingest relay subscriptions. `false` keeps the sources from connecting, but their tasks, the shard threads, seeding and the offline retry loop all still run. can also be toggled at runtime through `/ingestion`. only exactly `false` turns it off, so `0` or `no` leave it on | -| `FIREHOSE_WORKERS` | `8` (`24` full network) | number of firehose shards. messages go to a shard by a hash of their repo's did, so one repo's messages stay in order, and each shard is its own thread with a queue of 64 messages. an indexer runs two sets of shards, one to validate and one to write records, so `2n` threads, and a relay runs `n`. a shard thread blocks on identity lookups for repos it hasn't seen, so more shards means more of those in flight at once, but every commit that emits events still goes through one shared lock, so more shards don't add write throughput past that. backfill results also drain through the shards, so they run even with the firehose off. `0` crashes at startup | +| `SEED_POLL_INTERVAL` | `30min` | humantime interval between completed `listHosts` polls. set to `none` (any case) to seed once at startup. `0s` is an error | +| `ENABLE_FIREHOSE` | `true` | whether to ingest relay subscriptions. `false` keeps the sources from connecting, but their tasks, the shard threads, seeding and the offline retry loop all still run. can also be toggled at runtime through `/ingestion`. | +| `FIREHOSE_WORKERS` | `8` (`24` full network) | number of firehose shards. messages go to a shard by a hash of their repo's did, so one repo's messages stay in order, and each shard is its own thread with a queue of 64 messages. an indexer runs two sets of shards, one to validate and one to write records, so `2n` threads, and a relay runs `n`. a shard thread blocks on identity lookups for repos it hasn't seen, so more shards means more of those in flight at once, but every commit that emits events still goes through one shared lock, so more shards don't add write throughput past that. backfill results also drain through the shards, so they run even with the firehose off. at least `1` | | `FIREHOSE_MAX_FAILURES` | `15` | consecutive connect or stream failures before hydrant gives up on a source. the wait between tries doubles from 15 seconds and is capped at a minute, so `15` is about 14 minutes of retrying. failures only reset after a minute connected. a PDS that hits it is marked offline. given-up sources come back through `OFFLINE_HOST_RETRY_INTERVAL`. `0` and `1` both give up after the first failure | -| `OFFLINE_HOST_RETRY_INTERVAL` | `30min` | how often sources that were given up on get restarted, which also resets their failure count. exactly `none` (lowercase) disables it, anything else that isn't a duration means the default. don't set `0s`, that turns the retry loop into a hot loop that keeps restarting dead hosts | -| `CURSOR_SAVE_INTERVAL` | `3sec` | how often a background thread saves every firehose cursor, checkpoints counts and writes the journal buffer out to the os. it's a write to the os, not an fsync, so it protects against a crash of hydrant but not a power loss. after a crash hydrant re-reads up to this much of the firehose, which is cheap since already applied commits are skipped. every source's cursor is rewritten each tick, changed or not, which adds up on a relay with thousands of seeded sources. `0s` makes the thread spin, saving in a hot loop | +| `OFFLINE_HOST_RETRY_INTERVAL` | `30min` | how often sources that were given up on get restarted, which also resets their failure count. `none` (any case) disables it. `0s` is an error, it would restart dead hosts in a hot loop | +| `CURSOR_SAVE_INTERVAL` | `3sec` | how often a background thread saves every firehose cursor, checkpoints counts and writes the journal buffer out to the os. it's a write to the os, not an fsync, so it protects against a crash of hydrant but not a power loss. after a crash hydrant re-reads up to this much of the firehose, which is cheap since already applied commits are skipped. every source's cursor is rewritten each tick, changed or not, which adds up on a relay with thousands of seeded sources. `0s` is an error, it would save in a hot loop | ## identity & verification @@ -90,7 +92,7 @@ these set the database's stored filter, the same one `PATCH /filter` edits (see | :--- | :--- | :--- | | `PLC_URL` | `https://plc.wtf`, `https://plc.directory` (full network) | base URL(s) of the PLC directory, comma-separated. lookups rotate through them and move to the next one on a rate limit, a server error or a connection error. any invalid entry is a startup error. lookups refuse private and loopback addresses, so a PLC mirror on localhost won't work. each lookup can take up to 30 seconds and blocks the firehose shard that asked | | `IDENTITY_CACHE_SIZE` | `1000000` | max entries in the in-memory identity cache, an LRU of resolved did documents (pds endpoint, handle, signing key). the same number also caps two smaller caches the API fills (verified handles and identity reconcile cooldowns). a miss on the firehose path is a PLC lookup on that shard's thread, so a small cache on a full network firehose means a lookup on almost every commit. entries have no ttl, they leave on eviction or when hydrant invalidates them (a new account, a host mismatch, a signature failure). `0` doesn't turn it off, the cache rounds it up to a tiny minimum. starts empty on every start | -| `VERIFY_SIGNATURES` | `full` | signature verification level: `full`, `backfill-only`, or `none`. anything else means `full`. `full` checks every live commit's signature against the repo's signing key (an identity lookup on a cache miss), and also turns on MST inversion for every live commit, see `VERIFY_MST`. backfills are verified under `full` and `backfill-only`. relay builds have no backfill, so `backfill-only` is the same as `none` there. without it a relay or PDS can forge commits | +| `VERIFY_SIGNATURES` | `full` | signature verification level: `full`, `backfill-only`, or `none`. `full` checks every live commit's signature against the repo's signing key (an identity lookup on a cache miss), and also turns on MST inversion for every live commit, see `VERIFY_MST`. backfills are verified under `full` and `backfill-only`. relay builds have no backfill, so `backfill-only` is the same as `none` there. without it a relay or PDS can forge commits | | `VERIFY_MST` | `false` | run MST inversion on live commits: rebuild the commit's tree changes in reverse and check they land on the previous root, which copies the commit's blocks once more. **does nothing under `VERIFY_SIGNATURES=full`**, since full verification already runs it. it only matters with `backfill-only` or `none` | | `VERIFY_CIDS` | `true` | reject firehose and backfill CAR blocks whose content does not hash to their claimed CID, which costs a sha256 over every block. without this, signature verification does not cover the MST: a PDS or relay can forge nodes that map a key to a different record. record bodies are hashed before they are stored either way | | `REV_CLOCK_SKEW` | `300` | seconds a live commit's rev timestamp may be ahead of hydrant's clock before the commit is dropped. `0` drops commits from any PDS whose clock runs slightly fast, and a negative value drops nearly everything. too large lets a far-future rev through, and since later commits with lower revs count as stale, that repo's real commits get dropped until the clock catches up | @@ -100,13 +102,13 @@ these set the database's stored filter, the same one `PATCH /filter` edits (see | variable | default | description | | :--- | :--- | :--- | | `ENABLE_BACKFILL` | `true` | whether queued repos get backfilled. with `false` the queue still fills (from the crawler, the firehose and `/repos`) but nothing starts, and live commits for queued repos pile up on disk until they're backfilled. can also be toggled at runtime through `/ingestion` | -| `BACKFILL_CONCURRENCY_LIMIT` | `16` (`64` full network) | maximum number of concurrent backfill tasks. a task holds its slot for the whole repo: identity lookup, fetch, verification, the write, and the wait for its shard to apply live commits that arrived meanwhile. the queue is taken in random order, not first come first served. one PDS gets at most `PER_PDS_CONCURRENCY` or this minus one, whichever is lower, so another PDS always has room. every repo download also holds a thread from a shared pool of 512 that the database calls use too, so values in the hundreds start slowing everything else. `0` means nothing is ever backfilled | +| `BACKFILL_CONCURRENCY_LIMIT` | `16` (`64` full network) | maximum number of concurrent backfill tasks. a task holds its slot for the whole repo: identity lookup, fetch, verification, the write, and the wait for its shard to apply live commits that arrived meanwhile. the queue is taken in random order, not first come first served. one PDS gets at most `PER_PDS_CONCURRENCY` or this minus one, whichever is lower, so another PDS always has room. every repo download also holds a thread from a shared pool of 512 that the database calls use too, so values in the hundreds start slowing everything else. `0` is an error, turn backfill off with `ENABLE_BACKFILL=false` instead | | `BACKFILL_MEMORY_BYTES` | `4294967296` (4 GiB) | memory that unfiltered backfills may hold at once. each one counts its car's expected peak, about 6x the car, since it keeps every block, the records and the repo's write batch until it commits. a backfill that would go past it waits, except the oldest one running, which always goes ahead, so peak memory stays near this plus one repo. "unfiltered" means exactly `FILTER_COLLECTIONS` is empty, any collections at all (even in full network) means backfills stream and keep only what they want, so they don't count and this does nothing. a waiting backfill keeps its connection open and its `REPO_FETCH_TIMEOUT` running, so a long wait can turn into a timeout and a retry | | `BACKFILL_CHUNK_RECORDS` | `1000` | records a full backfill writes per step. each step is its own commit on the repo's indexer shard, so live events there wait at most one step behind a big repo, and the commit sequencer is held for one step at a time. an operator erase and a deleted account's wipe go in steps of this size too, an erase at most 4096 entries a step. `0` writes each repo in one step. a backfill stopped or crashed partway resumes from what its finished steps stored, but in ephemeral mode nothing is stored to compare against, so the redo emits the records of the finished steps again | | `BACKFILL_SPILL_BYTES` | `33554432` (32 MiB) | how big a whole car gets in memory before its backfill moves it to a file. a car in memory peaks at about 6x its size, so 32 MiB is about 200 MiB, while a spilled one holds about 64 bytes per block for its index plus one step: so a car of a few hundred MiB takes tens of MiB instead of GiBs. every spilled block is hashed against its cid as it lands and again when it's read back. `0` spills every whole car | | `BACKFILL_SPILL_DIR` | `spill` in the database directory | where spilled cars go. each file is unlinked as soon as it's made, so only the open backfill keeps it, and any `.car` file left in the directory is removed at start | | `BACKFILL_SPILLS` | `4` | how many backfills may spill at once. a spill always leaves 256 MiB free on its disk: it checks before it starts and again every 64 MiB it writes, and once less is left the backfill fails and is retried like a failed fetch, with nothing written. a backfill waiting for a slot holds what it has read so far in memory | -| `BACKFILL_STRATEGY` | `auto` | backfill strategy: `full`, `sparse-filter` or `auto`. anything else means `auto`. `full` always fetches the whole repo with `getRepo`. with `FILTER_COLLECTIONS` set it keeps only the matching records, but still downloads, hashes and decodes every block. `sparse-filter` instead walks just the parts of the repo's tree that can hold the filtered collections, fetching nodes with `getBlocks` in rounds, and falls back to `getRepo` if that fails (but not on a rate limit or connection error, which retries the repo later). `auto` is `sparse-filter`, except it switches to `getRepo` for shallow trees when the first probe didn't already cover what's needed. with no `FILTER_COLLECTIONS` both sparse modes act like `full` | +| `BACKFILL_STRATEGY` | `auto` | backfill strategy: `full`, `sparse-filter` or `auto`. `full` always fetches the whole repo with `getRepo`. with `FILTER_COLLECTIONS` set it keeps only the matching records, but still downloads, hashes and decodes every block. `sparse-filter` instead walks just the parts of the repo's tree that can hold the filtered collections, fetching nodes with `getBlocks` in rounds, and falls back to `getRepo` if that fails (but not on a rate limit or connection error, which retries the repo later). `auto` is `sparse-filter`, except it switches to `getRepo` for shallow trees when the first probe didn't already cover what's needed. with no `FILTER_COLLECTIONS` both sparse modes act like `full` | | `PER_PDS_CONCURRENCY` | `10` | max requests in flight to one PDS (per scheme, host and port), shared by backfill, crawler checks and PDS listings. a whole `getRepo` download holds one for its full length, and a sparse backfill can hold up to 4 at once. it also caps backfill tasks per PDS, see `BACKFILL_CONCURRENCY_LIMIT`, and a repo whose PDS is full gets put back and tried again later. `0` acts like `1` | | `MAX_CAR_BODY_BYTES` | `2147483648` (2 GiB) | largest response accepted from a PDS during backfill, measured after decompression. it covers `getRepo` and the sparse requests. a repo over it fails and retries on backoff (up to an hour apart) forever, it isn't skipped. for unfiltered backfills it bounds the worst case per task (about 6x this), for filtered ones it bounds the download, not memory | | `REPO_FETCH_TIMEOUT` | `5min` | total time allowed for each backfill request, from connecting until the whole body is read. applies to every `getRepo` download and every sparse request. a slow PDS that can't send a big repo in time never finishes it, while a long timeout lets a stalled PDS hold a backfill slot that long | @@ -117,7 +119,7 @@ these set the database's stored filter, the same one `PATCH /filter` edits (see | variable | default | description | | :--- | :--- | :--- | | `CRAWLER_URLS` | relay hosts as `list_repos` (full network), `by_collection::https://lightrail.microcosm.blue` (filter) | comma-separated list of `[mode::]url` crawler sources. mode is `list_repos`, `relay_first`, or `by_collection` (dashes work too), bare URLs use the default mode (`list_repos` with full network, `by_collection` otherwise). `list_repos` pages a relay's repo list, `by_collection` asks for repos holding each `FILTER_SIGNALS` collection and does nothing without signals. `relay_first` is an explicit coverage-over-moderation opt-in. an entry with an unknown mode or bad URL is dropped with a warning. set to empty string to disable the configured sources. sources added through `/crawler/sources` are saved in the database and still start | -| `ENABLE_CRAWLER` | `true` if full network or sources configured | whether to actively query the network for unknown repositories. since `CRAWLER_URLS` has a default in every mode, unset means on unless `CRAWLER_URLS` is empty. a value other than `true` or `false` counts as unset. with `false` the crawler tasks still exist and wait. can also be toggled at runtime through `/ingestion` | +| `ENABLE_CRAWLER` | `true` if full network or sources configured | whether to actively query the network for unknown repositories. since `CRAWLER_URLS` has a default in every mode, unset means on unless `CRAWLER_URLS` is empty. leave it unset (or empty) for that default. with `false` the crawler tasks still exist and wait. can also be toggled at runtime through `/ingestion` | | `CRAWLER_MAX_PENDING_REPOS` | `2000` (`25000` full network) | persistent discovery lookahead retained in the pending queue before the crawler pauses. this does not increase active backfill task concurrency. it's checked before each page of discovered repos is queued, so the queue can go over by one page (up to 2000 repos), and while paused the crawler rechecks every 5 seconds | | `CRAWLER_RESUME_PENDING_REPOS` | `1000` (`12500` full network) | pending-repo count at which discovery resumes. not checked against the max, and a value at or above it just resumes as soon as the queue is back under the max | @@ -139,13 +141,13 @@ the storage engine's knobs. the memtable, bloom filter and data compression ones | variable | default | description | | :--- | :--- | :--- | | `CACHE_SIZE` | `256` | the block cache, in MB, but **hydrant only gives it half**, so `256` is a 128 MB cache, and the startup config dump still prints `256 mb`. one cache shared by every keyspace, holding decompressed blocks. a miss reads the block from disk again and decompresses it again. some index and filter blocks are pinned in memory outside the cache, so they don't count against it. can change between restarts | -| `DATA_COMPRESSION` | `zstd` | compression for stored data blocks: `zstd`, `lz4` or `none` (lowercase, anything else means `zstd`). covers the record, history, repo, event and backlink keyspaces, but the top levels of some are left uncompressed on purpose. smaller tables mean less disk and less compaction I/O, at the cost of CPU on every flush, compaction and cache miss. fixed when the database is created | +| `DATA_COMPRESSION` | `zstd` | compression for stored data blocks: `zstd`, `lz4` or `none`. covers the record, history, repo, event and backlink keyspaces, but the top levels of some are left uncompressed on purpose. smaller tables mean less disk and less compaction I/O, at the cost of CPU on every flush, compaction and cache miss. fixed when the database is created | | `JOURNAL_COMPRESSION` | `lz4` | compression for the write-ahead journal: `lz4`, `zstd` or `none`. only values of 4 KiB and up get compressed, so most writes (repo state, counts, cursors, typical records) go in raw whatever this says. can change between restarts | -| `DB_WORKER_THREADS` | `4` (`8` full network) | background threads that flush memtables and compact. with more than one, the first thread never compacts so it stays free for flushes, which leaves `n-1` for compaction. with one, a long compaction holds up flushes, and writes stall once 4 memtables wait to flush. `POST /db/compact` doesn't use these threads, it compacts every keyspace at once. `0` crashes at startup | -| `DB_MAX_JOURNALING_SIZE_MB` | `400` (`1024` full network) | soft cap on journal files on disk. when it's over, keyspaces still holding the oldest journal get flushed early so it can go. it's only checked when a flush happens and the active journal is past about 64 MB, so it isn't a hard limit. unflushed journal is replayed into memory on every open, so this bounds startup time too. below `64` crashes at startup | -| `DB_REPOS_MEMTABLE_SIZE_MB` | `16` (`48` full network) | in-memory write buffer for the `repos` keyspace, and half of it (rounded down to whole MB) for `repo_metadata`. each keyspace can hold its active memtable plus up to 4 waiting to flush before writes stop, so memory per keyspace can reach 5x this. bigger means fewer, larger flushes and more to replay at startup. don't set `1`, it gives `repo_metadata` a zero size memtable that flushes on every write and soon stalls writes. fixed when the database is created | -| `DB_EVENTS_MEMTABLE_SIZE_MB` | `32` (`192` full network) | write buffer for the stream events keyspace (and `jetstream_events` and `relay_events` when built). event ids only go up, so flushed tables are never rewritten and this mostly sets their size and count. in ephemeral mode and in relays the ttl worker flushes it about once a minute anyway, so it only matters past a minute's worth of events. fixed when the database is created | -| `DB_RECORDS_MEMTABLE_SIZE_MB` | `20` (`128` full network) | write buffer for the record heads keyspace, and also for `backlinks` when built with that feature. record writes look up the current head first, so recent records found in the memtable skip a disk read. no effect in relay builds. fixed when the database is created | +| `DB_WORKER_THREADS` | `4` (`8` full network) | background threads that flush memtables and compact. with more than one, the first thread never compacts so it stays free for flushes, which leaves `n-1` for compaction. with one, a long compaction holds up flushes, and writes stall once 4 memtables wait to flush. `POST /db/compact` doesn't use these threads, it compacts every keyspace at once. at least `1` | +| `DB_MAX_JOURNALING_SIZE_MB` | `400` (`1024` full network) | soft cap on journal files on disk. when it's over, keyspaces still holding the oldest journal get flushed early so it can go. it's only checked when a flush happens and the active journal is past about 64 MB, so it isn't a hard limit. unflushed journal is replayed into memory on every open, so this bounds startup time too. at least `64` | +| `DB_REPOS_MEMTABLE_SIZE_MB` | `16` (`48` full network) | in-memory write buffer for the `repos` keyspace, and half of it (rounded down to whole MB) for `repo_metadata`. each keyspace can hold its active memtable plus up to 4 waiting to flush before writes stop, so memory per keyspace can reach 5x this. bigger means fewer, larger flushes and more to replay at startup. at least `2`, so `repo_metadata` doesn't get a zero size one. fixed when the database is created | +| `DB_EVENTS_MEMTABLE_SIZE_MB` | `32` (`192` full network) | write buffer for the stream events keyspace (and `jetstream_events` and `relay_events` when built). event ids only go up, so flushed tables are never rewritten and this mostly sets their size and count. in ephemeral mode and in relays the ttl worker flushes it about once a minute anyway, so it only matters past a minute's worth of events. at least `1`. fixed when the database is created | +| `DB_RECORDS_MEMTABLE_SIZE_MB` | `20` (`128` full network) | write buffer for the record heads keyspace, and also for `backlinks` when built with that feature. record writes look up the current head first, so recent records found in the memtable skip a disk read. no effect in relay builds. at least `1`. fixed when the database is created | | `DB_RECORDS_BLOOM_FILTERS` | `false` | build a bloom filter on the bottom level of the record heads keyspace too. `false` doesn't mean no bloom filters, the upper levels always have them, only the bottom level, which holds most of the data, goes without. without it, every lookup of a record that isn't stored yet reads index and data blocks from that level, and every first write of a record does that lookup, so backfilling new repos into a big database pays for it. `true` costs about 10 bits per record in disk and cache. fixed when the database is created | | `MAINTENANCE_TIMEOUT` | `6h` | how long `POST /db/compact` and `POST /db/train` keep ingestion paused before giving up. `0` or `0s` means no limit. on timeout ingestion resumes, but the compaction or training **keeps running** in the background, since it can't be cancelled, and a second compact request can start on top of it | diff --git a/ffi/src/lib.rs b/ffi/src/lib.rs index 9fec30e..57e73f1 100644 --- a/ffi/src/lib.rs +++ b/ffi/src/lib.rs @@ -306,3 +306,36 @@ unsafe fn set_err(err: *mut *mut c_char, msg: String) { // SAFETY: checked non-null above, the caller promises it's writable unsafe { *err = msg.into_raw() }; } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn bad_settings_are_errors_naming_the_variable() { + for (key, value) in [ + ("HYDRANT_ENABLE_FIREHOSE", "no"), + ("HYDRANT_FIREHOSE_WORKERS", "0"), + ("HYDRANT_CURSOR_SAVE_INTERVAL", "300"), + ] { + // if a bad value ever gets through, hydrant starts on a throwaway database + let db = std::env::temp_dir().join(format!("hydrant-ffi-test-{}", std::process::id())); + let db = db.to_string_lossy(); + let settings = HashMap::from([ + ("HYDRANT_API_BIND", "none"), + ("HYDRANT_DATABASE_PATH", &db), + ("HYDRANT_RELAY_HOSTS", ""), + ("HYDRANT_CRAWLER_URLS", ""), + ("HYDRANT_ENABLE_CRAWLER", "false"), + (key, value), + ]); + let Err(err) = start(&serde_json::to_string(&settings).unwrap()) else { + panic!("{key}={value:?} was accepted"); + }; + assert!( + err.contains(key) && err.contains(value), + "{key}={value:?}: {err}" + ); + } + } +} diff --git a/src/config.rs b/src/config.rs index b57c3e7..5f091ca 100644 --- a/src/config.rs +++ b/src/config.rs @@ -455,6 +455,8 @@ impl Config { /// validates cross-field invariants for this configuration. pub fn validate(&self) -> Result<()> { + self.validate_ranges()?; + #[cfg(all(feature = "indexer", not(feature = "indexer_stream")))] if self.ephemeral { return Err(miette::miette!( @@ -500,6 +502,72 @@ impl Config { Ok(()) } + + /// values that parse but would crash hydrant or spin a loop + fn validate_ranges(&self) -> Result<()> { + if self.firehose_workers == 0 { + miette::bail!("invalid HYDRANT_FIREHOSE_WORKERS `0`: want at least 1 firehose shard"); + } + if self.cursor_save_interval.is_zero() { + miette::bail!( + "invalid HYDRANT_CURSOR_SAVE_INTERVAL `0s`: the save thread would spin, want a duration like 3s" + ); + } + if self.seed_poll_interval.is_some_and(|d| d.is_zero()) { + miette::bail!( + help = "set it to none to only seed at startup", + "invalid HYDRANT_SEED_POLL_INTERVAL `0s`: want a duration like 30min" + ); + } + if self + .offline_host_retry_interval + .is_some_and(|d| d.is_zero()) + { + miette::bail!( + help = "set it to none to never retry offline hosts", + "invalid HYDRANT_OFFLINE_HOST_RETRY_INTERVAL `0s`: the retry loop would spin, want a duration like 30min" + ); + } + if self.db_worker_threads == 0 { + miette::bail!( + "invalid HYDRANT_DB_WORKER_THREADS `0`: the database needs at least 1 thread to flush" + ); + } + if self.db_max_journaling_size_mb < 64 { + miette::bail!( + "invalid HYDRANT_DB_MAX_JOURNALING_SIZE_MB `{}`: the storage engine wants at least 64", + self.db_max_journaling_size_mb + ); + } + if self.db_repos_memtable_size_mb < 2 { + miette::bail!( + "invalid HYDRANT_DB_REPOS_MEMTABLE_SIZE_MB `{}`: want at least 2, repo_metadata gets half of it", + self.db_repos_memtable_size_mb + ); + } + for (key, mb) in [ + ( + "HYDRANT_DB_EVENTS_MEMTABLE_SIZE_MB", + self.db_events_memtable_size_mb, + ), + ( + "HYDRANT_DB_RECORDS_MEMTABLE_SIZE_MB", + self.db_records_memtable_size_mb, + ), + ] { + if mb == 0 { + miette::bail!("invalid {key} `0`: want at least 1"); + } + } + #[cfg(feature = "indexer")] + if self.backfill_concurrency_limit == 0 { + miette::bail!( + help = "set HYDRANT_ENABLE_BACKFILL=false to turn backfill off", + "invalid HYDRANT_BACKFILL_CONCURRENCY_LIMIT `0`: want at least 1" + ); + } + Ok(()) + } } #[cfg(test)] diff --git a/src/config/env.rs b/src/config/env.rs index 076c72a..6c0fe88 100644 --- a/src/config/env.rs +++ b/src/config/env.rs @@ -5,30 +5,131 @@ use std::time::Duration; use url::Url; use super::Config; -use super::types::{CrawlerMode, CrawlerSource, FirehoseSource, RateTier}; +use super::types::{ + BackfillStrategy, Compression, CrawlerMode, CrawlerSource, FirehoseSource, RateTier, + SignatureVerification, +}; use crate::pds_meta::{TierPolicy, TierRule}; /// this is for internal use only, please don't use this macro. #[doc(hidden)] #[macro_export] macro_rules! __cfg { - (@val $lookup:ident, $key:expr) => { - $lookup(concat!("HYDRANT_", $key)) - }; - ($lookup:ident, $key:expr, $default:expr, sec) => { - $crate::__cfg!(@val $lookup, $key) - .and_then(|s| humantime::parse_duration(&s).ok()) - .unwrap_or($default) - }; - ($lookup:ident, $key:expr, $default:expr) => { - $crate::__cfg!(@val $lookup, $key) - .and_then(|s| s.parse().ok()) - .unwrap_or($default.to_owned()) - .into() + ($lookup:ident, $key:expr, $default:expr $(, sec)?) => { + $crate::config::env::read(&$lookup, concat!("HYDRANT_", $key), $default.to_owned())? }; } use crate::__cfg as cfg; +/// a type one `HYDRANT_*` variable can hold. +#[doc(hidden)] +pub trait EnvValue: Sized { + /// what a valid value looks like, for the error + const EXPECTED: &'static str; + fn parse_env(s: &str) -> Option; +} + +impl EnvValue for bool { + const EXPECTED: &'static str = "true, false, 1 or 0"; + fn parse_env(s: &str) -> Option { + match s { + "true" | "1" => Some(true), + "false" | "0" => Some(false), + _ => None, + } + } +} + +macro_rules! whole_number { + ($($t:ty),*) => {$( + impl EnvValue for $t { + const EXPECTED: &'static str = "a whole number"; + fn parse_env(s: &str) -> Option { + s.parse().ok() + } + } + )*}; +} +whole_number!(u16, u32, u64, usize, i64); + +impl EnvValue for Duration { + const EXPECTED: &'static str = "a duration with a unit, like 30s or 5min"; + fn parse_env(s: &str) -> Option { + humantime::parse_duration(s).ok() + } +} + +impl EnvValue for PathBuf { + const EXPECTED: &'static str = "a path"; + fn parse_env(s: &str) -> Option { + Some(PathBuf::from(s)) + } +} + +impl EnvValue for Compression { + const EXPECTED: &'static str = "zstd, lz4 or none"; + fn parse_env(s: &str) -> Option { + s.parse().ok() + } +} + +impl EnvValue for SignatureVerification { + const EXPECTED: &'static str = "full, backfill-only or none"; + fn parse_env(s: &str) -> Option { + s.parse().ok() + } +} + +impl EnvValue for BackfillStrategy { + const EXPECTED: &'static str = "auto, full or sparse-filter"; + fn parse_env(s: &str) -> Option { + s.parse().ok() + } +} + +fn invalid(key: &str, raw: &str, expected: &str) -> miette::Report { + miette::miette!("invalid {key} `{raw}`: want {expected}") +} + +/// reads `key`, `None` when it's unset or empty. a value that doesn't parse is an error, +/// since quietly running on the default hides the typo until something goes wrong +fn read_optional( + lookup: &impl Fn(&str) -> Option, + key: &str, +) -> Result> { + let Some(raw) = lookup(key) else { + return Ok(None); + }; + let value = raw.trim(); + if value.is_empty() { + return Ok(None); + } + T::parse_env(value) + .map(Some) + .ok_or_else(|| invalid(key, &raw, T::EXPECTED)) +} + +#[doc(hidden)] +pub fn read( + lookup: &impl Fn(&str) -> Option, + key: &str, + default: T, +) -> Result { + Ok(read_optional(lookup, key)?.unwrap_or(default)) +} + +/// like [`read`], but `none` (any case) turns it off +fn read_or_none( + lookup: &impl Fn(&str) -> Option, + key: &str, + default: Option, +) -> Result> { + match lookup(key) { + Some(raw) if raw.trim().eq_ignore_ascii_case("none") => Ok(None), + _ => Ok(read_optional(lookup, key)?.or(default)), + } +} + fn dotenv_assignments(contents: &str) -> impl Iterator { contents.lines().filter_map(|line| { let line = line.trim(); @@ -60,28 +161,6 @@ pub fn load_dotenv() { } } -fn parse_seed_poll_interval_value( - value: Option<&str>, - default: Option, -) -> Option { - match value.map(str::trim) { - Some(value) if value.eq_ignore_ascii_case("none") => None, - Some(value) => humantime::parse_duration(value) - .ok() - .filter(|duration| !duration.is_zero()) - .or(default), - None => default, - } -} - -fn parse_new_host_limit_value(value: Option<&str>, default: Option) -> Option { - match value.map(str::trim) { - Some(value) if value.eq_ignore_ascii_case("none") => None, - Some(value) => value.parse().ok().or(default), - None => default, - } -} - impl Config { /// reads and builds the config from environment variables, loading `.env` first if present. pub fn from_env() -> Result { @@ -120,10 +199,17 @@ impl Config { .collect(), // HYDRANT_RELAY_HOSTS explicitly set to "" Some(_) => vec![], - // not set at all, fall back to RELAY_HOST (bare URL, no pds:: prefix support here) + // not set at all, fall back to RELAY_HOST None => match lookup("HYDRANT_RELAY_HOST") { Some(s) if !s.trim().is_empty() => { - FirehoseSource::parse(s.trim()).into_iter().collect() + let source = FirehoseSource::parse(s.trim()).ok_or_else(|| { + invalid( + "HYDRANT_RELAY_HOST", + &s, + "a url, optionally with a pds:: prefix", + ) + })?; + vec![source] } _ => defaults.relays.clone(), }, @@ -172,9 +258,7 @@ impl Config { "BACKFILL_SPILL_BYTES", defaults.backfill_spill_bytes ); - let backfill_spill_dir = lookup("HYDRANT_BACKFILL_SPILL_DIR") - .filter(|dir| !dir.trim().is_empty()) - .map(|dir| PathBuf::from(dir.trim())); + let backfill_spill_dir = read_optional(&lookup, "HYDRANT_BACKFILL_SPILL_DIR")?; let backfill_spills = cfg!(lookup, "BACKFILL_SPILLS", defaults.backfill_spills); let ephemeral: bool = cfg!(lookup, "EPHEMERAL", defaults.ephemeral); @@ -197,7 +281,7 @@ impl Config { let rev_clock_skew_secs: i64 = cfg!(lookup, "REV_CLOCK_SKEW", defaults.rev_clock_skew_secs); let enable_firehose = cfg!(lookup, "ENABLE_FIREHOSE", defaults.enable_firehose); let enable_backfill = cfg!(lookup, "ENABLE_BACKFILL", defaults.enable_backfill); - let enable_crawler = lookup("HYDRANT_ENABLE_CRAWLER").and_then(|s| s.parse().ok()); + let enable_crawler = read_optional(&lookup, "HYDRANT_ENABLE_CRAWLER")?; let backfill_concurrency_limit = cfg!( lookup, @@ -323,23 +407,18 @@ impl Config { ); let verify_cids: bool = cfg!(lookup, "VERIFY_CIDS", defaults.verify_cids); let only_index_links: bool = cfg!(lookup, "ONLY_INDEX_LINKS", defaults.only_index_links); - let max_pds_added_per_day = parse_new_host_limit_value( - lookup("HYDRANT_NEW_HOST_LIMIT").as_deref(), - defaults.new_host_limit, - ); - let seed_poll_interval = parse_seed_poll_interval_value( - lookup("HYDRANT_SEED_POLL_INTERVAL").as_deref(), + let max_pds_added_per_day = + read_or_none(&lookup, "HYDRANT_NEW_HOST_LIMIT", defaults.new_host_limit)?; + let seed_poll_interval = read_or_none( + &lookup, + "HYDRANT_SEED_POLL_INTERVAL", defaults.seed_poll_interval, - ); - - let offline_retry_interval: Option = - match lookup("HYDRANT_OFFLINE_HOST_RETRY_INTERVAL").as_deref() { - None => defaults.offline_host_retry_interval, - Some("none") => None, - Some(s) => humantime::parse_duration(s) - .ok() - .or(defaults.offline_host_retry_interval), - }; + )?; + let offline_retry_interval = read_or_none( + &lookup, + "HYDRANT_OFFLINE_HOST_RETRY_INTERVAL", + defaults.offline_host_retry_interval, + )?; // start with built-in tier definitions, then layer in any env-defined overrides. // format: HYDRANT_RATE_TIERS=name:base/mul/hourly/daily,... @@ -391,7 +470,9 @@ impl Config { continue; } let Some((pattern_str, tier_name)) = entry.split_once(':') else { - tracing::warn!("ignoring invalid tier rule '{entry}': expected pattern:tier format"); + tracing::warn!( + "ignoring invalid tier rule '{entry}': expected pattern:tier format" + ); continue; }; let pattern_str = pattern_str.trim(); @@ -404,9 +485,9 @@ impl Config { }); tier_rules.push((pattern_str.to_string(), tier_name.to_string())); } - Err(e) => tracing::warn!( - "ignoring invalid tier rule pattern '{pattern_str}': {e}" - ), + Err(e) => { + tracing::warn!("ignoring invalid tier rule pattern '{pattern_str}': {e}") + } } } } @@ -528,36 +609,6 @@ mod tests { ("SINGLE", "three words"), ] ); - - assert_eq!(parse_new_host_limit_value(None, Some(10)), Some(10)); - assert_eq!(parse_new_host_limit_value(Some(" 42 "), Some(10)), Some(42)); - assert_eq!(parse_new_host_limit_value(Some("NoNe"), Some(10)), None); - assert_eq!( - parse_new_host_limit_value(Some("invalid"), Some(10)), - Some(10) - ); - assert_eq!(parse_new_host_limit_value(Some("invalid"), None), None); - let default_interval = Some(Duration::from_secs(1800)); - assert_eq!( - parse_seed_poll_interval_value(None, default_interval), - default_interval - ); - assert_eq!( - parse_seed_poll_interval_value(Some("30min"), None), - default_interval - ); - assert_eq!( - parse_seed_poll_interval_value(Some("none"), default_interval), - None - ); - assert_eq!( - parse_seed_poll_interval_value(Some("0s"), default_interval), - default_interval - ); - assert_eq!( - parse_seed_poll_interval_value(Some("invalid"), default_interval), - default_interval - ); } #[test] @@ -601,4 +652,134 @@ mod tests { .unwrap_err(); assert!(err.to_string().contains("HYDRANT_HISTORY_TTL"), "{err}"); } + + fn from_pairs(pairs: &[(&str, &str)]) -> Result { + Config::from_lookup(|key| { + pairs + .iter() + .find(|(k, _)| *k == key) + .map(|(_, v)| v.to_string()) + }) + } + + #[test] + fn from_lookup_accepts_good_values() { + for (key, value) in [ + ("HYDRANT_FULL_NETWORK", "true"), + ("HYDRANT_FULL_NETWORK", "0"), + ("HYDRANT_ENABLE_FIREHOSE", "false"), + ("HYDRANT_ENABLE_FIREHOSE", "1"), + ("HYDRANT_ENABLE_BACKFILL", " true "), + ("HYDRANT_ENABLE_CRAWLER", "1"), + ("HYDRANT_VERIFY_CIDS", "false"), + ("HYDRANT_FIREHOSE_WORKERS", "24"), + ("HYDRANT_FIREHOSE_WORKERS", ""), + ("HYDRANT_FIREHOSE_MAX_FAILURES", "1"), + ("HYDRANT_CRAWLER_MAX_PENDING_REPOS", "2"), + ("HYDRANT_CRAWLER_RESUME_PENDING_REPOS", "1"), + ("HYDRANT_REV_CLOCK_SKEW", "-5"), + ("HYDRANT_CACHE_SIZE", "128"), + ("HYDRANT_CURSOR_SAVE_INTERVAL", "3sec"), + ("HYDRANT_CURSOR_SAVE_INTERVAL", "1s"), + ("HYDRANT_CURSOR_SAVE_INTERVAL", "500ms"), + ("HYDRANT_EPHEMERAL_TTL", "60min"), + ("HYDRANT_EPHEMERAL_TTL", "3d"), + ("HYDRANT_MAINTENANCE_TIMEOUT", "0"), + ("HYDRANT_MAINTENANCE_TIMEOUT", "0s"), + ("HYDRANT_MAINTENANCE_TIMEOUT", "6h"), + ("HYDRANT_STREAM_REPLAY_CHUNK_PAUSE", "2ms"), + ("HYDRANT_STREAM_SEND_TIMEOUT", "0s"), + ("HYDRANT_SEED_POLL_INTERVAL", "none"), + ("HYDRANT_SEED_POLL_INTERVAL", "30min"), + ("HYDRANT_OFFLINE_HOST_RETRY_INTERVAL", "None"), + ("HYDRANT_OFFLINE_HOST_RETRY_INTERVAL", "30min"), + ("HYDRANT_NEW_HOST_LIMIT", "NoNe"), + ("HYDRANT_NEW_HOST_LIMIT", " 42 "), + ("HYDRANT_DATA_COMPRESSION", "lz4"), + ("HYDRANT_JOURNAL_COMPRESSION", "none"), + ("HYDRANT_VERIFY_SIGNATURES", "backfill-only"), + ("HYDRANT_BACKFILL_STRATEGY", "sparse-filter"), + ("HYDRANT_DB_WORKER_THREADS", "1"), + ("HYDRANT_DB_MAX_JOURNALING_SIZE_MB", "64"), + ("HYDRANT_DB_REPOS_MEMTABLE_SIZE_MB", "2"), + ("HYDRANT_DB_EVENTS_MEMTABLE_SIZE_MB", "1"), + ("HYDRANT_RELAY_HOST", "wss://relay.example.com/"), + ("HYDRANT_RELAY_HOST", "pds::wss://pds.example.com"), + ("HYDRANT_RATE_TIERS", "custom:1/1/1/1/0"), + ("HYDRANT_TIER_RULES", "127.0.0.*:custom"), + ] { + if let Err(e) = from_pairs(&[(key, value)]) { + panic!("{key}={value:?} was refused: {e}"); + } + } + + let config = from_pairs(&[ + ("HYDRANT_ENABLE_FIREHOSE", "0"), + ("HYDRANT_ENABLE_CRAWLER", "1"), + ("HYDRANT_FIREHOSE_WORKERS", ""), + ("HYDRANT_NEW_HOST_LIMIT", " 42 "), + ("HYDRANT_SEED_POLL_INTERVAL", "NONE"), + ("HYDRANT_MAINTENANCE_TIMEOUT", "0"), + ]) + .unwrap(); + assert!(!config.enable_firehose); + assert_eq!(config.enable_crawler, Some(true)); + assert_eq!( + config.firehose_workers, + from_pairs(&[]).unwrap().firehose_workers + ); + assert_eq!(config.new_host_limit, Some(42)); + assert_eq!(config.seed_poll_interval, None); + assert_eq!(config.maintenance_timeout, Duration::ZERO); + } + + #[test] + fn from_lookup_refuses_bad_values() { + let mut bad = vec![ + ("HYDRANT_FULL_NETWORK", "yes"), + ("HYDRANT_ENABLE_FIREHOSE", "no"), + ("HYDRANT_ENABLE_FIREHOSE", "TRUE"), + ("HYDRANT_ENABLE_CRAWLER", "on"), + ("HYDRANT_ENABLE_BACKLINKS", "y"), + ("HYDRANT_FIREHOSE_WORKERS", "4GiB"), + ("HYDRANT_FIREHOSE_WORKERS", "1_000"), + ("HYDRANT_FIREHOSE_WORKERS", "-1"), + ("HYDRANT_FIREHOSE_WORKERS", "0"), + ("HYDRANT_CACHE_SIZE", "1e9"), + ("HYDRANT_REV_CLOCK_SKEW", "5m"), + ("HYDRANT_CURSOR_SAVE_INTERVAL", "300"), + ("HYDRANT_CURSOR_SAVE_INTERVAL", "0s"), + ("HYDRANT_CURSOR_SAVE_INTERVAL", "0"), + ("HYDRANT_MAINTENANCE_TIMEOUT", "300"), + ("HYDRANT_SEED_POLL_INTERVAL", "0s"), + ("HYDRANT_SEED_POLL_INTERVAL", "never"), + ("HYDRANT_OFFLINE_HOST_RETRY_INTERVAL", "0s"), + ("HYDRANT_OFFLINE_HOST_RETRY_INTERVAL", "soon"), + ("HYDRANT_NEW_HOST_LIMIT", "lots"), + ("HYDRANT_DATA_COMPRESSION", "gzip"), + ("HYDRANT_JOURNAL_COMPRESSION", "LZ4"), + ("HYDRANT_VERIFY_SIGNATURES", "yes"), + ("HYDRANT_BACKFILL_STRATEGY", "fast"), + ("HYDRANT_DB_WORKER_THREADS", "0"), + ("HYDRANT_DB_MAX_JOURNALING_SIZE_MB", "63"), + ("HYDRANT_DB_REPOS_MEMTABLE_SIZE_MB", "1"), + ("HYDRANT_DB_EVENTS_MEMTABLE_SIZE_MB", "0"), + ("HYDRANT_DB_RECORDS_MEMTABLE_SIZE_MB", "0"), + ("HYDRANT_RELAY_HOST", "not a url"), + ]; + // relay builds have no backfill, so its knobs aren't checked there + if std::cfg!(feature = "indexer") { + bad.push(("HYDRANT_BACKFILL_CONCURRENCY_LIMIT", "0")); + } + for (key, value) in bad { + let Err(err) = from_pairs(&[(key, value)]) else { + panic!("{key}={value:?} was accepted"); + }; + let err = err.to_string(); + assert!( + err.contains(key) && err.contains(value), + "{key}={value:?}: the error should name both, got: {err}" + ); + } + } } diff --git a/src/control/hosts.rs b/src/control/hosts.rs index fa757e8..db96ac2 100644 --- a/src/control/hosts.rs +++ b/src/control/hosts.rs @@ -136,13 +136,9 @@ impl ApiBinds { if let Some(mode) = lookup("HYDRANT_API_SOCKET_MODE") { binds = binds.with_unix_mode(parse_socket_mode(&mode)?); } - if let Some(raw) = lookup("HYDRANT_API_TCP_MANAGEMENT") { - let enabled = raw.trim().parse().map_err(|_| { - miette::miette!("invalid HYDRANT_API_TCP_MANAGEMENT `{raw}`: want true or false") - })?; - binds = binds.with_tcp_management(enabled); - } - Ok(Some(binds)) + let tcp_management = + crate::config::env::read(&lookup, "HYDRANT_API_TCP_MANAGEMENT", binds.tcp_management)?; + Ok(Some(binds.with_tcp_management(tcp_management))) } } diff --git a/tests/crawler_full_network.nu b/tests/crawler_full_network.nu index ea0b1e6..83b6ed3 100644 --- a/tests/crawler_full_network.nu +++ b/tests/crawler_full_network.nu @@ -49,7 +49,7 @@ def main [] { HYDRANT_ENABLE_DEBUG: "true", # for stats checking HYDRANT_DEBUG_PORT: (resolve-test-debug-port ($port + 1) | into string), HYDRANT_LOG_LEVEL: "debug", - HYDRANT_CURSOR_SAVE_INTERVAL: "1" # faster save + HYDRANT_CURSOR_SAVE_INTERVAL: "1s" # faster save } { sh -c $"($binary) >($log_file) 2>&1 & echo $!" | str trim | into int }