HeavyPad snapshots #

Strata's public map is a deploy-time snapshot. The original interactive heatmap, local explorer and compatibility API still support live queries of the most recently published summaries. Neither experience needs a database write per archive collection.

Consumers checked for the October 2026 migration #

A source search across the local Tangled repositories and the available personal GitHub checkout found these consumers. This is a source inventory, not a claim that public API access logs establish every external consumer.

All existing read endpoints remain supported, including arbitrary valid heatmap bucket sizes and the top-60 ranking limit. Public/private Parquet policy is unchanged. The old /ingest write endpoint is retired after cutover. No event bodies, profiles, block data or additional archive attributes are collected by the summary job.

Data path #

The existing hourly Prefect flow runs on HeavyPad. Its SQLite checkpoint is /home/stoat/prefect-analytics/strata/checkpoint.sqlite. A file lock prevents overlapping writers. The job lists sealed segments, compares checksums locally, and reads only new or changed collection footers. Collection rows are updated only when their counts change; removed collections are deleted in the same transaction as segment metadata.

The builder produces progress, checksums, collection and namespace totals, segment metadata, global top-60 heatmap series with an other remainder, and per-namespace series. Object names are SHA-256 content hashes. Unchanged objects are reused. The initial D1 export produced 2,511 objects totaling 13.5 MB, including 2,504 namespace objects; these are derived serving data, not additional archive gathering.

The authenticated publisher uploads objects to the existing private R2 bucket under snapshots/objects/, then replaces snapshots/manifest. A failed object upload leaves the previous manifest serving. Each request reads one manifest generation. The Worker caches immutable objects and API responses, and aggregates only the requested view. HeavyPad is never in the public request path. API response caching remains five minutes. The public static map's deployment cadence and assets remain unchanged.

The builder records uploaded hashes locally and removes objects not referenced for seven days. Identical generations skip publication and checkpoint upload entirely. It never deletes an object referenced by the current manifest. This bounds historical storage while retaining recent rollback data. A compressed SQLite checkpoint is backed up to private R2 before publication. Existing D1 data is retained for rollback; the final Worker has no D1 binding and cannot write to it.

Recovery and rollout #

Restore explicitly by running strata-hourly with restore_checkpoint=true. This only installs a checkpoint when the local file is absent; it verifies SQLite integrity and atomically renames the restored file. It clears upload bookkeeping so a recovered job cannot assume expired R2 objects still exist. A missing checkpoint without the explicit restore flag fails instead of launching an accidental full archive crawl.

For bootstrap, use max_segments=0 with restoration to validate recovery/publication without reading archive footers. Then run normal reconciliation before enabling the hourly schedule. Runtime secrets remain existing Prefect Secret block references.

The migration compares seeded snapshot responses against saved live D1 responses before cutover. scripts/verify-snapshot.mjs SNAPSHOT_DIRECTORY BASELINE_DIRECTORY, run with node --import tsx, verifies data equivalence. Worker tests compare heatmap aggregation against real SQLite, including sparse indices and zero counts. Python tests cover transaction rollback, unchanged count writes, removals and failed publication.

Rollback can restore snapshots/previous-manifest while its objects remain retained or redeploy the previous D1 Worker version. Keep the D1-writing hourly deployment disabled; restoring that old ingestion path would restore its spend. D1 and the new checkpoint diverge after cutover, so D1 is only a dated fallback, never the current source of truth.