Freshness Benchmark

Inspect it. Repeat it. Extend it.

Everything behind a result.

Download the evidence first, or run the same protocol on your own hardware. No paid model API is required.

Download the benchmark

Complete releaseReport, figures, exact vectors, raw evidence, and checksums.
Benchmark sourceThe maintained 0.5.0 benchmark code, tests, protocol, and original website.
Complete experiment evidenceEvery raw observation, calibration trace, and continuation record.
Exact data and vectorsThe exact measured cache, with upstream terms and attribution.
Results as CSVA compact comparison for your own analysis.
Results and provenance as JSONAggregates, per-trial metadata, and validity checks.
Exact source at experiment launchThe immutable measurement source snapshot. Use maintained source as the full project scaffold.
Verification reportResults of the deep cache, retrieval-quality, and raw-evidence audit.
Download checksumsSHA-256 inventory. Checksums establish integrity, not publisher authenticity.

The current Astro website source is in the repository’s web directory. The immutable 0.5.0 source archive retains the website that accompanied that release.

Run it on your hardware

Use Python 3.12, uv, and Docker. Download and unzip the source, or clone the repository, then run these commands from its root. Data preparation downloads public archives and model weights.

For exact measured input vectors, extract reference-data-v2.tar.gz at the repository root and skip both preparation commands. CPU regeneration creates a different cache hash; report it.

Repository root
uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python -e '.[publication]'
docker compose -p freshness-v2 -f docker-compose.v2.yml up -d
.venv/bin/python scripts/prepare_v2_data.py scifact --device cpu
.venv/bin/python scripts/prepare_v2_data.py nq --device cpu

# Start with a functional smoke run.
.venv/bin/python scripts/run_streaming.py \
  --out artifacts/my-smoke --scenarios scifact-small \
  --trials 1 --duration 1

# Run the complete reference matrix.
.venv/bin/python scripts/run_streaming.py \
  --out artifacts/my-v2 --trials 5 --duration 30
.venv/bin/python scripts/verify_v2.py artifacts/my-v2 --check-data

A complete run builds 60 fresh collections, including 30 at 100k scale. Keep other heavy workloads off the host and run one matrix at a time. A smoke run checks functionality; it is not a reference result.

Full environment, build, and reproduction guide →

Audit the downloaded evidence

tar -xzf benchmark-evidence.tar.gz
.venv/bin/python scripts/verify_v2.py publication-v2
# With the exact measured cache available:
.venv/bin/python scripts/verify_v2.py publication-v2 --check-data

The verifier reconstructs confirmations and summary metrics from raw probes. The deep mode also checks cache hashes and recomputes retrieval quality. The idle baseline is retained as aggregates; the deep query-quality audit covers the measured mixed phase.

A passing verifier establishes internal consistency, not independent authentication or performance replication. A report can honestly retain failed validity checks and still verify.

Share a complete experiment

Include every planned trial, raw event and probe traces, exact data and model hashes, source, image versions, resource limits, calibration, and failed checks. Explain configuration changes and keep hardware tracks separate.

Community results remain unverified until independently reproduced.

Submit results on GitHub ↗

Cite the protocol and result version

Aadithya Senthil Kumaran. Freshness Benchmark: Mixed-Mutation Visibility Protocol v2. Software version 0.5.0, 2026.

Identify the hardware track, link to the release, and cite the original datasets separately. Machine-readable citation ↗