SciFact
Scientific claims paired with research abstracts and evidence judgments. The workload uses 4,000 immutable documents.
- Cached documents
- 5,183
- Selected questions
- 100
- Vector dimensions
- 384
License: Corpus: ODC-By-1.0; claims/evidence: CC-BY-4.0
BEIR and original dataset authors; Wikipedia contributors for NQ, SciFact authors and cited article authors for SciFact.
200-word title+text prefix; seeded reservoir of documents with relevant documents for selected test queries retained at front. Simulated mutations; no historical replay.
Natural Questions
Questions from real search users paired with Wikipedia passages. Workloads use 10,000 or 100,000 immutable documents.
- Cached documents
- 124,000
- Selected questions
- 100
- Vector dimensions
- 384
License: CC-BY-SA-3.0
BEIR and original dataset authors; Wikipedia contributors for NQ, SciFact authors and cited article authors for SciFact.
200-word title+text prefix; seeded reservoir of documents with relevant documents for selected test queries retained at front. Simulated mutations; no historical replay.
How the subsets are prepared
A fixed seed selects 100 test questions. All documents judged relevant to those questions are retained first; remaining documents are selected with a seeded reservoir. This deliberately preserves relevance judgments inside the tested collection. The resulting scores are not full BEIR scores.
One embedding model, outside the clock
We cache normalized 384-dimensional float32 vectors from all-MiniLM-L6-v2 at revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41. Input is limited to a 200-word prefix followed by the model’s 256-token limit. Model license: Apache-2.0.
The reference vectors were prepared on Apple MPS. Regeneration on other devices can change floating-point values. Use the exact cache download for byte-identical input data, or report your regenerated cache hash.
The MIT code license does not replace dataset terms. See the full attribution and transformation notice.
Download exact data and vectors ↓