Three everyday changes
- Insert
- Add a new document. It must be present in a direct read and appear with its new version in vector search.
- Replace
- Keep the document ID, but replace its content and vector. Finding the old version does not count. Before the write, the target must be absent from search with its new vector.
- Delete
- Remove a previously retrievable document. It must become absent in both direct reads and repeated searches with its old vector. A missing search hit alone is insufficient.
A clock that starts with the application
We start the clock when the client submits a change. Separate readers begin before the write acknowledgement and check the direct-ID and vector-search APIs independently.
Two useful milestones
First visible: the later of the first correct response on each read channel. Both channels have shown the target state by that time, although they were not sampled at the same instant.
Repeated confirmation: each channel has returned at least three correct responses spanning 50 ms. Incorrect responses and read errors reset that channel’s confirmation sequence.
Confirmation includes its observation window. The result also includes client scheduling, serialization, transport, and query execution. It does not isolate time inside the index.
A change has five seconds to succeed. Late responses stay in the evidence but cannot establish success. Each read channel stops after confirmation, so this finite window cannot prove permanent visibility.
How to read a result
What does p95 mean?
Within a trial, the 95th percentile (p95) is a time at or below which 95% of successful measurements fall. It helps describe the slower end of the measurements without relying on a single maximum.
Each engine and workload has five trials. The comparison shows the median of those five trial p95s. It is not the p95 of every event pooled together.
What is the 95% interval?
We resample whole trials 5,000 times to form a bootstrap interval around the estimate. Five trials give coarse uncertainty. The interval does not establish statistical significance or a universal winner.
Why show confirmation rates?
Latency numbers describe successful changes. The confirmation rate uses all offered changes, including errors, timeouts, and client drops. A short time is not useful on its own if many changes fail.
How is search quality checked?
One hundred real questions query an unchanged part of the same index while mutations target separate document IDs. Approximate search is compared with exhaustive cosine search. At least 95% recall is required before and during mutations. Equal-score neighbors at the tenth boundary receive credit within a 10⁻⁶ tolerance.
The separate nDCG@10 score uses relevance judgments from the dataset. These are scores for declared subsets, not full BEIR leaderboard scores. Quality queries are filtered to the fixed cohort; mutation probes search the whole collection.
What makes a comparison valid?
Every planned trial must finish and pass all eight checks. Failed trials and failed checks stay visible.
- Changes were submitted on schedule.
- Observation reads started on schedule.
- Background queries started on schedule.
- Search recall met the 95% target.
- Every mutation had a valid starting state.
- The client dropped no mutations.
- Quality queries stayed in the intended corpus.
- The configured search index was verified.
Exact timing and index rules
Arrival and background-query dispatch p95 must not exceed their requested interarrival interval. Probe dispatch p95 must be at most 10 ms. Positive probes are spaced at least 25 ms apart; negative probes at least 10 ms apart.
All engines use cosine similarity and HNSW parameters m=16, construction effort 128, and search effort 128. PostgreSQL records an HNSW query plan with sequential scans disabled. Weaviate uses a zero flat-search cutoff. Qdrant requires at least 95% indexed base vectors; remaining small segments can still use scans.
We also test the measurement harness
Known-delay controls inject visibility delays of 0, 25, and 100 ms. A separate 100 ms acknowledgement delay checks that readers are independent of write acknowledgement.
| Visibility delay | Acknowledgement delay | Confirmed changes |
|---|---|---|
| 0 ms | 0 ms | 20/20 |
| 25 ms | 0 ms | 20/20 |
| 100 ms | 0 ms | 20/20 |
| 0 ms | 100 ms | 20/20 |
Where these results apply
This reference uses one Apple M4 host, Docker Desktop, and pinned images. Each database gets two virtual CPUs and 3 GiB of memory. Trials use fresh collections, five repetitions, 30-second measured phases, ten changes/s, and ten background queries/s. Bursts offer 40 changes/s for one second every ten seconds. Observation reads add traffic.
- The datasets supply real text and questions, but document changes are simulated on disjoint IDs.
- Static relevant-first subsets, one embedding model, and a filtered quality workload bound the conclusions.
- The client and containers share physical hardware. Other hardware and configurations define different comparison tracks.
- Qdrant uses wait=true, Weaviate uses synchronous indexing with REST writes and GraphQL searches, and PostgreSQL uses autocommit with independent connections. These are different API and acknowledgement semantics.
- The experiment does not establish maximum throughput, durability, distributed consistency, or performance during long-running churn.
- An earlier host interruption retained six complete trials and restarted a preserved partial seventh attempt. A later user pause followed trial 44 at a clean boundary. Calibration and environment checks were repeated; container restarts may affect cache state.
Independent performance replication on another host remains future work. All raw observations and continuation records are available to inspect.
Related work
ANN-Benchmarks, VectorDBBench, FreshDiskANN, and BEIR provide important context. This project makes no claim to be the first freshness benchmark.
Read the full versioned protocol →