The RAG Cost Curve
When to reach for an index
Haystack EU · September 2026
RAG is Dead?
Where the answer is obvious
No text to grep
Images, music, molecules
Pathology, chemistry, media archives. The vector is the query.
Query volume
Support & helpdesk
Thousands of questions a day against one corpus. Break-even arrives before lunch.
Similarity over exactness
Fraud investigations
Identifying clusters of fraudulent activity and classifying new cases.
Corpus size
Legal & e-discovery
Millions of documents, where missing one is a legal problem.
Churn
E-commerce catalogues
Prices and stock move hourly. Churn voids the cache long before it troubles the index.
Speed at scale
Matching & recommenders
Finding matches in a billion-scale corpus, in near real time.
Three things my abstract promised
Reframed
16x to 32x compressed vector index costs single-digit recall loss
01
Held
Recall is a proxy, I'll show where it breaks
02
Reframed
Break-even against live search lands in tens-to-hundreds of queries per day
03
All open source & reproducible
My intuition
At ~100 queries per day, indexed code search is cheaper than agentic.
Defining break-even
Simple maths!
fixed daily
What the index costs on a day you ask it nothing: the instance divided by 30.4, the embedding run amortised over a year, re-embedding for churn.
saving per query
Billed dollars, live minus indexed. Invoiced amounts off a real account, not token estimates off a price list.
at matched quality
The subtraction is only legal when both arms answer as well as each other. A cheaper wrong answer is not a saving.
Where the cost comes from
Four ways for an agent to answer the same question, the search itself is not expensive.
No search, no payload. You pay for the answer, and nothing else.
Small results each time. But the whole transcript goes back every turn.
One fat payload of chunks, and a box that bills daily whether you ask or not.
The corpus is written to cache once, in full. Reads after that are cheap.
Only the index has a fixed cost, everything else is per query or per embedding. The same logic applies for code search and agentic memory.
ANN 201
The big idea
ANN: more than HNSW
Choose IVF for the best balance of performance and cost.
🚀higher QPS · ⚡️lower latency · 💰more expensive to serve · ⏳slower to build
| Index | OSS since | Lives in | QPS | Latency | Cost | Build | Tune with |
|---|---|---|---|---|---|---|---|
| HNSW | 2016 | RAM | 🚀🚀🚀🚀🚀 | ⚡️⚡️⚡️⚡️⚡️ | 💰💰💰💰💰 | ⏳⏳⏳⏳⏳ | ef at query time |
| ScaNN | 2020 | RAM | 🚀🚀🚀🚀🚀 | ⚡️⚡️⚡️⚡️⚡️ | 💰💰💰💰💰 | ⏳⏳⏳⏳⏳ | nprobe plus reorder depth |
| IVF | 2017 | RAM or disk | 🚀🚀🚀🚀🚀 | ⚡️⚡️⚡️⚡️⚡️ | 💰💰💰💰💰 | ⏳⏳⏳⏳⏳ | nprobe cells per query |
| DiskANN | 2020 | SSD, some RAM | 🚀🚀🚀🚀🚀 | ⚡️⚡️⚡️⚡️⚡️ | 💰💰💰💰💰 | ⏳⏳⏳⏳⏳ | beam width |
| AISAQ | 2025 | SSD, flat RAM | 🚀🚀🚀🚀🚀 | ⚡️⚡️⚡️⚡️⚡️ | 💰💰💰💰💰 | ⏳⏳⏳⏳⏳ | beam width |
ANN Benefits
Who doesn't love a trade-off triangle
ANN algorithms all trade perfection for reduced latency and cost.
The size problem
No matter what algorithm you use, embeddings are big. In RAM or on disk, size matters.
100M chunks and you are holding 1.23 TB before a single query runs.
Quantisation:
smaller numbers
Scalar quantisation
Round float32 → int8: 4x smaller embeddings, a small recall hit, almost no work.
RaBitQ: one bit per dimension
Rotate the space to reduce error, then keep just the sign of each dimension - one bit.
Product quantisation
Scalar quantisation shrinks every number, PQ shrinks the whole vector.
What it costs you (in theory)
Every lost bit risks recall, but the curve is surprisingly forgiving.
Quantisation shifts everything cheaper & faster
Each algorithm can use quantisation to trade accuracy for significantly reduced latency and cost.
Dimensionality reduction:
fewer numbers
PCA: rotate, drop the quiet axes
PCA finds the directions of greatest variance and keeps the top k. Fewer dimensions, full precision.
BenefitKeep one number instead of two and 94% of the variance - linear, fast, deterministic.
DrawbackMaximises for variance, not meaning: structure on a low-variance axis is discarded, and it must be refit when the data shifts.
MRL: one vector, many lengths
The dimensions are ordered by importance, so a prefix is a complete vector.
vs a model not trained for it
MRL tunes the model so the dimensions are ordered by importance. OpenAI's text-embedding-3-large is 3072-D native, but you can ask for any prefix down to 256-D via the dimensions parameter.
BenefitOne model, pick the length per query - short prefix to shortlist fast, full vector to re-rank. Degrades gracefully.
DrawbackOnly works if the model was trained this way - an ordinary embedding survives a light trim, then falls off a cliff once you cut hard (the berry line).
Fewer dimensions, small accuracy hit
Dimensionality reduction nudges any index toward fast and cheap.
Refine: scan cheap, rescore precise
Build time compression and dimensionality reduction both trade accuracy to buy speed and scale. Refinement wins accuracy back at query time.
- Coarse pass - scan
nprobeof the lists using the 1-bit RaBitQ codes. Cheap, and on its own it stops at 0.778. - Refine pass - Milvus keeps SQ8 copies beside the codes and rescores
refine_k × limitcandidates with them. Same candidates, better distances: 0.986. - The last point comes from the coarse pass, not the refine.
nprobe256 to 1024 buys 0.992, at a quarter of the throughput. - Re-ranking is a different axis. A cross-encoder reorders the k you already retrieved. Better ordering, identical recall.
Refinement pulls the other way
PCA and Matryoshka trade accuracy for speed and cost. Refinement trades both to buy accuracy back.
Does recall even matter?
How an answer is scored
Every number in this section is a judged answer, not a retrieval metric
Judge accuracy is the share of questions Opus 5 marks correct, returning a {correct, reason} verdict without being told which arm produced the answer. Hedges, refusals, and a gold answer named inside a denial all score incorrect.
Strict and binary, so treat the absolute level as a floor: NQ's 2018 gold answers against a 2023 corpus put every arm between 0.22 and 0.50. Compare the arms with each other rather than reading the height.
NQ-Open10M Wikipedia 2023-11mxbai-embed-large-v1Sonnet 5 answersOpus 5 judgesseed 42
Recall barely impacts answer quality?
450 NQ-Open validation questions, 10M Wikipedia chunks, twelve arm-runs, paired bootstrap.
+0.235[0.122, 0.353]
judge accuracy per unit of recall@10
+0.0133[-0.0133, +0.0400]
judge accuracy gap to full precision: a tie
9.22x
measured footprint at PCA-384 + SQ8, vs the IVF_FLAT 1024d index
Most questions never had a choice
The same 450 questions, counted by how many of the eleven retrieval arms got each one right.
38 %
173 questions no arm answered. Better retrieval was never going to be the fix.
33 %
150 questions every arm answered, from the crudest quantisation up.
7 %
32 questions genuinely contested. The whole budget a retrieval knob has to play with.
"where are the winter olympics and when do they start"
Gold: "Pyeongchang, 9 Feb 2018". All arms answered "Beijing, 4 February 2022", the corpus is newer than the gold.
"who was the viceroy when the simon commission visited india"
Gold: Lord Irwin. Eleven arms, eleven byte-identical answers, right down to the one-bit RaBitQ index.
"who wrote the theme song to law and order"
Gold: Mike Post. Six arms said Mike Post. The other five said "I don't know". Not one got it wrong.
Questions filtered 450 → 264 to maximise answer density.
Depth recovers some accuracy
Reference arm sq8@512, the 264-question eligible stratum.
Aim for recall@k ≳ 0.95
Judge accuracy against measured recall, at three depths. Same 264 questions, four SQ8 arms.
The same recall range buys 0.064 accuracy points at k=1 and 0.144 at k=10.
Measuring quantisation
How the recall numbers were measured Method
Index family against compression against nprobe against refine_k, swept at 1M and 10M, every arm scored against exact top-100 neighbours.
Milvus v2.6.18mxbai-embed-large-v1 1024-dWikipedia 2023-11 chunksNQ-opennlist 4096seed 42
Absolute throughput here is a laptop, single-node Docker on Apple Silicon.
Recall vs compression
The 16-32x in the abstract ships as 14.6x in memory. 10M embeddings, measured against the 42.31 GB IVF_FLAT index.
SQ8 won on its own
SQ8 vs RaBitQ + refine (k=2), matched pairs at 10M.
One code, no refine
IVF_SQ8 wins
- recall 0.9917, p50 81 ms
- 8.8 QPS at the 0.99 point
- 37.9 QPS at 0.95
- 10.8 GiB resident, 3.7x under the fp32 index
1-bit codes plus a refine pass
RaBitQ + SQ8 k=2
- recall 0.9920, the same answer
- 0.88 QPS at the 0.99 point, a 10x gap
- 12.1 QPS at 0.95, a 3.1x gap
- 12.3 GiB resident, 3.2x under the fp32 index
10M rowsmatched pairsnlist 4096laptop QPS, read the ratio
Refine keeps the SQ8 vectors anyway and adds the 1-bit codes on top, so it gives up 10x the throughput and 1.5 GiB of memory to land on the same recall.
Measuring code search
break-even
The study Method
Same questions, same repositories, same prompt. Only the tool changes.
parametric
no repository access
agentic
grep / read / glob loop
indexed
claude-context search_code over a Milvus index
indexed k=3
the same tool with limit bound to 3
stuffed
whole repo in context, prompt-cached
fastapi + agentic-hil40 questions eachclaude-context 0.1.15Milvuspaired bootstrap
Training data matters
A blind model answered 25 of 40 questions on fastapi, and 0 on a repository published after training cutoff.
Reading the paired medians
How to readMedian of differences, each question paired with itself. An unpaired chart of the very same cells can point the other way.
Unpaired: +4,616 against grep. Paired: -4,717. Grep's median question is cheap, its tail is not. Pairing is within a corpus, so these are four estimates, not one test.
Cost per correct answer
Total agent spend over the whole run, divided by the answers the judge marked correct
Code it knows
Every retrieval arm lands within ~15% of the others.
Code it has never seen
The index is ~40% cheaper per correct answer.
A second workload: conversation memory
Synthetic history at 98k, 392k and 1.2M tokens. The live arm replays the whole transcript into context every query; the index arm searches it
Replay gets dearer, the index does not
Median billed cost per query. Caching live on the replay side, memsearch uncached
The cost curve(s), assembled
Break-even on conversation memory
Replay per query against the index per query, on a dedicated box
Code the model already knows
fastapi at 259k, 1.1M and 6.2M tokens. Against grep, no volume repays the index
Code it has never seen
Indexed unseen code repays at 24 to 83 queries per day, on a dedicated box
The floor can go to zero
Same index, same corpus, the same measured token costs. Only the thing underneath it changes
a box of your own
A dedicated r8g.large at $86.07 a month to hold the index. Break-even lands at 24 - 334 queries per day.
a box you already run
Milvus Lite on your laptop or spare space on an existing box. Same index, same prices, break-even on the first query.
no box at all
Serverless, metered per query with no floor to amortise. Every corpus combined fits into the Zilliz free-forever tier.
Rates read off zilliz.com/pricing#calculator on 2026-09-15: $4 per million vCU, a 1536-dim FP16 write costing 0.75 vCU and a read on a 1M-vector collection 15 vCU. Reads grow with collection size, so 15 is an upper bound here. Free tier 5 GB storage plus 2.5M vCU a month, up to 5 collections.
What we discovered
Live wins
A short agent history, asked a few times a day, should not be indexed.
334queries a day before a dedicated index pays, at 98k tokens of conversation memory
Live wins
On a repository the model already knows, grep is cheaper at every size.
neverbreak-even is infinite at S, M and L on fastapi: no volume repays the index
Live wins
Discount the cache and the crossover is still in the hundreds.
232queries a day at S with the prompt cache held at 99 %, against 334 measured
Index wins
Agentic memory: the curve collapses as the history grows.
15 / 10queries a day at 392k and 1.2M tokens of history, against 334 at 98k
Index wins
On code the model has never seen, the index is cheaper per right answer.
40%cheaper per correct answer than grep, and more accurate: 39 of 40 against 36
Index wins
Against stuffing the same repository into context, the index repays quickly.
12queries a day on fastapi, $0.29 a query re-sent against $0.067 indexed
Reproduce any number
github.com/simonhearne/rag-cost-curve
make setup && source .venv/bin/activate
make up # docker compose up -d + Milvus health wait
make data-1m # or data-10m
jupyter lab # notebooks 01 to 06b in orderRequires Docker with 16 GB RAM or more
Everything I showed you is reproducible on your laptop.
Try it yourself
memsearch
Install the plugin, and memsearch captures conversations automatically and provides semantic recall with zero configuration.
/zilliztech/memsearch
claude-context
MCP plugin that adds semantic code search to AI coding agents, giving them deep context from your entire codebase.
/zilliztech/claude-context
Thank you!
simon @ zilliz.com
What I would run next
Six gaps that I will address.
range
BM25 and dense, measured apart
Both indexed arms already searched dense plus BM25, fused with RRF, at their tools' defaults. Every agent result here carries a lexical half that nothing isolates.
range
GraphRAG & facts that span chunks
On prose the index paid about 4x in both powered strata, and the hop contrast came back beyond reach. Every planted fact sat in one chunk, so the deeper tiers were never called.
range
BrowseComp benchmark
One issues corpus is one point, extend the answer questioning benchmark set with BrowseComp for better comparability.
quality
A better agent, and more than one
One model answers every question once, with no temperature to pin. The plateau on the recall slide may be the reader's ceiling rather than retrieval's, and nothing here separates the two.
quality
Questions the best arm can fail
memsearch answered 62 of 62 correctly. A benchmark its best arm never fails can rank cost. It cannot rank quality.
quality
Size the run from measured spread
n = 40 was inherited for comparability, not chosen for power, and on prose the power analysis missed by about 5x. Pilot the spread first, then pre-register the n.
Backup: MRL vs PCA
Truncation does not preserve retrieval recall on this model. Identity recall is stricter than the task metrics MRL is usually sold on.
WS4 later added the other half: at matched recall, truncation answers as well as any other mechanism.
Backup: the PCA cost curve
Footprint compression, vs the measured IVF_FLAT 1024d index.
Backup: nlist and the segment heap
At 128d the fixed heap is 54 MiB against 135 MiB of index, 189 MiB total: 28 % of the footprint. At 1024d the same 54 MiB is 5 % of a much larger footprint. The heap does not shrink with the vector, so it becomes a bigger share of a smaller index.
Backup: the centring trap
At 512 dimensions: MRL truncation 0.700, PCA centred 0.827, PCA uncentred 0.964.
The centred PCA line is flat from 896d down to 384d: whatever dimension PCA drops in that range, center=True had already thrown away roughly the same 0.17 points of recall, regardless of how many more dimensions it keeps.
Backup: recall vs compression (1M)
The 16-32x in the abstract ships as 14.0x in memory. 1M embeddings, measured against the 4.26 GB IVF_FLAT index. Same denominator as the 10M chart in the main deck, so the two are comparable number for number.
Backup: refine_k = 2 does everything
1M scale. Bare RaBitQ against a refine pass at k=1, 2, 5 and 10, recall against nprobe.
Backup: the 10M k-anchor table
| arm | refine k | best nprobe | recall@10 | qps |
|---|---|---|---|---|
| rabitq (bare) | none | 256 | 0.7776 | 2.72 |
| rabitq_refine_sq8 | 1 | 256 | 0.9793 | 1.94 |
| rabitq_refine_sq8 | 2 | 1024 | 0.9920 | 0.88 |
| rabitq_refine_sq8 | 5 | 1024 | 0.9921 | 0.64 |
| rabitq_refine_sq8 | 10 | 256 | 0.9849 | 1.89 |
k=2 and k=5 land the same recall, 0.992, at less than half the throughput of k=1. k=10 recovers some QPS but loses half a point of recall: each row is its own best-nprobe operating point (nprobe is the sweep variable, not k), so QPS is not monotone in k on its own.
Backup: three passes at one question Method
Does retrieval quality predict answer quality? Three passes over the same corpus
pass 1
Does recall predict quality? 450 questions, 11 arms, one depth (k=10)
pass 2
Is there a threshold? 264-question eligible stratum, 15 arms, the densest ladder we could build
pass 3
Does depth change the answer? the same 264, 4 arms, swept at k = 1, 3, 5, 10
Backup: ground-truth verification detail
| scale | queries checked | k | full set match | exact order match | near-tie mismatches | true mismatches |
|---|---|---|---|---|---|---|
| 1M | 50 | 100 | 50/50 | 45/50 | 12 | 0 |
| 10M | 50 | 100 | 50/50 | 50/50 | 0 | 0 |
Mean and minimum set overlap were 100 % at both scales. Method: exact brute-force float64 cosine, seed 42.
No separate ground-truth answer-presence file exists. Answer-presence@10 (the fraction of questions with the gold answer inside the retrieved top 10) ranges 0.7378 (sq8@4) to 0.8178 (sq8@512, the reference arm) across the WS4 arm sweep, read from the WS4 summary instead.
Backup: the judge prompts
WS4 (the recall sweep, K1)
results/ws4/judge_prompt.txt, 944 B
WS6a (code search)
results/ws6a/judge_prompt.txt, 1262 B
WS6b (memory)
results/ws6b/judge_prompt.txt, 1160 B
WS8 (prose)
results/ws8/judge_prompt.txt, 1357 B
All four are graded by Opus 5 and require a JSON object with exactly two keys: correct (boolean) and reason (one sentence, at most 30 words).
GuardWS4: "...or names a gold answer only inside a statement denying that it can answer."
WS6b: "...or names the expected value only inside a statement denying that it can answer."
Catches a model that says "I don't know, though it might be X" landing on the right X: without the guard that reads as a hedge, not a correct answer named and then disowned.
A seeded 60-row sample was judged a second time: 60 of 60 verdicts agreed, against a pre-registered 0.95 threshold.
Backup: WS6a setup effort
| arm | steps | wall-clock (min) | accounts | credentials | failed attempts |
|---|---|---|---|---|---|
| parametric | 0 | 0 | 0 | 0 | 0 |
| stuffed | 4 | 0 | 0 | 0 | 0 |
| agentic | 4 | 0 | 0 | 0 | 1 |
| indexed | 8 | 70 | 2 | 3 | 4 |
Three of the four indexed-arm failures were silent: nothing told the operator anything had gone wrong.
Backup: the inversion
I tested claude-context against fastapi. The indexed arm used more tokens, not fewer.
Easy interpretation
The index looked 73% dearer. That set one arm's typical question against the other's, rather than comparing the two on the same question.
What the data supports
Question by question, the index is still dearer, but by a median 2,857 tokens, on an interval reaching down to nine.
Backup: 24x the corpus, nothing we can detect
Ten questions per cell, at 259k, 1.14M and 6.23M corpus tokens. Every paired interval crosses zero: the agentic arm's widest span is +1,041 tokens, CI [-108, +3,890], with five of the ten growing.
Every judge-accuracy delta is exactly 0.0, CI [0, 0], and accuracy is 1.000 in all six cells.
Backup: tuning retrieval depth
On fastapi the indexed arm cost more tokens than grep, not fewer: a paired median of 2,857 a question, on an interval reaching down to nine. The obvious response is to ask for fewer chunks.
Turn it down and the tokens fall
Asking for 3 chunks instead of 10 saves about 4,400 tokens a question. The knob does exactly what you would expect.
And it still does not win
The gap to grep is now too small to call either way, the model ran more searches to make up the payload.
Backup: the same knob, on code it has never seen
Bars are unpaired per-arm medians over the same 40 questions on agentic-hil.
The same knob does nothing here
On code the model has never seen, turning it down is as likely to cost tokens as to save them.
k=10 gives better results with fewer tokens
On the unseen dataset, indexed retrieval outperforms agentic search with 15% fewer input tokens.
Backup: the prose test Method
7,647 GitHub issue and PR threads from kubernetes, rust-lang and fastapi, fetched in a June to August 2026 window that postdates the training cutoff. 75 questions, stratified by how many threads the answer spans.
parametric
no corpus access at all. This arm is the contamination gate
agentic
grep / read / glob over the thread files
indexed
Milvus over 45,888 chunks of 1,800 chars, top-k 10, $0.46 to embed
text-embedding-3-small1 / 2 / 3+ hopspaired bootstrap10,000 resamplesseed 42
Backup: fine-tuning required
The index costs about 4x more for no gain
Four times the tokens per question, and it did not score higher. The model searched just as often, so the cost is in how much each search hands back.
The hop question is beyond this study's reach
The run was sized on code, and prose questions vary far more. A real difference would have to be about five times larger before this design could see it.
Backup: WS8 power and robustness
Why the null is uninformative below ~100,000 tokens, and why the 1-hop separation is not outliers.
| injected gradient (tokens) | power at n=30 |
|---|---|
| 0 (null calibration) | 0.043 |
| 25,000 (declared floor) | 0.065 |
| 50,000 | 0.142 |
| 100,000 | 0.728 |
| 125,000 | 0.863 |
| 200,000 | 0.985 |
Leave-k-out, 1 hop
Minimum interval lower bound across every subset: +17,402 at k=1, +9,390 at k=2, +1,377 at k=3. All 4,525 subsets still exclude zero.
False-positive rate
The paired test on hop1's own centred shape: 0.056 ± 0.003 against a nominal 0.05. Slightly anti-conservative, not enough to manufacture the separation.
The contrast is not rigged
Under a true null at the observed dispersion: 0.034 ± 0.003, slightly conservative. Holds under all three defensible null pools.
Backup: the M2 pair table
| pair | comparison | diff | face 95% CI | Bonferroni CI | matched |
|---|---|---|---|---|---|
| A (primary) | pca_uc_512_sq8@512 vs refine@64 | -0.0089 | [-0.0356, +0.0178] | [-0.0444, +0.0244] | yes |
| B | pca_uc_512_sq8@512 vs sq8@64 | +0.0067 | [-0.0178, +0.0311] | [-0.0244, +0.0400] | yes |
| C | pca_uc_384_sq8@512 vs sq8@16 | +0.0133 | [-0.0156, +0.0422] | [-0.0244, +0.0511] | yes |
| D | mrl_512_sq8@512 vs sq8@4 | +0.0333 | [-0.0022, +0.0689] | [-0.0133, +0.0800] | no |
| E | pq@256 vs sq8@8 | +0.0133 | [-0.0200, +0.0467] | [-0.0289, +0.0556] | no |
| F | mrl_512_sq8@512 vs rabitq@256 | -0.0089 | [-0.0378, +0.0200] | [-0.0467, +0.0289] | yes |
Pair A is the pre-registered primary and is matched: within noise. All matched pairs, A, B, C and F, sit inside plus or minus 0.05 on the Bonferroni-adjusted interval. D and E are unmatched exactly as pre-registered: their recall gap exceeds the matching tolerance, so they are shown but not counted as evidence for or against M2.
Backup: the answer was there, the agent ignored it
MechanismTruncation, quantisation and pruning give the same answer quality at the same measured recall.
Six pairs, corrected for multiple comparisons. The primary pair: -0.9 points [-3.6, +1.8]. Every matched pair inside ±0.05.
Backup: the mediator, three panels
Layout mirrors results/ws4/chart_mediator.png: the first panel moves with recall, the other two do not. That gap between panel one and panels two and three is where the "spend on the reader, not the retriever" argument lives.
Backup: WS4 secondary metrics
| arm | recall | presence | EM | F1 | judge |
|---|---|---|---|---|---|
| sq8@512 (reference) | 0.9920 | reference | reference | reference | reference |
| sq8@64 | 0.9691 | DROP | PLATEAU | PLATEAU | PLATEAU |
| refine@64 | 0.9673 | PLATEAU | PLATEAU | PLATEAU | PLATEAU |
| pca_uc_512_sq8@512 | 0.9642 | PLATEAU | PLATEAU | PLATEAU | PLATEAU |
| pca_uc_384_sq8@512 | 0.9231 | PLATEAU | PLATEAU | PLATEAU | PLATEAU |
| sq8@16 | 0.9184 | DROP | PLATEAU | PLATEAU | DROP |
| sq8@8 | 0.8660 | DROP | PLATEAU | PLATEAU | PLATEAU |
| pq@256 | 0.8171 | PLATEAU | PLATEAU | PLATEAU | PLATEAU |
| sq8@4 | 0.8007 | DROP | DROP | DROP | DROP |
| rabitq@256 | 0.7811 | DROP | PLATEAU | PLATEAU | PLATEAU |
| mrl_512_sq8@512 | 0.7576 | DROP | PLATEAU | PLATEAU | DROP |
EM and F1 put their lowest PLATEAU at recall 0.817 (pq@256): only sq8@4 drops below it. The judge is primary; the two highlighted rows are where judge and EM disagree. No boundary on this table is a locatable threshold: see the notes.
Backup: what goes into the curve Method
One index, one baseline, three corpus sizes per workload. Every input below is a committed parameter
the index side
Embed once, then pay per search. The footprint is PCA to 384 dims plus SQ8, priced on one r8g.large at $86.07 a month, which is $2.83 a day whether you ask one question or ten thousand.
the live side
The whole corpus back through the model every query, prompt-cached at the measured hit rate: 1.00, 0.92, 0.97 at S, M and L on the memory workload. Three price lists, all as of 2026-09-07, a URL per row.
the gate
Fed each configuration's own parameters, the model reproduces the measured bill to within 0.09% across 24 rows. Dashed segments interpolate between measured points and never past them.
memory S / M / L = 98k / 392k / 1.2M tokensr8g.large, 16 GiBmodel prices 2026-09-07, instance 2026-09-15seed 42
Backup: the WS5 sensitivity table
The memory workload. Since WS9 the same file carries the unseen-corpus rows too, where infra_mode spreads break-even by 7,421x rather than 68,945x
| knob | setting | ratio vs base | spread | weakens claim | strengthens claim |
|---|---|---|---|---|---|
| infra_mode | marginal GiB, no floor | 0.00001x to 0.0002x | 68,944.94x | no | yes |
| live_uncached | list price | 0.05x to 0.25x | 21.85x | no | yes |
| infra_usd_per_gib_month | x0.5 / x2 | 0.50x to 2.00x | 4.00x | no | yes |
| claude_price | x0.5 / x1.5 | 0.67x to 2.00x | 3.00x | no | yes |
| cache_hit_rate | miss/2 / miss x2 | 0.61x to 1.48x | 2.45x | no | yes |
| regime | openai / open_weight | window-limited, some infeasible | 1.38x | yes | no |
| cache_ttl | 5m write (1.25x) | 1.00x to 1.35x | 1.35x | no | no |
| footprint | sq8 / refine_k2 | 1.00x, no change | 1.00x | no | no |
| open_weight_usd_per_mtok | x0.5 / x2 | not computable, never repays | 1.00x | yes | no |
Sanity check: fed each configuration's own parameters, the cost model reproduces the measured mean bill to within 0.09 %, mean-billed basis, across 24 rows. That is what licenses interpolating between the three measured sizes rather than only quoting the three points.
Backup: what the index costs to keep
Backup: HNSW, navigate a graph RAM
Hierarchical Navigable Small World. Multi-layer graph: top layers have long-range highways, lower layers have local connections. Start at the top, walk greedily closer, drop down a layer, repeat.
Backup: IVF, partition the space RAM or Disk
IVF clusters the vectors into nlist cells. At query time, only search within the nearest nprobe cells.
Backup: DiskANN, when RAM runs out Disk
Graph index, engineered for SSD. Minimises random reads, index billions of vectors on ~GBs of RAM.