0014: Reference data's RAG economics — what the evidence licenses

Status: accepted Date: 2026-08-03

Context

reference-data argues that a corpus belongs on disk and enters the window as a query-shaped slice, and it argues where that move becomes RAG. Both arguments reach for numbers — retrieval failure rates, per-query bills, a corpus size at which frontloading is cheaper — and the field supplies numbers of wildly unequal grade. A research pass ran 2026-08-02 against the primary sources, and returned three things that could not all be carried at once: one vendor's measured ablation, one vendor's recommended threshold, and a first-party decision whose own author grades it as vibes.

The pressure is that a pattern node is read as settled. A threshold quoted in it becomes a rule someone designs against; a rate quoted in it becomes the rate. This record fixes which of those the evidence licenses.

Method and evidence grades

Sources fetched directly: Anthropic's contextual-retrieval and context-engineering posts, Anthropic's embeddings and pricing docs, Lewis et al. 2020, Liu et al. 2023, and the OpenAI, Voyage and Pinecone pricing pages. Read at abstract level only: the 2026 agentic-retrieval benchmarks. Read secondhand: a practitioner interview write-up, a vendor counter-post, a debate round-up. Four grades result, and the node is bound to respect the difference:

• Measured and published with method. Anthropic's retrieval ablation: top-20 retrieval failure 5.7% at baseline, cut 35% by contextual embeddings (→3.7%), 49% by adding contextual BM25 (→2.9%), 67% by adding reranking (→1.9%). One vendor, one stack, one evaluation — measured, not general. Liu et al.'s U-curve is the other measured result carried: recall highest at the edges of a long context, "significantly degrades" in the middle, even for explicitly long-context models.
• Vendor-published prices, checked 2026-08-02. Facts about a bill, not about behaviour, and true until the page changes.
• First-party assertion, self-graded. Claude Code shipped with RAG and dropped it for agentic search — the decision and its stated reasons (simplicity, security/privacy, staleness, reliability) are beyond dispute; the outperformance is self-described as internal benchmarks "plus vibes." No published Anthropic benchmark of the comparison exists.
• Directional only. The 2026 benchmarks at abstract depth; a vendor's 40%-fewer-tokens claim with a commercial stake and no methodology; a secondhand account of an internal study at another company. These characterize a failure regime. They do not measure one.

One figure was refused: a 55.2%-vs-44.0% grep-vs-vector result circulating in search summaries could not be confirmed against the fetched abstract, and is carried nowhere.

Decision

1 · The node carries no numeric floor (ruled 2026-08-02). The candidate was first-party and specific: below ~200,000 tokens (about 500 pages), include the whole knowledge base in the prompt, with prompt caching at 0.1× base input absorbing the dollars. It is rejected as a threshold on two grounds — the source bills resident tokens, so the guidance is vendor-interested rather than neutral ground truth; and the figure predates a tokenizer change costing ~30% more tokens for the same text, per the pricing page's own note. Frontloading-with-caching is carried as a competing posture, named and priced — for a small, cold, high-fire-rate corpus it genuinely competes. What the node does not do is quote a size. "Too big to frontload" is qualitative: the corpus that visibly cannot fit, or that degrades the window when it does.

2 · Query breadth is its own crossover dimension (ruled 2026-08-02), and does not fold into vocabulary match. Grep holds when you know what you are looking for; a broad search is a flood — an exploratory query turns each pass into a noise dump regardless of whether the words match. Vocabulary bounds reachability; breadth bounds admissible signal per search turn. A targeted query with the wrong vocabulary is unreachable; a broad query with the right vocabulary is a flood, and the two need different fixes.

3 · BM25's seat belongs to the measured stack, not to RAG at large (ruled 2026-08-03; ablation reading corrected 2026-08-06). Contextual embeddings produced the largest incremental reduction in Anthropic's ablation (5.7%→3.7%); adding contextual BM25 improved it further (3.7%→2.9%), and reranking reduced it again (2.9%→1.9%). The result supports grep's statistical cousin surviving inside this measured stack, not the earlier claim that the hybrid step produced its largest single gain. The memo carries its own counter-datum — NVIDIA's CORTEXA numbers, code-tuned embeddings at 71.95% recall against BM25's 40.67% on SWE-bench file retrieval (secondary, via a practitioner round-up). That does not refute the ablation; the two measure different things. It does show that BM25's standing is stack-dependent, which is exactly what a universal claim denies. The node states the contribution where it was measured and claims nothing about stacks nobody measured.

4 · The crossover is stated as dimensions, never as thresholds. Independent, vendor-neutral head-to-head evaluations at controlled corpus sizes do not exist yet. The crossover the node publishes is a synthesis across unequal sources — it earns the shape of the argument, not a curve.

5 · The RAG analogy is scoped by the fence. It holds for the verbatim-return portion of a pipeline, which admits. Chunk rewriting — contextual retrieval's preprocessing is a model writing new tokens at $1.02 per million document tokens — mints, which makes it distillation composed with deferral under ADR 0006. Production stacks are mixed; the classification is per portion, not per product name.

The two cost shapes

The economics the node carries reduce to a shape difference, and both halves are anchored in published prices checked 2026-08-02.

The searched corpus pays per query, in residency. No build, no hosting, no reindex; the entire bill is search turns admitted to the window at input rates and re-billed on every remaining turn. Order of magnitude: a 5-turn search admitting ~2k tokens per turn, in a session with ~10 turns still to run, is ~10k admitted tokens ≈ 100k token-turns ≈ $0.10–$0.50 per query at Haiku 4.5 through Opus 5 input rates ($1–$5 per MTok), before caching.

The indexed corpus pays per corpus, plus a floor. Embedding is trivia — $0.02–$0.18 to embed a 1M-token corpus, from OpenAI's text-embedding-3-small at $0.02/MTok to voyage-code-3 at $0.18/MTok. The bill lives in standing infrastructure: Pinecone serverless Standard runs storage at $0.33/GB/mo and reads at $16–18 per million read units against a $50/month minimum, and at skill scale the floor plus the pipeline you now operate — chunking, reindex-on-change, monitoring — is the whole cost.

So break-even is query-volume-driven, and at small scale the dollars were never the argument — which is why the one first-party decision on record was made on operational grounds. That is also why caching at 0.1× is load-bearing on the parent's arithmetic rather than a footnote: the competing frontload posture exists only because of it.

Consequences

• No threshold enters reference-data. Reinstating one needs a vendor-neutral measurement at controlled corpus sizes, not a restated vendor recommendation.
• Only the numbers with published method are quoted in-body: the 5.7%→1.9% ablation, the U-curve, and prices. Everything else travels as direction.
• The 2026 agentic-retrieval benchmarks stay cited for direction. A deep read is owed before any of their specific figures enter published prose.
• Claims sourced to a party with a commercial stake in the answer are labelled as such wherever they appear, in both directions — the vendor recommending residency and the vendor recommending vectors get the same treatment.

Related

• Reference data — the consuming node; these rulings bind its body.
• Deferred context — owns the fetch arithmetic and the caching section the cost shapes above price against.
• Cost relocation — owns the admits-vs-mints fence that scopes the RAG analogy.
• ADR 0006 — why a chunk-rewriting pipeline is relocation composed with transformation rather than a breach of either.
• The research memo, 2026-08-02 — the full workshop record: every source, the survey this record compresses, and the questions it left open.