Why RAG Retrieval QualityFails, and How toFix It

A note on bias first: Darvi Labs builds retrieval systems for a living, so we have an interest in this topic. Everything measured below comes from public sources you can check yourself.

When a RAG system answers vaguely or confidently wrongly, the generator is usually not at fault. The model can only reason over what you put in its context window, and most production failures are cases where the passage that held the answer was never retrieved at all. Swapping in a stronger model does not fix a retrieval miss; it just makes the wrong answer more fluent.

Retrieval quality is a stack, not a setting. Keyword and vector search fused together, chunks that carry enough context to be findable, a reranker that actually reads the candidates, and a labelled evaluation set that tells you which of those changes helped. Teams that skip the last part end up tuning blind.

A single-vector embedding compresses a passage into a few hundred or a few thousand numbers. Nearest-neighbour search over those vectors finds what is topically similar to the query. Topical similarity is not the same thing as containing the answer, and the gap between the two is where most retrieval bugs live.

This is a structural limit, not only a quality gap that better training will close. The paper On the Theoretical Limitations of Embedding-Based Retrieval, published at ICLR 2026 and available as arXiv 2508.21038, shows that the number of top-k document subsets a single-vector retriever can return is bounded by the embedding dimension. That bound holds even when the embeddings are freely parameterised and optimised directly on the test set, which rules out the usual explanation that the model simply needs more data. The authors built a deliberately simple dataset, LIMIT, to stress this property, and state-of-the-art embedding models fail on it.

The everyday symptoms are recognisable once you look for them. Negation is the classic one: a query for contracts that do not contain an arbitration clause embeds very close to contracts that do, because almost every word is shared. Exact identifiers are another, where an error code, a SKU, a migration number or a version string needs a literal match and semantic neighbourhood is pure noise. A third is any question whose answer is assembled from several documents rather than sitting in one.

The practical conclusion is not that vector search is bad. It is that a vector index should be treated as one signal among several, with known blind spots, rather than as the search engine.

Separate your metrics or you will not know what you changed. Retrieval has its own numbers: recall at k, meaning whether the passage you needed made it into the candidate set at all, and a rank-aware measure such as nDCG at 10 for whether it landed near the top. Answer quality is a downstream question and mixing the two hides which half is broken.

Anthropic's contextual retrieval write-up uses a framing worth borrowing: express retrieval failure as one minus recall at 20, the share of relevant chunks that did not appear in the top twenty. It gives you a single number to watch across changes, and it is honest about what it does not capture, since a chunk retrieved at rank nineteen still has to survive reranking and the generator's attention.

For scoring the ordering, the Ragas library's context precision is the average of precision at k across the retrieved chunks, and it is rank-aware by design. In their worked example, moving an irrelevant chunk from second place to first drops the score from roughly 1.0 to roughly 0.5. Their context recall covers the complementary question of whether everything the reference answer needed was actually retrieved. Both come in LLM-judged and non-LLM variants, the latter comparing retrieved chunks against reference contexts by string similarity or by matching context ids, which is cheaper and fully deterministic.

Build that evaluation set from real queries, not invented ones. A hundred to two hundred labelled examples drawn from what users actually typed will tell you more than any public leaderboard. Leaderboards are genuinely misleading here, and the field now admits it: Hugging Face introduced RTEB specifically because MTEB scores inflate when a model's training data overlaps the evaluation sets, a problem they call teaching to the test. RTEB keeps part of its datasets private, evaluated by the maintainers, reports nDCG at 10, and treats a large drop between open and private performance as a sign of overfitting. They also point at zero-shot score, the proportion of the evaluation set a provider has said it trained on, and note that a model slightly lower on the public ranking but higher on zero-shot score is often the better bet.

Lexical and dense retrieval fail in different directions. BM25 handles rare terms, identifiers and exact phrases well and is helpless when the user's vocabulary differs from the document's. Dense retrieval handles paraphrase well and is weak on exactly the cases BM25 covers. Running both and merging the results is the single highest-return change most teams can make, and it needs no new model.

The merge is where implementations go wrong. Combining raw similarity scores from two systems means normalising numbers that are not on the same scale and whose distributions shift per query. Reciprocal rank fusion sidesteps this by ignoring scores entirely and using only positions. The formula from Cormack, Clarke and Buettcher's SIGIR 2009 paper is RRFscore(d) equals the sum over all rankings of 1 divided by (k plus r(d)), where r(d) is the document's rank in that ranking. They set k to 60 during a pilot and never tuned it afterwards, on the reasoning that high ranks should dominate without lower ranks becoming negligible, and that the constant mitigates the impact of high rankings by outlier systems.

The result holds up. On the LETOR 3 dataset the paper reports RRF reaching a mean average precision of 0.6051 against 0.5917 for Condorcet Fuse and 0.5846 for ListNet, the best individual method, and across their TREC experiments RRF beat Condorcet Fuse in every case. Seventeen years on it is still the default because it is three lines of code and has nothing to tune.

If you are on PostgreSQL you can run both halves in one database: a tsvector column with a GIN index for the lexical side, pgvector for the dense side, and the fusion in application code or a CTE. One version detail matters in practice. pgvector 0.8.0 added iterative index scans, controlled by hnsw.iterative_scan with hnsw.max_scan_tuples, and ivfflat.iterative_scan with ivfflat.max_probes, to fix over-filtering: a query with a WHERE clause where the approximate index scan finds too few rows matching the filter and silently returns fewer results than you asked for. Anyone adding a tenant id or a permission predicate to a vector query has met this bug, usually without realising it. The same release improved the planner's estimate of when to use an ANN index at all, and the project's own advice is worth repeating: if exact search is fast enough on your data, prefer it, because it gives 100 percent recall by definition.

A chunk cut out of a document loses the things that made it findable. A sentence reading that revenue grew three percent over the previous quarter is nearly useless on its own, because the company, the quarter and the business unit all lived in surrounding text that went into a different chunk. The embedding faithfully represents an ambiguous fragment.

Anthropic's contextual retrieval addresses this by generating a short, document-aware context for each chunk and prepending it before both embedding and lexical indexing. Their reported numbers, averaged over codebases, fiction, arXiv papers and science papers using Gemini Text 004 embeddings, start from a baseline failure rate of 5.7 percent on one minus recall at 20. Contextual embeddings alone brought that to 3.7 percent, a 35 percent reduction. Adding a contextual BM25 index took it to 2.9 percent, a 49 percent reduction. Layering reranking on top reached 1.9 percent, a 67 percent reduction overall. The one-time preprocessing cost they quote is about 1.02 US dollars per million document tokens under their assumptions.

Their caveats are as useful as the headline. Generic document summaries produced very limited gains, so the context has to be specific to the chunk. Domain-tailored prompts, for instance ones carrying a glossary, may do better than the generic one. Gains varied by embedding model. Top-20 retrieval outperformed top-5 and top-10 in their tests. And if your entire corpus is under roughly 200,000 tokens, they suggest putting it in the prompt with caching and skipping retrieval altogether, which is the kind of advice that saves a quarter of engineering time.

The broader point is that chunk size, boundary placement and overlap are parameters you own and must measure, not framework defaults to inherit. A splitter that cuts mid-table or mid-function will cap your recall no matter which embedding model sits behind it.

First-stage retrieval should be tuned for recall: get the right passage into the candidate set, accept noise. Precision is the reranker's job. A cross-encoder reads the query and a candidate passage together in one pass, so it can model interactions between them that a bi-encoder, which must embed each side independently and in advance, structurally cannot. The cost is one model call per candidate, which is why it runs over a shortlist rather than the corpus.

The shape of the pipeline is consistent across published setups. Anthropic's configuration scored the top 150 retrieved chunks with the Cohere reranker and kept the top 20 for the prompt. Retrieve wide, rerank, pass few. The numbers to tune are the shortlist size and the final count, and both are latency decisions as much as quality ones.

When a cross-encoder does not fit the latency budget, late interaction is the middle path. ColBERT, described in arXiv 2004.12832, encodes query and document separately so document representations can be precomputed offline, then scores relevance with a cheap fine-grained comparison at query time. Against the BERT-based rankers it compares with, the paper reports roughly two orders of magnitude lower latency and about four orders of magnitude fewer FLOPs per query, at competitive effectiveness. That is the trade worth understanding before concluding that good reranking is unaffordable.

Whichever you pick, put a number on it. A reranker over 150 candidates is real milliseconds and a real line on the invoice, and it is the stage most likely to be left at a tutorial default. Measure the recall and nDCG gain against the added latency, and be willing to find that a shortlist of 50 buys most of the benefit.

Agentic AI engineering and scalable application architecture are the two things we concentrate on at Darvi Labs, a software engineering company in Kathmandu. Retrieval sits squarely between them, which is part of why it is so often mishandled: the index is a database and infrastructure problem, while relevance is an evaluation problem, and the two are usually owned by different people.

The way we approach it is deliberately unglamorous. Measure retrieval on its own before changing anything, treat hybrid search as the default rather than an optimisation, keep an evaluation set built from real queries, and make changes pass it before they ship. More of our engineering writing, including an account of building a production MCP server against the stateless specification, is on our Blogs page, and examples of delivered work are on our Portfolio page.

If you are debugging a RAG system that answers confidently and wrongly, or planning one and would rather not learn these lessons the expensive way, get in touch through our Contact page. We are glad to start with a small, measurable pilot against your own data.