When a chunk is retrieved, can it actually answer the question?

A retrieval system ranks the chunks of a page by how close each one sits to the query, keeps the nearest few, and often drops anything below a similarity threshold. Either way, closeness is the signal the whole pipeline runs on: a chunk is near the query, so it is presumed useful. We tested whether that holds, by asking whether a retrieved chunk can answer its query on its own. Most cannot. How many is harder to pin down than we expected: a three-model panel put it at 89%, but when the operator hand-checked a sample, a human put it closer to 60%, and the two disagreed often enough that the gap became part of the finding. A chunk that stands on its own is a robust retrieval; a chunk that only makes sense beside the rest of the page is a fragile one, needing the retriever to fetch the missing pieces and the model to reassemble them. By either count, retrieval returns the fragile kind most of the time.

Standard retrieval metrics do not catch this. Recall@5 asks only whether a relevant chunk lands in the top five. It never asks whether that chunk, read on its own, contains the answer. The gap between “close enough to retrieve” and “enough to answer” is invisible to the metric, so it is invisible to most teams tuning a RAG stack. This study measures that gap directly.

What the field already knows

The gap between being retrieved and being able to answer is recognized in the research, but rarely measured head-on. RAGAS, the widely used RAG evaluation toolkit, scores context relevance as a topical match, not as whether a chunk can stand alone. Google’s Sufficient Context work (ICLR 2025) showed that RAG failures divide into two kinds, context that is insufficient and a model that fails to use sufficient context, and that the first kind is common and under-measured. CRUX made the parallel point for long-form answers: relevance ranking does not tell you whether the retrieved context covers what the answer needs. Two recent surveys of RAG evaluation catalog the standard retrieval metrics, relevance, precision, recall, and ranking, and none reports the number this study does, how often a retrieved chunk still cannot answer the query on its own.

It is also the middle of our own series. The Findability Study asked whether the right page gets retrieved. The Architecture Study asked whether a concept needs its own URL or just its own developed section. This study goes one level down: when a chunk is retrieved, does it answer the query? A planned follow-on retrieval case study will ask whether fixing the source structure closes the gap.

The setup

We reused the Findability Study corpus: 30 real pages spread across five content roles (explain, guide, convert, compare, evaluate), with 166 People Also Ask questions locked to those pages. These pages were chosen, in that study, because each under-develops a concept, which makes them the right test bed here: the proximity-sufficiency gap should be widest exactly where content is thin. So read the rate below as what thin content looks like to a retriever, not a web-wide average. Each page was chunked into roughly 512-token passages, embedded with text-embedding-3-large, and queried for its top five chunks (standard top-k retrieval). A three-model cross-family panel (Claude Haiku, DeepSeek V4 Pro, GPT-4.1-mini, majority of three) then judged each retrieved chunk, blind to its cosine score: if an LLM saw only this chunk, could it produce a correct, complete, grounded answer to the query?

We treat the cosine score as an analytical lens, not a gate. Production retrieval takes the top-k chunks regardless of absolute score, so we sort the retrieved chunks into cosine bands and report the false-positive rate, the share where the answer is no, in each. With this embedding model the scores are compressed: the median top hit scores 0.40, and across all 166 queries only 16 retrieved chunks (from 9 queries) exceed 0.60. So 0.50 is the densest band we can report on, and 0.60 is a sparse tail.

Finding 1: most retrieved chunks cannot answer on their own

By the panel’s count, the false-positive rate is high at every cosine band: 93% at 0.40, 89% at the primary 0.50, 81% at 0.55. It falls as the bar rises, exactly as it should, but never gets low where there are enough chunks to measure it well. By this measure, a chunk being near the query is a weak signal that it can answer the question by itself. But this is the panel’s strict reading, and the next finding is what happened when we checked it against a human.

Finding 2: how strict is “can it answer?”

The panel was asked whether a chunk could produce a correct, complete, grounded answer on its own. To see whether that bar matched a person’s, the operator hand-graded a 20-pair sample blind, answering the same question with the cosine score and the panel’s verdict hidden. They agreed only 65% of the time, short of our pre-registered 75% gate, and every disagreement ran the same way: the panel said a chunk could not answer where the human said it could. On those pairs the human accepted a chunk that substantially answered even when a detail was missing; the panel held out for completeness.

So the rate is bracketed, not pinned. By a human’s bar, closer to 60% of retrieved chunks cannot stand alone; by the panel’s, 89%. The direction is robust, both say most. But whether a chunk “answers” a query is a subjective call with no objective ground truth, and our human side is a single grader on 20 pairs, one careful reading rather than a verdict the panel failed against. A second person would draw the line somewhere else again. So treat 60% as a soft anchor, not a correction: the honest takeaway is a range and a disagreement, not a true number. We report the panel figures through the rest of this page because they cover the full corpus, with this gap as the health warning on every one.

Finding 3: the score sorts them only weakly where it counts

Here is the counterintuitive part. Cosine does predict answerability, with moderate strength overall: area under the curve is 0.76 at the primary threshold (95% interval 0.63 to 0.91, above chance). But that power lives in a range almost nothing reaches. Above 0.60 a higher score really does mean a better chunk. Below it, in the 0.40 to 0.60 band where essentially all real retrieval happens, the score is flat: a 0.58 chunk is no likelier to answer than a 0.42 one. So in the range that matters, a higher score barely moves the odds. And you cannot filter your way to the good chunks: keeping only those above 0.60 would throw away all but 16 of 302 retrieved chunks.

Finding 4: sufficiency is distributed, not concentrated

No single chunk answers the query 81% of the time, but the full top-5 set answers it 42% of the time. Both numbers are measured over the same five retrieved chunks, so the comparison is fair, and some of the gap is simply that five chunks carry more text than one. The sharper number is the orphan rate: for 25% of queries the answer exists only across the set and is missing from every chunk on its own. There retrieval genuinely assembles an answer that no single chunk holds, which is why a pipeline that grades or reranks chunks one at a time is measuring the wrong unit.

Finding 5: why a chunk falls short

The failures are not random. The two largest are partial answers, where the chunk holds half of what the query needs and the rest is elsewhere on the page, and topic matches with no specific answer, where the chunk is about the right subject but never states the thing asked for. Both are the fingerprint of a page that names a concept without developing it: enough vocabulary to score cosine-near, too little substance to stand alone.

What this means for your content

AI does not read your page. It reads a slice of it, roughly one section at a time, and tries to answer from that slice alone. Four things follow for anyone writing or structuring the pages it reads.

1. 01Every section has to stand on its own. Most retrieved slices could not answer the question on their own (a human judged about 60%, our model panel 89%), usually because they leaned on the rest of the page for context the reader never received. Write each section so it answers its own question without assuming what came before it.
2. 02Mentioning a concept is not the same as covering it. The most common failures were sections that were on the right topic but never stated the specific answer, or that only introduced the idea. Surface vocabulary is enough to get you retrieved. Only a complete explanation is enough to get you used. A page can rank for a term and still answer none of the questions behind it.
3. 03Keep each answer in one place. For 25% of questions the answer existed only when several parts of the page were stitched together, and retrieval often will not do that stitching for you. If the answer to a question is scattered across the page, gather it into one self-contained section.
4. 04You cannot fix this by getting closer to the query. The failing sections were already close to the question, that is why they were retrieved. Adding the keyword again just moves you nearer in the same way that already failed. The lever is depth, not proximity: a concept is done when one section answers its question end to end. That is the kind of done ContentGrapher is built to flag, the underexplained and weakly integrated concepts that read as present but cannot answer.

Tuning the retrieval system, the threshold, the reranker, the chunk size, can move these numbers a little, but it cannot put an answer into a section that does not contain one. The content can. It is the one lever that acts before retrieval is ever attempted.

What this study does not claim

1. 01It is not a random sample of the web. The corpus was selected for an earlier study because the pages under-develop a concept, so the absolute false-positive rate here is closer to an upper bound than a population rate. The shape of the result, falling with threshold and flat within the band, is the transferable part.
2. 02The single-chunk bar is strict. A chunk that holds part of the answer is counted as a false positive, by the pre-registered rule. A looser bar would lower the rate; the partial-answer share is reported so you can re-draw the line.
3. 03The absolute rate is judge-dependent, and the panel did not pass human calibration. On a 20-pair blind sample, panel-vs-human agreement was 65% (gate: 75%), with the panel consistently stricter. We report the panel rate as a strict upper estimate, bracketed by the human’s looser reading near 60%, not as a settled number. And the human side is itself one operator’s subjective judgment on 20 pairs, not a second-annotator-validated truth; whether a chunk "answers" is a fuzzy call, so the bracket reflects genuine disagreement about the construct, not a measurement error. The direction, that most retrieved chunks fall short, is what survives.
4. 04It measures one embedding model and one chunking. Other models compress the cosine range differently; where the bands fall is model-specific.
5. 05It measures retrieval sufficiency, not generation. Whether an LLM uses a sufficient set correctly is a separate failure mode and out of scope.