Lead: When an AI answers wrongly, the problem is often the material it was handed. This preprint replaces the step that fetches that material.
A strong model cannot use evidence the retriever never found
Most question-answering systems today use RAG, retrieval-augmented generation. A retriever pulls a few passages from a document store, and a language model writes the answer from them.
Ask such a system, "Who is the main character in Christopher Nolan's film from the year 2000?" A keyword retriever may rank the page for Gladiator, another 2000 film, above the page for Memento, which holds the answer. Chao Wan and colleagues at Cornell call this a capability asymmetry. Generators keep getting stronger, while the retrievers that decide what they see still count words or compare fixed embeddings. Whatever the retriever misses, the generator cannot recover.
At a glance: DecisionRAG picks material level by level, from topics to pages to paragraphs
DecisionRAG navigates the corpus instead of searching it
DecisionRAG drops the first-stage retriever and lets a decision model choose which branches to open, one level at a time.
Think of how you use a library: rather than reading every page of every book, you walk to the film section, pull a few books about Nolan, and only then turn to the page you need. DecisionRAG follows the same order. The corpus is arranged as a tree of topic clusters, pages, and paragraphs. At each level a decision model judges which branches are worth exploring. Branches it rejects are pruned, and their paragraphs are never scored.
A decision model does not write text. You give it the question and a list of candidates, and in a single forward pass it returns a probability that each candidate is relevant. Because every candidate is judged against the same shared context, like books laid side by side on a desk, the scores can be compared directly. On MultiDoc2Dial, scoring passages together instead of one at a time raised top-1 recall by 7.7 percentage points.
Figure 1: How DecisionRAG narrows the search on Natural Questions, level by level
2.68 million Wikipedia paragraphs in about 1.7 seconds
Speed depends on how many levels the tree has, far more than on how much text it holds.
For the Natural Questions benchmark, the authors grouped 2.68 million Wikipedia paragraphs into 1,000 topic clusters. The model kept 150 clusters and read only 16% of the page titles. Compared with dense retrieval followed by an 8-billion-parameter reranker, DecisionRAG ranked the gold paragraph first 17.4 percentage points more often. Its median retrieval time was about 1.7 seconds per query.
The authors also replaced the decision model with GPT-5 writing out, token by token, which branches to keep. Accuracy was mixed across the two benchmarks they tried, but that version was 34 to 145 times slower.
Figure 2: Accuracy and latency of three retrieval approaches on Natural Questions
There are clear limits. This is a preprint that has not been peer reviewed, and the main experiments rely on Jev, a commercial decision-model service. Pruning cannot be undone: if the cluster holding the answer is dropped, that answer is gone for the query. Poorly organized documents or vague titles also hurt results. GPT-5 reranking still scored higher on ConditionalQA top-1 accuracy, and on Natural Questions it cost less per query. The systems were timed on different hardware, and the authors state that the speed ratios are not controlled comparisons.
Further reading: who arranges the shelves, and what tidying costs
Three other recent preprints look at how the shelves get arranged and what reorganizing information can cost.
Librarian Agents trains an agent to reorganize documents into folders before any question arrives, which is like arranging the shelves in advance. On 50 held-out BrowseComp-Plus tasks it beat the strongest baseline by 33 percentage points. DecisionRAG works the other way round: it takes whatever structure already exists and navigates it when the question arrives.
Memory Tax tested agent memory systems on three personal-assistant conversation benchmarks and found that many of them did worse than plain RAG over the raw history. Reorganizing information can lose or obscure the evidence. DecisionRAG's cluster summaries are also a kind of reorganization; a poor summary can get a whole branch pruned by mistake.
Beyond Token Savings studied context compression in coding agents. On Terminal-Bench with a Qwen model, policies that used about a third of the tokens could still take 20% to 80% longer, partly because of the extra summarization calls. DecisionRAG's latency, too, depends mainly on how many sequential calls it makes, not on how many tokens it reads.
The next time an AI gives you an off-target answer, check which passages it was given. The cause is often there.