Retrieval, tools and agents: giving a model information and actions

A model’s knowledge stops at its training data, which is months old by release and never included your company’s documents. And a model only produces text: it cannot look up today’s order status or run a calculation exactly. Two techniques close these gaps without retraining. Retrieval puts relevant information into the context at question time. Tool use lets the model’s output trigger programs whose results come back as text. Combined in a loop, they make an agent. This post explains each, with the prompting post as background.

Retrieval

Retrieval-augmented generation (RAG) answers a question in two steps: retrieve the passages most relevant to it from a document collection, then give them to the model with the question. Lewis et al. introduced the name for models that combine a model’s knowledge in its weights with a searchable index of Wikipedia, and set the state of the art on three open-domain question-answering tasks; they also noted the practical advantages: answers can cite their sources, and knowledge can be updated by changing the index rather than retraining (Lewis et al., 2020, abstract).

A retrieval pipeline has a few parts, each a design choice:

1. Chunking: split documents into passages small enough to fit several in the context, large enough to be self-contained.
2. Indexing and search. Lexical search (such as BM25) scores passages by shared words, weighted by how rare each word is. Dense search embeds the question and passages as vectors with an embedding model and finds the nearest ones, matching meaning rather than words. Many systems combine both.
3. Reranking (optional): a slower, more accurate model reorders the top candidates.
4. Generation: put the top $k$ passages in the context, usually with an instruction to answer only from them and to cite which one supports each claim.

Retrieval quality caps answer quality: if the right passage is not retrieved, the model cannot use it. So evaluate retrieval on its own, with recall@k, the fraction of questions whose relevant passage is among the top $k$ retrieved.

A tiny example shows why lexical search is not enough. Six help-centre passages, about refunds, shipping, warranty and so on, and four questions, searched with BM25:

Recall@1 is 2 of 4. A crude stemmer that strips plural “s” fixes the third question (3 of 4), but nothing lexical fixes the fourth: it means the same as the refund passage and shares none of its words. That is the case dense retrieval exists for, and why real systems measure recall on real questions before tuning anything downstream.

Tools

A tool is a function the model can ask to run: a search, a calculator, a database query, an API. The application describes each tool to the model (name, purpose, arguments, usually as a JSON schema); the model, post-trained for this, responds with a structured call instead of an answer; the application executes it and appends the result to the conversation; the model continues. The call is generated text, so constrained decoding can guarantee it is well-formed, but not that it is the right call.

Tools turn approximate abilities into exact ones (arithmetic, current data) and let the model act. They also extend what can go wrong: a wrong argument, a misread result, or a tool result containing text that tries to give the model instructions, the prompt-injection risk covered in the application-evaluation post.

Agents: reasoning and acting in a loop

An agent repeats: think about the next step, call a tool, read the result, decide what to do next, until the task is done. ReAct showed the value of interleaving reasoning traces with actions: on question answering and fact checking with a simple Wikipedia API it reduced hallucination compared with reasoning alone, and on two interactive benchmarks it beat imitation-learning and RL methods by 34 and 10 points of absolute success rate, prompted with only one or two examples (Yao et al., 2023, abstract).

The cost of a loop is compounding failure. If each step succeeds independently with probability $p$, a task of $n$ steps succeeds with probability $p^n$:

Try it: is an agent justified? A task takes 8 steps, each of which your model gets right 95% of the time. How often does the whole task succeed? What if each step’s result can be checked and retried once on failure?

Answer. $0.95^8 \approx$ 66%: one task in three fails, although every step looks reliable. With a check and one retry, each step fails only if both attempts fail, $0.05^2 = 0.25\%$, so the task succeeds $0.9975^8 \approx$ 98% of the time, assuming failures are detectable and independent. Verification at each step matters more than a slightly better model.

When is an open-ended loop justified? When the steps cannot be known in advance, because each depends on what the previous one found, as in research or debugging. When they can be known, a fixed workflow (retrieve, then draft, then check, in code) is cheaper, faster, easier to test and fails less. Start with the simplest structure that solves the task: a single prompt, then retrieval, then a fixed workflow, and an agent only when the task demands it.

Everyday picture: an assistant with a filing cabinet (retrieval) and a phone (tools). Sending them to fetch one file is reliable. Asking them to make ten calls, each depending on the last, to plan a trip, works only if they confirm each booking before making the next.

What to remember

• RAG retrieves relevant passages at question time; it adds current and private knowledge and makes citations possible.
• Retrieval quality caps answer quality: measure recall@k on real questions. Lexical search misses synonyms; dense or hybrid retrieval addresses that.
• A tool call is structured text the application executes; constrained decoding makes it well-formed, not correct.
• Agents interleave reasoning and tool calls; reliability compounds as $p^n$, so verification per step matters more than a slightly better model.
• Prefer a single prompt, then retrieval, then a fixed workflow; use an open-ended agent only when steps cannot be planned in advance.

Sources

• P. Lewis, E. Perez, A. Piktus, F. Petroni et al. (2020). “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” NeurIPS 2020. arXiv:2005.11401.
• S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan and Y. Cao (2023). “ReAct: Synergizing Reasoning and Acting in Language Models.” ICLR 2023. arXiv:2210.03629.
• S. Robertson and H. Zaragoza (2009). “The Probabilistic Relevance Framework: BM25 and Beyond.” Foundations and Trends in Information Retrieval 3(4).

Further reading

• Berkeley CS294-280 (Spring 2025), Yu Su, “On Reasoning, Memory, and Planning of Language Agents” (10 Feb), and the agent-security lecture by Dawn Song (28 Apr).