The document is in the database. Why can’t the chat see it?
A customer asks about exchanging a product. The chat replies with terms from the old instructions, even though the new policy is already in the database. First instinct: change the model. But wait — perhaps it was given the wrong passage? A better model could inherit exactly the same problem.
That is an important detail of RAG. The program first searches for materials, then passes selected pieces to the model that writes the answer. Let’s examine both parts. Anthropic’s description of Contextual Retrieval highlights passages losing context when cut out of a document. A file being in the database does not yet tell us what the model read.
Separate two questions
Collect employees’ questions. For each, identify the correct document, passage and expected meaning of the answer. Include a product-code question, one requiring two sources and one with no answer in the database. Finding only easy sentences will not tell us enough.
The first evaluation column asks: did it find the right material? The second: did the answer preserve its meaning? If the policy never reached the results, we work on retrieval. If it did, but the model distorted it, we look at answer generation. The card below keeps that distinction clear even when several people make changes.
What does a number mean without its table heading?
Open the text extracted from the PDF. Look especially at tables. A number separated from its heading may lose its unit. “Above” may refer to a condition left in an entirely different passage. It is hard to write a sound answer from that material, even if the sentence itself sounds fine.
Then check document versions. For product codes, compare exact-word search with searching for similar meaning. Does the right code, name or date appear in the results at all? This helps decide whether to improve document preparation, retrieval or model instructions. One method need not fit every collection.
Show the user what supports the answer
Attach the source and state what you established. When documents conflict, show their dates and the difference. A polished paragraph should not pretend the conflict disappeared. If material is missing, the answer should say so directly.
After a change, return to the same questions. Record both fixed cases and new errors. An average will not explain why the old policy still sometimes beats the new one. Check accounts with different permissions too: a citation should help someone reach a document, but must not expose it to someone without access.
Sources and further reading
The sources behind this article. Instructions and example scenarios are suggestions from our editorial team.
- Anthropic: Contextual Retrieval (opens in a new tab)anthropic.com