An assistant can change its answer without a model upgrade or an application deployment. A source page moved. A chunking rule changed. A retired document remained in the index. If the release record names only the code and model, it misses the inputs that explain what the user is seeing.
Record the inputs that shape an answer
For a hypothetical product-support assistant, record the approved collection identifier, source revisions, indexing configuration, retrieval settings and prompt revision. Link those references to the evaluation run. The purpose is to distinguish two candidates and explain their behavior.
Copying every retrieved passage into general application logs creates a different problem. Those passages may contain the information access controls are supposed to protect. Prefer identifiers and narrowly scoped diagnostic evidence. Decide deliberately who can inspect any content retained for a failure investigation and how long it is needed.
Remove withdrawn content from answers
Removing a source file is only the first step in an illustrative withdrawal test. Inspect the index, cached responses, generated summaries and any other copy that can supply the answer. Ask the same question again under the identities that used to have access.
OWASP’s RAG security guidance connects provenance, access inheritance, cache boundaries and content removal. These are distinct responsibilities. Knowing which document supplied an answer helps an investigation, but provenance alone does not stop a withdrawn document from being used.
Source: OWASP · RAG Security Cheat Sheet.
Test each context change
Create a synthetic collection with a current procedure, an obsolete procedure and a restricted appendix. Keep a fixed set of questions. Run it before and after each change so that a retrieval difference can be distinguished from a generation difference.
- Update the current procedure: does the answer use the intended revision?
- Withdraw the obsolete procedure: can it still influence an answer?
- Remove access to the appendix: do snippets and cached answers respect the change?
- Restore an earlier index: does the restoration also restore material that should remain unavailable?
Handle questions the sources cannot answer
A missing source is a product state, not an invitation to improvise. The application might ask a narrower question, explain that the approved material does not cover the request or return the person to an existing support path.
Include that state in the evaluation. An assistant that admits a gap can be useful. An assistant that fills the gap with an authoritative-sounding procedure can turn a routine content update into an operational problem. The release decision needs to account for both.