How do you test an AI agent for hallucinations?
You test for hallucinations by comparing what the agent says and does against a known ground truth, and by deliberately feeding it questions it should not be able to answer. A well-behaved agent cites real data and declines when information is missing. A hallucinating one invents a record, a field, or an answer, and that is exactly what your test set should be built to catch.
Two kinds of hallucination for agents
For a plain chatbot, a hallucination is a made-up fact. For an agent, it is worse: the agent can invent an action. It might claim it updated a record it never touched, reference an ID that does not exist, or report success on a call that actually failed. Testing has to cover both the words and the actions.
How to catch it
- Check against ground truth: verify the agent's claims against the actual state of the system, not against its own summary.
- Plant unanswerable tasks: include scenarios where the right response is "I do not have that information" and score whether the agent admits it.
- Verify every action: after a run, confirm each reported action actually happened and had the effect it claimed.
Tooling for faithfulness
Groundedness and faithfulness metrics are built into Ragas, DeepEval, Patronus AI, and Giskard, which can flag when an answer is not supported by the source material. Pair them with the action verification above so you catch both invented facts and invented actions.