What does the testing pyramid look like for AI agents?
The classic testing pyramid still applies to agents, adapted. At the base are fast unit tests of individual tools and functions using mocks. In the middle are integration tests that run the agent against a stateful sandbox to check real tool calls and short workflows. At the top are end-to-end evaluations of full tasks, run repeatedly and scored on success rate. Each layer catches a different class of bug, and you want many cheap tests at the bottom and fewer expensive ones at the top.
The three layers
- Unit (base): test each tool and helper in isolation with mocks. Fast, deterministic, run on every save. These confirm your code calls APIs correctly.
- Integration (middle): run the agent against a sandbox that holds state, so real tool calls and short workflows are exercised. These confirm the service accepts what the agent does.
- End-to-end (top): run full tasks through the whole agent, many times, and score success rate. These confirm the agent completes real work.
Why the shape matters
End-to-end agent runs are slow, costly, and non-deterministic, so you cannot rely on them alone. Push as much coverage as possible down into fast unit and integration tests, and reserve full evals for the scenarios that truly need them. This keeps your suite fast enough to run often, which is the only way it stays useful.
Where the tools fit
Unit tests use your normal test runner with mocks. Integration tests need a stateful environment, whether a vendor test mode or a cross-service sandbox. End-to-end evals run through a dedicated eval framework, of which there are many (see the tools guide).