How do you handle non-determinism in agent tests?
Agents are non-deterministic, so the same input can produce different runs. The fix is to stop asserting on exact outputs and start asserting on rates and invariants. Run each scenario many times, require a minimum success rate, and check properties that must always hold, like 'exactly one charge was created,' instead of matching a specific string.
Why exact-match tests break
Traditional tests assert that output equals an expected value. Agents defeat that immediately, because a valid run can phrase things differently, take a different but correct path, or call tools in a slightly different order. If your test demands one exact transcript, it will fail on perfectly good behavior and you will start ignoring it.
Assert on invariants instead
Shift from "did the output match" to "did the important things stay true." Invariants describe the outcome, not the route:
- Exactly one record was created, never zero and never two
- No action was taken outside the agent's permissions
- The final state is internally consistent
Measure rates, set thresholds
Run each scenario many times and require a passing rate, for example "succeeds on at least 95 percent of runs." Frameworks like Promptfoo support repeated runs and threshold-based assertions, which turns a flaky pass or fail into a stable signal that matches how the agent behaves in production.
A repeated-run reliability check
Run the same scenario N times against a fresh state, count the runs that satisfy an invariant, and assert on the rate:
pythondef test_refund_agent_reliability():
runs, passed = 20, 0
for _ in range(runs):
env = sandbox.reset() # fresh state each run
run_agent("refund my last order", env)
# invariant, not exact output: exactly one refund happened
if env.stripe.refund_count() == 1:
passed += 1
rate = passed / runs
assert rate >= 0.95, f"success rate {rate:.0%} is below the 95% bar"
One green run tells you nothing here. A rate over 20 runs does.