What metrics matter for agent testing?
The most useful agent metrics measure outcomes, not vibes. Task success rate tells you how often the agent finishes the job correctly. Tool-call accuracy tells you whether it uses tools properly. Workflow completion tells you if it handles multi-step tasks end to end. Recovery rate tells you how it does when something fails. Cost and latency tell you if it is viable. Track each across many runs, because averages over repeated attempts are what predict production behavior.
The metrics worth tracking
- Task success rate: the share of runs where the final state is correct. This is the headline number.
- Tool-call accuracy: how often the agent picks the right tool and passes valid arguments.
- Workflow completion rate: how often a multi-step task finishes without stalling or corrupting state.
- Failure recovery rate: when an action is rejected or times out, how often the agent recovers gracefully.
- Cost and latency: tokens and time per task, which decide whether the agent is viable at scale.
Why single-run numbers lie
An agent can nail a task once and fail the same task the next time, so any metric taken from one run is noise. Report every metric as a rate over repeated runs. Observability tools like Langfuse, Helicone, and Arize Phoenix help you capture these numbers on both test and live traffic.
Watch the trend, not just the number
The point of tracking metrics is to catch regressions. A prompt change or model upgrade that quietly drops task success from 92 to 78 percent is exactly the kind of thing that slips through manual review. Store the numbers over time so you can see a drop the moment it happens.