Evaluation & metrics

How do you evaluate an LLM agent?

Short answer

You evaluate an LLM agent by running it against a fixed set of representative tasks and scoring more than the final text. Grade whether it reasoned correctly, whether it called the right tools with the right arguments, and whether it actually completed the task. Because agents are non-deterministic, run each scenario several times and track how often it succeeds, not just whether it passed once.

Build a scenario set first

Evaluation is only as good as the tasks you run. Collect scenarios that mirror what the agent will really be asked to do, including the awkward ones: ambiguous requests, missing information, and tasks it should refuse. This set is your benchmark, and it is worth versioning like code. See OpenAI Evals and similar projects for how teams structure eval suites.

Score at more than one level

Reasoning quality: did the agent form a sensible plan and stay on task
Tool-call accuracy: right tool, right arguments, sensible order
Task success: did the end state match what a correct run should produce

A number of frameworks make it straightforward to define these checks and run them repeatedly. Braintrust is one common choice, and the tools guide compares the rest.

Account for non-determinism

The same input can produce different behavior on different runs, so a single pass tells you almost nothing. Run each scenario multiple times and report a success rate. An agent that completes a task 70 percent of the time is a very different thing from one that does it 99 percent of the time, and only repetition reveals which you have.

pythondef success_rate(scenario, trials=10):
    passed = 0
    for _ in range(trials):
        result = run_scenario(scenario)      # your harness runs the agent
        if scenario.check(result):           # end-state check, returns bool
            passed += 1
    return passed / trials

for s in scenario_set:
    rate = success_rate(s, trials=10)
    print(f"{s.name:30} {rate:.0%}")
    assert rate >= s.threshold               # each scenario sets its own bar

Choosing a framework, honestly

No tool wins outright. What fits depends on your stack and how much you value hosted convenience over control:

Promptfoo: open source and config-first, so it drops into CI cleanly and is strong for prompt comparison and red-teaming. The YAML-driven model gets awkward once your assertions need real program logic or stateful multi-step setup.
DeepEval: a pytest-style, assertion-first workflow with many built-in metrics, comfortable if you already think in unit tests. Several of its metrics lean on LLM-as-judge calls, which cost money, add variance, and need calibrating before you trust them.
LangSmith: tracing and evaluation that are tightly integrated if you are on LangChain or LangGraph. Most of that value assumes that ecosystem, and it is a hosted commercial product.
Braintrust: a hosted playground, dataset, and eval workflow that suits a team iterating on prompts together. The tradeoff is the same as any hosted option: your eval data lives in a third party.

A reasonable default: Promptfoo or DeepEval while you are wiring up CI, and a hosted platform later if you need shared dashboards and history.