What is a golden dataset and how do you build one for agents?
A golden dataset is your trusted set of test cases: representative tasks paired with the outcomes a correct agent should produce. It is the fixed benchmark you evaluate against every time something changes. You build it from real usage, cover the hard and unusual cases on purpose, and treat it as versioned, maintained data rather than a one-time snapshot.
What makes a dataset golden
The word golden means trusted and stable. Each entry is a task plus the expected outcome, reviewed carefully enough that you are willing to treat a failure against it as a real regression. Quality matters more than size. A hundred well-chosen cases beat ten thousand noisy ones.
How to build it
- Mine real usage. Production logs and support tickets are the best source of tasks that actually happen.
- Cover the edges deliberately. Include ambiguous requests, missing data, permission failures, and tasks that should be refused.
- Define the expected outcome at the level you can check, often the final state rather than an exact transcript.
- Version it alongside your code so you can see how the benchmark evolved.
Keep it alive
The most valuable new cases come from production. Every real failure the agent hits should become a new entry, so the same problem is caught automatically next time. Most eval platforms provide dataset management and let you attach expected outputs and run history to each case (see the tools guide).