Home Insights Evaluation Sets: How to Know Your AI Is Still Right
AI-ready data
Evaluation sets: how to know your AI is still right
Last reviewed: 29 September 2026
Key takeaways
- An evaluation set is a collection of realistic cases with verified answers, used to test AI before release and after every change.
- Without one, quality decays silently as models, prompts and data change.
- Start small — 50 to 200 well-chosen cases — and grow the set from real production errors.
An evaluation set is a curated collection of realistic inputs with agreed correct outputs, used to measure how well an AI system performs a specific task. It is to AI what a test suite is to software — and most organisations running AI in production don’t have one.
Why you need one
AI systems change even when you don’t touch them. Vendors update models. Your data and documents change. Users bring new kinds of requests. Prompts get tweaked. Each change can improve one thing and quietly break another. Without a fixed set of tests, you only find out when a customer or auditor does.
An evaluation set gives you three things: a release gate (does the system meet the threshold?), a regression check (did the latest change make anything worse?) and a common language with the business about what "good enough" means.
What goes into a good evaluation set
Typical cases
the everyday inputs the system sees most.
Edge cases
incomplete, ambiguous or unusual inputs.
Known failure cases
examples that have gone wrong before.
Adversarial cases
inputs designed to mislead the system, including instructions hidden in documents.
Out-of-scope cases
requests the system should decline or escalate.
Every language and market
the system serves, if multilingual.
Each case needs an agreed correct answer — or, for open-ended tasks, clear criteria for an acceptable answer — reviewed by someone who knows the domain.
How to score
| Task type | Typical measure |
|---|---|
| Extraction (orders, invoices) | Field-level accuracy; share of documents fully correct |
| Classification (triage, routing) | Accuracy; errors on high-impact categories |
| Question answering | Correctness against source; citation accuracy; refusal when unsure |
| Drafting | Rubric scores from reviewers: accuracy, completeness, tone, compliance |
| Agent tasks | Task completion; wrong actions taken; escalations |
Automated scoring works well for extraction and classification. For open-ended outputs, combine automated checks with human review of a sample, using a clear rubric.
Setting thresholds
Agree thresholds with the business owner before release, and make them proportionate to impact. A drafting assistant reviewed by people can tolerate more variation than an agent that posts transactions. Separate thresholds for high-impact errors — a wrong bank account is worse than a misspelled name.
Keeping the set alive
- Re-run the full set before every model, prompt, data or tool change.
- Add every significant production error to the set once it’s fixed.
- Refresh cases periodically so the set reflects current reality.
- Record results over time — they are valuable evidence for governance and audits.
Start small
You don’t need thousands of cases. Fifty to two hundred well-chosen cases, reviewed by a domain expert, catch most problems. Build the first set during the pilot, not after go-live.
Find out which of your data sources are holding AI back.
Frequently asked questions
Who should build the evaluation set?
Domain experts define correct answers; the technical team builds the scoring. Both are needed.
Can AI help build evaluation sets?
Yes, for generating candidate cases and variations — but people must verify the correct answers.
How often should we re-test?
Before every change, and on a schedule (for example monthly) to catch vendor model updates.