A metric can be technically correct and still lie to you.
Pick a scenario below and watch a real evaluator judge it, including cases where the verdict is technically correct but still misleading.
- Eval
- A systematic test of how well an AI system performs against a defined set of tasks or criteria — usually many examples run at once, not just one.
- Evaluator (or grader)
- The method used to score one individual result within an eval: pass or fail, or a number in between. Evaluators come in several kinds — deterministic checks like the ones below (exact match, regex, schema validation), statistical metrics (precision, recall), model-based grading (another AI judges the output, often called "LLM-as-judge"), and human review. This lab focuses on deterministic evaluators specifically, because their exact behaviour (and exact blind spots) can be shown precisely — they're one part of the picture, not the whole of it.
- Exact match / contains / regex / JSON schema
- Four common deterministic evaluator types used below. Each checks something different and specific: an identical string, a substring, a text pattern, or a JSON shape. Each has its own blind spots, which this lab lets you find.
- "Make the metric lie"
- An evaluator can be implemented correctly and still give a misleading verdict: passing something a human would flag, or failing something a human would accept. Watch for the scenarios marked below that demonstrate this directly.
Prompt
What is 12 × 4?
Model output (illustrative, not a live model call)
48
Expected (exact)48
PASS (exact_match)1.00
The output matches the expected string exactly, so exact_match correctly passes it.
Run every scenario's evaluator at once and see the overall pass rate. This is what a real eval suite run looks like, just with a handful of cases instead of hundreds.