A metric can be technically correct and still lie to you.
Pick a scenario below and watch a real evaluator judge it, including cases where the verdict is technically correct but still misleading.
- Evaluator
- An automated check that scores a model's output: pass or fail, or a number in between. Teams run evaluators across many examples to catch regressions before shipping a change.
- Exact match / contains / regex / JSON schema
- Four common, deterministic evaluator types used below. Each checks something different and specific: an identical string, a substring, a text pattern, or a JSON shape. Each has its own blind spots, which this lab lets you find.
- "Make the metric lie"
- An evaluator can be implemented correctly and still give a misleading verdict: passing something a human would flag, or failing something a human would accept. Watch for the scenarios marked below that demonstrate this directly.
Prompt
What is 12 × 4?
Model output (illustrative, not a live model call)
48
Expected (exact)48
PASS (exact_match)1.00
The output matches the expected string exactly, so exact_match correctly passes it.
Run every scenario's evaluator at once and see the overall pass rate. This is what a real eval suite run looks like, just with a handful of cases instead of hundreds.