Skip to content
AI LAB
5/5
LAB 05 / EVALS

A metric can be technically correct and still lie to you.

Pick a scenario below and watch a real evaluator judge it, including cases where the verdict is technically correct but still misleading.

Eval
A systematic test of how well an AI system performs against a defined set of tasks or criteria — usually many examples run at once, not just one.
Evaluator (or grader)
The method used to score one individual result within an eval: pass or fail, or a number in between. Evaluators come in several kinds — deterministic checks like the ones below (exact match, regex, schema validation), statistical metrics (precision, recall), model-based grading (another AI judges the output, often called "LLM-as-judge"), and human review. This lab focuses on deterministic evaluators specifically, because their exact behaviour (and exact blind spots) can be shown precisely — they're one part of the picture, not the whole of it.
Exact match / contains / regex / JSON schema
Four common deterministic evaluator types used below. Each checks something different and specific: an identical string, a substring, a text pattern, or a JSON shape. Each has its own blind spots, which this lab lets you find.
"Make the metric lie"
An evaluator can be implemented correctly and still give a misleading verdict: passing something a human would flag, or failing something a human would accept. Watch for the scenarios marked below that demonstrate this directly.

Prompt

What is 12 × 4?

Model output (illustrative, not a live model call)

48

Expected (exact)48
PASS (exact_match)1.00

The output matches the expected string exactly, so exact_match correctly passes it.

Run every scenario's evaluator at once and see the overall pass rate. This is what a real eval suite run looks like, just with a handful of cases instead of hundreds.