A single accuracy number is a lie. OpenAI, Anthropic, and Google all ship models behind a wall of evals — golden sets, LLM-as-judge, behavioral tests, red-teaming, A/B tests with real users. Without that wall, you ship a model that scores 92% on your test set and silently fails on the 8% your users care about most.
Learning Objectives
After this lesson, you will be able to:
Design a layered evaluation plan that combines automated metrics, AI-based scoring, human review, and behavioral tests
Run A/B tests for ML models the right way -- with proper statistics and without common mistakes like peeking at results too early
Build a red teaming process that systematically finds model failures before your users do
Evaluation might not sound as exciting as training a model, but it is the skill that separates models that work in demos from models that work in the real world. Master this, and you will never ship a model that silently embarrasses you.
Unit & Integration TestsCode correctness (not model quality)
Try it: Adjust the evaluation thresholdInteractive
Loading visualization...
Each layer catches different types of failures. Skip a layer and you have blind spots.
Try it! Pick any classifier you have (even a simple if "positive" in text rule). Write 5 normal test cases and 5 tricky edge cases (negation, sarcasm, ambiguity). Run them through and count how many pass. You will be surprised how quickly "95% accurate" models fail on adversarial inputs.
#Layer 2: Behavioral Testing (CheckList / Unit Tests for Models)
Behavioral tests define specific input-output expectations, like unit tests for models:
Invariance tests: Output should not change when input is paraphrased.
Invariance Test: Paraphrases Should Match
Input
Expected Output
Constraint
What is the capital of France?
Paris
Baseline
Tell me the capital city of France
Paris
Must match
France's capital is what?
Paris
Must match
Directional tests: Changing input in a specific way should change output predictably.
Directional Test: Negation Should Flip Sentiment
Input
Expected Output
Constraint
Rate this review: Great product!
Positive
Baseline
Rate this review: Terrible product!
Negative
Must flip
Minimum functionality tests: The model must handle basic cases.
Minimum Functionality Test: Basic Cases
Input
Expected Output
2 + 2
4
What color is the sky?
blue (or similar)
pythonplayground.py · Pyodide
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
Tests · Run the full behavioral test suite. Identify which test types have the highest failure rate. The robustness tests should reveal model fragility.
Use a powerful LLM (GPT-4, Claude) to evaluate outputs from the model being tested:
Pairwise comparison: "Which response is better, A or B?"
Pointwise scoring: "Rate this response 1-5 on helpfulness, accuracy, and safety."
Reference-based: "Does this response match the reference answer? Score 1-5."
Why it works: LLM judges correlate 80-90% with human judgments on many tasks, at 100x lower cost and 1000x higher speed.
Where it fails
LLMs have positional bias (prefer the first response in A/B comparisons)
LLMs are verbose -- they may prefer longer responses regardless of quality
LLMs may not catch domain-specific factual errors
Self-evaluation (using the same model as judge and generator) has obvious bias
Best practices
Randomize position in pairwise comparisons (swap A and B, average scores)
Use a stronger model as judge than the model being evaluated
Calibrate against human annotations on a subset
Use structured rubrics, not open-ended "is this good?"
What Do You Think?
You use GPT-4 as a judge to evaluate Claude's outputs. GPT-4 rates Claude's responses an average of 3.2 out of 5. Is this a reliable evaluation?
Cross-model evaluation can introduce systematic bias. Always calibrate your LLM judge against a gold-standard set of human annotations. The LLM judge's value is in scaling human-calibrated evaluation, not replacing it.
The gold standard for measuring business impact: expose real users to two model versions and measure outcomes.
Statistical rigor requirements
Pre-define: sample size, test duration, primary metric, and success criteria before starting
Random assignment: users randomly split into control (old model) and treatment (new model)
Sufficient sample size: use a power analysis to determine how many users you need
No peeking: do not check results daily and stop when they look good (inflates false positive rate)
Multiple comparison correction: if testing multiple metrics, apply Bonferroni or FDR correction
CUPED / variance reduction: use pre-experiment covariates (historical user behavior) to shrink required sample size by 20-50%
Interleaving for ranking systems: when comparing rankings (e.g., search models), team-draft interleaving needs 10-100x fewer users than parallel A/B for the same statistical power.
Before any traffic-split A/B, run the candidate in shadow mode — duplicate every production request to the new model but only the control model's output is returned to the user. The candidate's output is logged for offline comparison.
Use shadow mode to:
Verify the candidate matches or exceeds the control on a representative live traffic distribution (not a static eval set).
Catch latency, error-rate, and cost regressions before any user sees them.
Build a labeled comparison corpus you can later score with LLM-as-judge.
Shadow then canary then full A/B. Skipping shadow is the most common reason an A/B test gets killed at day 2 for guardrail violations.
#Sizing an A/B Test (the math you should be able to do)
Most ML A/B tests are underpowered because no one runs the sample-size calculation. Do it explicitly.
Multi-layered evaluation covers what single metrics miss. Combine automated metrics (BLEU, ROUGE), LLM-as-judge scoring, human evaluation, and behavioral testing for comprehensive quality assessment
A/B testing is the gold standard for production evaluation. Statistical rigor (proper sample sizes, significance testing, no peeking) is essential to avoid false conclusions about model improvements
Red teaming finds failures before users do. Systematically probing for adversarial inputs, edge cases, bias, and safety violations reveals weaknesses that standard benchmarks miss
Behavioral testing checks specific capabilities. Like unit tests for software, behavioral tests verify that the model handles specific input patterns correctly (negation, ambiguity, format changes) regardless of overall accuracy
With a robust evaluation strategy, you can deploy models with confidence. Next up: Observability & Monitoring -- how to keep your models healthy in production after deployment.