Testing ML Systems: Start Here

#Why a Running Program Can Still Be Wrong

Think of a teacher marking a class. A test that checks a calculator is simple: press 2 + 2, the screen must say 4, always. If it says 5, something is broken, and the same press always gives the same answer.

A model is not like that. There is no single right answer to "will this customer leave?" for every case. The model gives a best guess, and the guess is sometimes wrong even when everything works. So "no errors" tells you very little. You need other questions, and each one is a kind of check.

Picture a school hiring a new teacher to replace a retiring one. You would not ask only "did the teacher arrive on time?" You would ask three different things. Are the textbooks and attendance sheets in order? Do the students do at least as well in a trial exam as with the old teacher? And in a few classroom situations where you know the right behaviour, does the new teacher do the sensible thing? These three questions map onto the three checks in this lesson.

In one sentence: a model has no single right answer per case, so you need several different checks, not only the absence of errors.

#Three Kinds of Check

We keep the same tiny customer model from the last lesson: score = 2 x calls + 3 x late - 0.5 x months, and a score at or above the cutoff means "leaves". A teammate has trained a new candidate. Before it replaces the one in production, run three checks.

Two of five rows have a problem. If you train on them without looking, the model learns from numbers that cannot be true. This check does not ask anything about the model. It asks about the food before cooking.

Notice what check 3 did that check 2 could not. The held-out set may hold few customers with late payments, so a wrong sign can hide there. A small hand-made pair finds it at once.

In one sentence: check the data before training, the score on unseen cases after, and the behaviour on pairs where you know the right direction.

#The Gate

Trace the gate on the candidate. Suppose both models get 0.8 on the held-out set.

• Floor: 0.8 is at least 0.70, pass.
• Not worse than production: 0.8 is at least 0.8, pass.
• Behaviour: one more late payment lowered the score, fail.

One failed check is enough. The gate says BLOCK and prints "one more late payment lowered the leaving score". A person reads the reason, finds the sign problem, retrains and tries again. Without the behaviour check the candidate would have shipped with an equal score and a backwards rule, and nobody would have known until customers behaved oddly.

In one sentence: a gate runs every check automatically and blocks the model with a reason if any one fails.

#Run the Gate

Do: Run the cell. Change the candidate's late weight from -1 to 3 and run again. Then, keeping 3, set cutoff to 10 and run a third time.

Notice: First run: two data problems, both accuracies 0.8, and the gate says BLOCK because of the late-payment pair. Weight 3: no behaviour failures, accuracy 1.0, PASS. Cutoff 10: the candidate says "stays" for everyone, accuracy 0.5, blocked twice over.

Why: Each check catches a different mistake, and only the gate turns them into one decision. The weights are made up, so only how the checks combine matters.

# Copy to Colab (needs the pytest package and your own model code; it cannot run in the browser)
# The same checks written as tests. Each function that starts with test_ is one check.
from my_model import CURRENT, CANDIDATE, accuracy, behaviour_problems, data_problems, training_rows


def test_training_data_is_sane():
    assert data_problems(training_rows) == []


def test_candidate_clears_the_floor():
    assert accuracy(CANDIDATE) >= 0.70


def test_candidate_not_worse_than_production():
    assert accuracy(CANDIDATE) >= accuracy(CURRENT)


def test_candidate_behaves_sensibly():
    assert behaviour_problems(CANDIDATE) == []

In one sentence: the gate is a few lines of code, and a test library only runs those same checks automatically on every change.

#Checks That Fool You

A check can pass for the wrong reason. Three slips to know now.

• A floor set too low. If the floor is 0.50 and the two answers are balanced, a coin flip meets it. Set the floor from the production model's score and from what the business can accept, not from what is easy to pass.
• A test set that is too small. With 10 held-out cases, one case is 10 points of accuracy. A model that gets 8 right and a model that gets 7 right are not clearly different. Use as many cases as you can, and treat small differences as noise.
• Leaking the answers. If any held-out case also appears in the training table, the score is flattering. Keep the held-out set away from training, always.

Here is an extra worth knowing. A passing gate is not a promise. It lowers the chance of a bad release, and the monitoring from the last lesson still has to watch the model after it ships. Checks before, alerts after: both are needed.

In one sentence: set the floor from a real need, keep the test set large and separate, and remember that a passing gate lowers risk without removing it.

#Where to Go Next

You can now read three checks and a gate, and say why a tied score can still be blocked.

• Next, the story: AI Ops History & Future tells how the field arrived at habits like these.
• The full loop: The ML Production Lifecycle shows where each check sits in the whole cycle.
• Putting gates in automation: MLOps: CI/CD for ML covers running checks on every change.
• Go back a step: MLOps: Start Here if the six stops still feel new.

In one sentence: read the history lesson next, then the full lifecycle, and come back to this gate when you reach CI/CD.

#Key Takeaways

• A model has no single right answer per case, so no errors is not the same as good
• Three kinds of check cover three kinds of mistake: the data, the score on unseen cases, and the behaviour on pairs
• A gate runs every check and blocks the model with a reason if any one fails
• A tied score can still hide a backwards rule, which only a behaviour check finds
• A floor set too low, a test set too small, or leaked answers can all make a check pass for the wrong reason

#Quick Check