What Is a Model? Learning a Rule from Examples

#A Rule You Already Use

Think about how you pick a coat. You look outside, feel how warm it is, and think: warm means no coat, cold means a thick coat. Nobody gave you a formula. Over the years you saw many days and many coats, and your head built a rule that turns weather into a choice.

Here is a small, real-feeling case. Mina runs an ice cream stand. At the end of each day she writes down two things: the afternoon temperature, and how many cones she sold. After ten days her notebook looks like this.

Tomorrow the forecast says 28 degrees. How many cones should Mina prepare? Before you read on, make a guess.

Your head just did the job of a model. In the next section we will write that job down as a rule you can check with a pencil.

A hand-written rule has a weakness that learning avoids. The next demo sets the two side by side on a different job, telling dogs from other animals. It is a scripted illustration, a drawn stand-in and not a trained model, so its accuracy numbers are scripted too.

Do: On the Rule-Based tab press Play and watch the five tricky animals go through the if-then questions. Then switch to the Machine Learning tab and press Play again to see the accuracy bar climb and the same five animals get answered.

Notice: The rules are fixed questions about legs, fur and barking, so an unusual dog or a dog-like fox is handled wrongly. In this scripted run the learned side improves as it sees more examples and answers all five correctly.

Why: Rules only cover cases their author thought of, while a model adjusts itself to the examples it is shown. Real models still make mistakes, and you would check them on kept-out rows, as the honest test section explains.

In one sentence: a model is a rule that learned from rows with known answers and gives a guess for a new row.

#A Model Is a Rule With Knobs

Look at the table and spot the pattern. Each extra degree adds about 5 cones. The cooler days start low, near 20 cones at 18 degrees. If you were to draw the table on a graph, the dots would climb in a rough straight line.

Here is a rule in everyday words: start counting at 14 degrees, and for every degree above 14, expect 5 cones. In short form:

Is this setting any good? You can check it with a pencil. For each day, work out the rule's guess, then work out the miss, which is how far the guess is from the truth. Here is the whole check for the ten days.

Now use the rule for tomorrow. At 28 degrees: 5 x (28 - 14) = 5 x 14 = 70 cones. That is close to the 69 you guessed from the neighbouring days, and now it came from a written rule that anyone can check.

In one sentence: a model is a rule with knobs, and the average miss on rows with known answers tells you how good the current knob setting is.

#Learning Is Turning the Knobs

So here is the simplest possible learning method. Try many knob settings. For each, work out the average miss. Keep the setting with the smallest one. Computers are good at exactly this kind of long, dull checking. The cell below does it for 16 settings: slopes 3, 4, 5, 6 and starting points 10, 12, 14, 16.

Before you press Run, make a guess.

Do: Press Run. Then change the slope list to (3, 4) and run again.

Notice: With only slopes 3 and 4 on the list, the best average miss is worse than 1.3. The search can only pick from what you offer it.

Why: Learning here is a search over knob settings, scored by the average miss. Real programs search far cleverer than a list, but the scoring idea is the same.

Real projects do not write this loop. They use a library that finds the best knobs directly. The block below does that with scikit-learn. It is a read-only copy for Google Colab, because it uses a library that is not part of the browser cell.

# Copy to Colab (uses scikit-learn; it is not part of the browser cell above)
import numpy as np
from sklearn.linear_model import LinearRegression

temps = np.array([18, 20, 22, 24, 25, 27, 29, 30, 32, 34]).reshape(-1, 1)
cones = np.array([22, 31, 38, 52, 55, 63, 75, 78, 90, 98])

model = LinearRegression().fit(temps, cones)
print("slope:", round(float(model.coef_[0]), 2))
print("start (where the rule reaches zero):", round(float(-model.intercept_ / model.coef_[0]), 2))
print("guess for 28 degrees:", round(float(model.predict([[28]])[0]), 1))
print("average miss on these days:", round(float(abs(model.predict(temps) - cones).mean()), 2))

When you run it, expect a slope near 4.8 and a start near 13.6, close to our 5 and 14. The library can use any decimal for the knobs, so its average miss, about 1.06, is a little lower than our whole-number 1.3. The guess for 28 degrees is about 69.4. This is the same idea, a rule with two knobs, turned more finely.

In one sentence: training means trying knob settings and keeping the one whose average miss on the known rows is smallest.

#The Honest Test

Here is a trap. The average miss of 1.3 was measured on the same ten days we used to pick the knobs. Of course the rule looks good on them. A student who has seen the exam questions in advance also scores well. What you really want to know is how the rule does on a day it has never seen.

Predict before you run the next cell.

Do: Press Run and read the four miss numbers. Then change hidden to [22, 27, 32] and run again.

Notice: The memoriser scores a perfect 0.00 on training days every time, yet its miss on hidden days is clearly larger than the rule's. The rule's miss on hidden days stays small, even a little smaller than on training days in the first run.

Why: A perfect score on rows you have already seen only measures memory. Only rows kept out of training tell you how a model will do on tomorrow's real data. With just three hidden days this is a rough check, not a proof.

In one sentence: judge a model on rows it never saw, because a score on its own training rows only measures memory.

#Three Kinds of Jobs

The next demo is about the second kind of job. You will draw a dividing line yourself, then see the line the computer found.

Do: Press Start Drawing. Drag across the chart where you think the dividing line between the two colours should go. Let go to see the computer's line and a score for how close yours was. Try Again redraws on the same dots.

Notice: Your line and the computer's line are both just straight dividing lines, and both give a guess for any new dot: which side it lands on. Your line may be as good as the computer's, or a bit off.

Why: The computer tried many settings of the line's knobs and kept the one with the fewest mistakes, the same search as the ice cream rule. Panel names such as Logistic Regression and SVM are two ways of doing that search; skip them for now.

In one sentence: guessing a number is regression, picking a label is classification, and finding groups without answers is clustering.

#Too Simple or Too Wiggly

One more idea completes the picture. A model can be too simple to catch the real pattern, or so flexible that it chases every random bump in the rows.

Do: In Dataset choose Quadratic, and make sure Fit Method shows Normal Equation. Slide Polynomial Degree to 1, then 2, then 10. Click the chart to add a few dots.

Notice: Degree 1 is a straight stick that cannot follow the bend. Degree 2 follows it. Degree 10 starts to wiggle toward single dots. The MSE number beside the chart, an average of squared misses, keeps getting smaller or stays level as the degree rises, even when the curve starts to look silly.

Why: That MSE is measured on the same dots the curve was fitted to, so it rewards wiggling. This is exactly why the honest test, with rows kept out, is the only fair judge. Gradient Descent and other panel names are covered in the Linear Regression lesson; skip them for now.

Every model in this track has a setting that moves it between too simple and too wiggly. Learning to find the middle is most of the craft, and the lessons on Bias and Variance, Regularization and Hyperparameter Tuning are about exactly that.

In one sentence: a model that is too simple misses the pattern, a model that is too wiggly learns the luck, and only the honest test tells them apart.

#Where to Go Next

You now have the main picture: a model is a rule with knobs, training turns the knobs to shrink the average miss, and an honest test uses rows the model never saw. The rest of the track fills this picture in with real rules.

• Next, in this track: Neighbours and Questions. Two very different ways to guess, by asking the nearest examples and by asking yes or no questions, each worked out by hand.
• The bigger map: The ML Framework puts the three kinds of learning side by side and adds the training loop.
• Measuring honestly: Model Evaluation and Selection turns the honest test into proper scores for labels and numbers.
• Straight-line rules in full: Linear and Polynomial Regression is the ice cream rule with the maths filled in.

In one sentence: read Neighbours and Questions next, then follow the row that fits your goal.

#Key Takeaways

• A model is a rule that turns something you know into a guess about something you do not
• A rule has knobs, called parameters, and a model is the rule's shape plus its knob settings
• The miss is how far a guess is from the truth, and the average miss scores a whole setting
• Training means turning the knobs to shrink the average miss on rows with known answers
• Judge a model on rows it never saw, because a perfect score on training rows can be pure memory
• Guessing a number is regression, picking a label is classification, and finding groups is clustering
• Too simple a rule underfits, too wiggly a rule overfits, and the honest test shows which is happening

#Quick Check