Neighbours and Questions: Two Ways to Guess

#Ask the Neighbours

Imagine you move to a new town and want to pick a restaurant. You do not read a rulebook. You ask the people who live closest to you, because they eat in places near you and probably like what you like. If most of them say "go to the noodle place", you go there.

A new animal arrives. It weighs 8.2 kg and stands 34 cm tall. Is it a cat or a dog? Make a guess before we measure anything.

In one sentence: nearest neighbours labels a new thing by letting the most similar known things vote.

#Measuring Close

To say which pet is closest, we need a way to measure closeness. Picture a chart with weight along the bottom and height up the side. Every pet is a dot. The new animal is one more dot. Close means a short straight line between two dots, the line you would draw with a ruler.

You can work out that length with the triangle rule from school. Go across by the weight gap, go up by the height gap, and the straight line is the long side of the triangle. The length is the square root of (across squared plus up squared). For Rex: across 1.2, up 0. The length is the square root of (1.44 + 0), which is 1.2. For Leo: across 0.8, up 4. The length is the square root of (0.64 + 16), which is about 4.1.

Doing this for all eight pets gives the whole ranking.

Rex is the closest, as you guessed. If only the nearest one votes, the new animal is a dog.

There is a trap hiding in this ruler. The across gap is measured in kilograms and the up gap in centimetres. They are different units that happen to sit in one sum. Suppose we had written weight in grams instead. The new animal would weigh 8,200 and Rex 7,000. The across gap would be 1,200 for Rex and only 800 for Leo, while the up gaps are tiny next to those. Now Leo is the closest, not Rex, and the nearest neighbour says cat. Nothing about the animals changed. Only the unit did.

In one sentence: closeness is a straight-line ruler distance, and the units of your features decide who counts as close.

#How Many Neighbours Vote

• K = 1: Rex only. Rex is a dog, so the answer is dog.
• K = 3: Rex (dog), Leo (cat), Kit (cat). Two cats beat one dog, so the answer is cat.
• K = 5: add Max (dog) and Tom (cat). That is 3 cats and 2 dogs, so the answer is cat.
• K = 7: add Mia (cat) and Bo (dog). That is 4 cats and 3 dogs, so the answer is cat.
• K = 8: all eight vote, 4 cats and 4 dogs, a tie, and the vote cannot decide.

The answer flipped between K = 1 and K = 3. When K is 1, one unusual example controls the result. Rex is a small dog, and a single small dog is enough to pull the answer to dog. When K is larger, one odd example gets outvoted. But if K is too large, far-away animals with little to do with the new one start voting, and in the extreme case everyone votes and the answer is just whichever label is most common overall. That is why people like an odd K, which avoids ties, and a middle value, not the smallest or the largest.

Before you try the demo, make a guess.

Do: Make sure Algorithm shows KNN. In Dataset pick Moons. Slide K (neighbors) from 1 up to about 25, one stop at a time. Then try Circles.

Notice: At K = 1 the coloured regions have small islands around single dots. As K grows the border smooths out. When K gets close to the number of dots on the chart, the border loses its shape, because almost every dot votes on everything.

Why: K sets how many neighbours vote. A small K trusts single examples, which follows the dots too closely. A big K trusts the crowd, which can wash out real detail. The slider is a knob you choose before looking, not one the computer turns for you.

Now do the same thing in code. The cell below reproduces the ranking table, the four votes, and the unit trap. It uses only the Python standard library.

Do: Press Run and compare the printed ranking with the table above. Then change mystery to (20, 50) and run again.

Notice: The first run matches the table and shows the answer flipping from dog to cat between K = 1 and K = 3. The grams line shows the nearest pet changing from Rex to Leo with no change to any animal. For (20, 50), every K now agrees on dog.

Why: A new animal that sits deep inside one group gets the same answer from any K. Only animals near the border are sensitive to K and to units.

In one sentence: K is how many neighbours vote, and a very small or very large K both give worse answers than a middle one.

#Ask Yes or No Questions

Now try a worse question: "is the weight above 10.5 kg?". Mia, Tom, Kit and Leo weigh 4, 5, 6 and 9, so they answer no and are called cat. Right. Max, Bo and Ace weigh 12, 18 and 26, so they answer yes and are called dog. Right. But Rex weighs 7, answers no, and is called a cat, while Rex is a dog. That is 1 mistake out of 8.

Before the next section, test your own judgement.

In one sentence: a yes or no question is good when few known rows end up on the wrong side of it.

#Choosing the Best Question

A computer picks the first question by doing what you just did, only for every possible question. For each feature, it tries a cut-off halfway between each pair of neighbouring values. It counts the mistakes for each. It keeps the question with the fewest mistakes. The cell below does this for weight and height on the eight pets.

Do: Press Run and find the line for height above 32.0. Then change Leo's height from 30 to 36 in the list and run again.

Notice: At first one question, height above 32, gets all eight right. After the change the last line says no perfect question exists, and the best one has 1 mistake.

Why: The search is the same idea as the knob search from the last lesson: try the options, score each by its mistakes, keep the best. When one question is not enough, the computer asks a second question on the side that still has mixed labels, and so on.

Here is that growing process, step by step, on a chart of dots. Watch for the last step.

Do: Press Next Split one step at a time through all five splits. Watch the tree on one side and the coloured boxes on the other.

Notice: Each split adds one straight cut and one yes or no box to the tree. The early splits make big clean areas. The last split carves a tiny box around a single dot.

Why: A tree can keep asking questions until every known dot is alone in its own box, which gives zero mistakes on the known dots, but the tiny boxes are memorising luck, like the memoriser in the last lesson. A good tree stops before that.

In one sentence: pick the question with the fewest mistakes, repeat on whatever is still mixed, and stop before the tree memorises.

#Neighbours or Questions

You now have two very different rules. They answered our mystery animal differently: K = 1 neighbours said dog, K = 3 neighbours said cat, and the single height question said dog. None of them is the right answer by itself. Each is a guess from a different kind of rule, and the next step is the one from the last lesson: hide some known pets, and see which rule guesses them better.

How do they compare in practice?

Neither rule wins everywhere. Neighbours are easy to start with and work well when similar things really do have similar labels. Questions are easy to read aloud and handle features of mixed units without rescaling. Real projects often combine many trees, and later lessons on Decision Trees and Ensembles show how.

For classroom work in Python you would use a library for both. The block below is a read-only copy for Google Colab, because it uses scikit-learn.

# Copy to Colab (uses scikit-learn; it is not part of the browser cells above)
import numpy as np
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier, export_text

X = np.array([[4, 25], [5, 27], [6, 29], [9, 30], [7, 34], [12, 40], [18, 48], [26, 58]])
y = np.array(["cat", "cat", "cat", "cat", "dog", "dog", "dog", "dog"])
mystery = [[8.2, 34]]

for k in (1, 3, 5):
    knn = KNeighborsClassifier(n_neighbors=k).fit(X, y)
    print(f"K = {k}:", knn.predict(mystery)[0])

stump = DecisionTreeClassifier(max_depth=1, random_state=0).fit(X, y)
print(export_text(stump, feature_names=["weight", "height"]))
print("tree says:", stump.predict(mystery)[0])

Expect dog for K = 1, cat for K = 3 and K = 5, and a one-question tree that splits on height at 32 and says dog. These match what you worked out with a pencil.

In one sentence: neighbours store the rows and vote, questions search for a short rule, and the honest test decides which one is better for your data.

#Where to Go Next

You can now guess a label two different ways and check each one by hand. The lessons below build on exactly these two rules.

• Neighbours in full: K-Nearest Neighbors adds other ways to measure distance, a weighted vote and the problem of many features.
• Questions in full: Decision Trees replaces counting mistakes with a smoother score and shows how trees overfit.
• Many trees together: Ensemble Methods shows why a crowd of trees beats one tree.
• Judging honestly: Model Evaluation and Selection turns the hide-some-rows test into proper scores.
• The map of everything: The ML Framework sets these rules beside the other kinds of learning.

In one sentence: read K-Nearest Neighbors or Decision Trees next, whichever rule you want to see in more depth.

#Key Takeaways

• Nearest neighbours labels a new thing by letting the most similar known things vote, with no knobs to turn
• Closeness is a straight-line ruler distance, so the units of your features can change who counts as close
• K is how many neighbours vote: K = 1 trusts single examples and a huge K trusts only the crowd
• A decision tree is a list of yes or no questions, and the best question has the fewest mistakes on known rows
• Adding questions until every known row is right is memorising and fails on new rows
• Neighbours and questions can disagree on a new row, so hide some known rows and see which rule guesses better

#Quick Check