Correlation: Pearson, Spearman & Multicollinearity

Here is where this two-lesson story ends up. Three real-looking results that mislead, each of which you will be able to catch by the end of Part 2:

• A classifier is trained on photos where most camels stand on sand and most cows on grass. It quietly learns "sand means camel", scores well on a test set from the same collection, and fails the day a camel stands on grass.
• A hospital model learns that patients given an intensive-care bed are higher risk. That is true in the data and useless as advice, because nobody should withhold the bed.
• Model A beats model B on overall accuracy while B is better inside every segment of users, which is Simpson's paradox in an evaluation table.
• Each is a correlation that is real in the data and still a poor guide to action. This lesson builds the measuring tools, and Part 2 gives you the questions that expose them.

#Why Correlation Is the Most Misused Number

Correlation answers one narrow question: if I know X, how well can I guess Y with a straight line? It says nothing about why, and nothing about what happens if you change X.

Every section below uses numbers you can verify by hand or by running the code. All counts and simulations are synthetic and labelled as such. The one real dataset is Anscombe's quartet, a famous public teaching example from 1973.

#Covariance Recap and Pearson r

Covariance measures whether X and Y sit above or below their means together:

The trouble with covariance is its units. Heights in centimetres and weights in kilograms give a covariance in cm·kg, and changing centimetres to metres changes the number by a factor of 100 without changing the relationship. Dividing by both standard deviations removes the units:

How to read the value:

• r = +1 means every point lies exactly on a rising line. r = −1 means exactly on a falling line.
• r = 0 means no linear association. It does not mean no association (see Anscombe below).
• r squared is the fraction of the variance of Y that the best straight line through X accounts for. With r = 0.5, only 25 percent.
• r is symmetric (r(X, Y) = r(Y, X)) and unchanged if you rescale or shift either variable, for example grams to kilograms.

#Pearson r by Hand on Eight Points

Eight students record hours studied (x) and exam score (y). Compute r step by step.

So Sxy = 153, Sxx = 52, Syy = 458.

Notice what each part did. Student 4 studied 5 hours, exactly the average, so the x deviation is 0 and the product contributes nothing. Students 1 and 8 sit far from the centre on both axes and contribute the most. Correlation is dominated by the points that are far from both means.

Now let the code confirm every number, and move straight on to the case that breaks the rule.

#What r Measures and What It Does Not: Anscombe's Quartet

Here are the four datasets, which are public and widely reproduced. Datasets I, II and III share the same x values. In set IV, ten points have x = 8 and one has x = 19.

Running the code below confirms that for all four sets the mean of x is 9.00, the mean of y is 7.50, the variance of x is 11.00, the variance of y is about 4.13, the best-fit line is y ≈ 3.00 + 0.50x, and r is 0.816 for I, II and III and 0.817 for IV (about 0.816 in every case).

What the plots show:

• I: a noisy straight line. r is a fair summary here.
• II: a perfect smooth curve (an inverted U). The relationship is exact, yet r is still 0.816, because r only measures the straight-line part.
• III: an exact straight line with one outlier. Without the outlier r would be 1.000 and the line would be different. One point drags r down and tilts the slope.
• IV: every x is 8 except one point at 19. The entire correlation comes from that single point. Remove it and there is no relationship at all.

Picture what r is built to detect. A scatter of points that forms a tilted oval cloud has r well away from zero, and the thinner the oval, the closer |r| is to 1. A round cloud has r near 0. A tilted oval is exactly the shape r can detect, and the only shape it can detect: a U or a ring is far from random, yet its r is near zero, which is Anscombe's set II again.

#Spearman Rank Correlation

Spearman's ρ (rho) answers a looser question: when X goes up, does Y tend to go up, whatever the shape? The recipe is simple. Replace each value by its rank (1 for the smallest, n for the largest, ties share the average rank), then compute Pearson's r on the ranks.

Spearman is not magic. A U-shape is not monotone, so it fails there too. And it discards the sizes of the gaps, so if magnitude matters, it throws information away.

#The Correlation Matrix and Multicollinearity

In ML you read the matrix for three things:

• Feature to label. A large |r| with the target suggests a feature carries a linear signal. Remember the limits above: it can also be a confounder's footprint.
• Feature to feature. Pairs near ±1 are redundant. Two columns that are the same quantity in different units (square metres and square feet) add no information, only trouble.
• Clusters of features. Blocks of high correlation hint at a smaller number of underlying factors, which is the idea behind PCA.

#The Same Tools in pandas: corr(), groupby and a Heat-Map

Everything above is a correlation. It tells you that short sessions go with churn, not that lengthening a session would keep a user. Treat the heat-map as a list of places to look, and ask the three-explanations question of every red or blue square, which Part 2 spells out.

#Try It Yourself

Three short exercises put this lesson's main numbers in your hands: Spearman on a curved, tied dataset, the same correlations in pandas, and an outlier that moves Pearson but not Spearman. The pearson helper is given. Fill in the TODOs, then compare with the solution.

#Key Takeaways

• r is covariance without units — r = Sxy / √(Sxx · Syy); on the eight-student data Sxy = 153, Sxx = 52, Syy = 458, so r = 0.9914 and r squared = 0.983
• r only sees straight lines — Anscombe's four datasets share mean 7.50, variance 4.13, line y = 3 + 0.5x and r about 0.816, yet they are a line, a curve, a line with an outlier, and one lone point creating the correlation
• Spearman is Pearson on ranks — it gives 1.000 on a monotone curve where Pearson gives 0.7905, and 0.9286 with an outlier where Pearson gives 0.6955; it still fails on U-shapes
• Correlated inputs destabilise coefficients — with r about 0.998 between two features each coefficient's standard deviation rises from about 0.2 to about 2.85, while their sum stays stable at about 0.15
• pandas does the bookkeeping — df.corr() gives the matrix, groupby().mean() the per-group view and imshow the heat-map; on StreamBox, churn correlates −0.561 with sessions per week and −0.343 with session minutes

#Quick Check