Python for Machine Learning: sklearn & Beyond

This lesson is the bridge from "Python developer" to "ML engineer." Same language, completely different vocabulary: tensors instead of lists, vectorization instead of loops, broadcasting instead of nested iteration, fit/predict instead of compute. scikit-learn's fit/predict API is so well-designed that PyTorch, XGBoost, and Hugging Face all copied it.

Learning Objectives

After this lesson, you will be able to:

Follow the sklearn workflow: load, split, train, predict, evaluate
Use train_test_split to create training and testing sets
Build pipelines that combine preprocessing and modeling steps
Apply common models: LinearRegression, DecisionTreeClassifier, KMeans
Connect everything learned in Python to a real ML workflow

Prerequisites

Before You Start

Review

#The sklearn Workflow

Every sklearn project follows the same five-step pattern:

pythonrunnable cell

# Step 1: Load data
from sklearn.datasets import load_iris
import numpy as np

iris = load_iris()
X = iris.data        # features: sepal length, sepal width, petal length, petal width
y = iris.target      # labels: 0=setosa, 1=versicolor, 2=virginica

print(f"Features shape: {X.shape}")   # (150, 4) -- 150 samples, 4 features
print(f"Labels shape: {y.shape}")     # (150,) -- 150 labels
print(f"Feature names: {iris.feature_names}")
print(f"Classes: {iris.target_names}")  # ['setosa', 'versicolor', 'virginica']

# Step 2: Split into training and testing sets
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
print(f"Training set: {X_train.shape[0]} samples")  # 120
print(f"Test set: {X_test.shape[0]} samples")        # 30

# Step 3: Choose and create a model
from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier(max_depth=3, random_state=42)

# Step 4: Train the model
model.fit(X_train, y_train)

# Step 5: Predict and evaluate
y_pred = model.predict(X_test)

from sklearn.metrics import accuracy_score
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy: {accuracy:.2%}")  # ~96.67%

What Do You Think?

We trained five models on the same Iris dataset (120 training samples, 30 test samples). Which model do you predict will have the HIGHEST accuracy?

#The Universal API: fit(), predict(), score()

Every sklearn model follows the same interface:

pythonreference · read-only

# This works for ANY sklearn model -- just swap the class
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier

models = {
    "Decision Tree": DecisionTreeClassifier(max_depth=3, random_state=42),
    "Logistic Regression": LogisticRegression(max_iter=200, random_state=42),
    "Random Forest": RandomForestClassifier(n_estimators=100, random_state=42),
    "SVM": SVC(random_state=42),
    "KNN (k=5)": KNeighborsClassifier(n_neighbors=5),
}

print("Model comparison on Iris dataset:")
print("-" * 40)

for name, model in models.items():
    model.fit(X_train, y_train)
    accuracy = model.score(X_test, y_test)  # score() = predict() + accuracy
    print(f"  {name:25s} accuracy: {accuracy:.2%}")

# Decision Tree              accuracy: 96.67%
# Logistic Regression        accuracy: 100.00%
# Random Forest              accuracy: 96.67%
# SVM                        accuracy: 96.67%
# KNN (k=5)                  accuracy: 96.67%

Notice how every model has the same fit() / predict() / score() API. This is the power of sklearn's consistent design — you learned the pattern once, now you can use hundreds of algorithms.

The ML Workflow — From Data to PredictionInteractive

Loading visualization...

#Regression with sklearn

Classification predicts categories (spam/not spam). Regression predicts continuous values (price, temperature, score).

pythonreference · read-only

from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

# Generate synthetic regression data
X, y = make_regression(n_samples=200, n_features=1, noise=20, random_state=42)

# Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train
reg = LinearRegression()
reg.fit(X_train, y_train)

# Evaluate
y_pred = reg.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

print(f"Slope (coefficient): {reg.coef_[0]:.2f}")
print(f"Intercept: {reg.intercept_:.2f}")
print(f"Mean Squared Error: {mse:.2f}")
print(f"R^2 Score: {r2:.4f}")  # 1.0 = perfect, 0.0 = no better than mean

#Evaluation Metrics Explained

pythonrunnable cell

# For Classification — re-load Iris because the regression demo above
# reassigned X_train/y_train. Always re-establish your splits before reusing
# the names!
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import classification_report, confusion_matrix

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=42, stratify=iris.target
)

tree = DecisionTreeClassifier(max_depth=3, random_state=42)
tree.fit(X_train, y_train)
y_pred = tree.predict(X_test)

print("Confusion Matrix:")
print(confusion_matrix(y_test, y_pred))
# (Rows = true class, cols = predicted class — a diagonal-heavy matrix
#  is a good sign. Exact counts depend on the split.)

print("\nClassification Report:")
print(classification_report(y_test, y_pred, target_names=iris.target_names))
# Shows precision, recall, f1-score per class

# For Regression:
# MSE (Mean Squared Error) -- average of squared errors, penalizes large errors
# R^2 Score -- how much variance the model explains (1.0 = perfect)
# MAE (Mean Absolute Error) -- average of absolute errors, easier to interpret

What Do You Think?

A regression model scores R² = 0.95 on the training set and 0.42 on the test set. What has gone wrong?

#Pipelines

A pipeline chains preprocessing and modeling into a single object. This prevents data leakage and makes your workflow reproducible.

pythonreference · read-only

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

# Without pipeline (error-prone -- easy to forget a step)
# scaler = StandardScaler()
# X_train_scaled = scaler.fit_transform(X_train)
# X_test_scaled = scaler.transform(X_test)  # must use transform, not fit_transform!

# With pipeline (clean, safe, reproducible)
pipe = Pipeline([
    ("scaler", StandardScaler()),         # Step 1: normalize features
    ("pca", PCA(n_components=2)),         # Step 2: reduce to 2 dimensions
    ("classifier", LogisticRegression()), # Step 3: classify
])

# The pipeline handles everything correctly
pipe.fit(X_train, y_train)
accuracy = pipe.score(X_test, y_test)
print(f"Pipeline accuracy: {accuracy:.2%}")

# Under the hood, pipe.fit() does:
# 1. scaler.fit_transform(X_train) -- learn mean/std, then scale
# 2. pca.fit_transform(X_train_scaled) -- learn components, then project
# 3. classifier.fit(X_train_projected, y_train) -- learn decision boundary
#
# And pipe.predict() does:
# 1. scaler.transform(X_test) -- scale using training stats
# 2. pca.transform(X_test_scaled) -- project using training components
# 3. classifier.predict(X_test_projected) -- make predictions

Watch Out

Common Mistake

Mistake: using fit_transform() on test data instead of just transform().

pythonreference · read-only

# WRONG -- the scaler learns from test data statistics
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # OK
X_test_scaled = scaler.fit_transform(X_test)    # BUG: re-fits on test data!

# RIGHT -- test data uses the mean/std learned from training
X_train_scaled = scaler.fit_transform(X_train)  # learns mean/std
X_test_scaled = scaler.transform(X_test)         # applies the same mean/std

When you call fit_transform() on test data, the scaler learns the test set's statistics — its own mean and standard deviation. Now your test set is "normalized to itself," not normalized the same way as training data. This is data leakage, and it artificially inflates test accuracy. Always fit on train, transform on both.

#Why Pipelines Matter: Preventing Data Leakage

pythonreference · read-only

# DATA LEAKAGE -- a subtle but devastating bug
# BAD: fitting the scaler on ALL data (including test)
# scaler.fit(X)  # <-- sees test data statistics!
# X_scaled = scaler.transform(X)
# X_train, X_test = split(X_scaled)  # test set is "contaminated"

# GOOD: pipeline ensures test data is never seen during fit
pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("model", RandomForestClassifier(n_estimators=100, random_state=42)),
])

# fit() only sees training data, predict() applies the same transformations
pipe.fit(X_train, y_train)
print(f"No leakage accuracy: {pipe.score(X_test, y_test):.2%}")

#ColumnTransformer: Different Preprocessing for Different Columns

Real datasets have mixed types: some columns are numeric (need scaling), others are categorical (need encoding). ColumnTransformer applies the right transformer to each column group and can be dropped into a Pipeline:

pythonrunnable cell

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

# Example dataset: mix of numeric and categorical features
data = pd.DataFrame({
    "age":        [25, 35, 45, 28, 52, 31],
    "income":     [50000, 80000, 120000, 62000, 95000, 71000],
    "education":  ["bachelor", "master", "phd", "bachelor", "master", "bachelor"],
    "city":       ["NYC", "SF", "NYC", "Austin", "SF", "NYC"],
    "bought":     [0, 1, 1, 0, 1, 1],
})

X = data.drop("bought", axis=1)
y = data["bought"]

# Identify column types
numeric_features = ["age", "income"]
categorical_features = ["education", "city"]

# ColumnTransformer: apply the right transformer to each group
preprocessor = ColumnTransformer(transformers=[
    ("num", StandardScaler(), numeric_features),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical_features),
    # Any columns NOT listed are dropped by default (set remainder="passthrough" to keep)
])

# Full pipeline: preprocess + model
full_pipeline = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", RandomForestClassifier(n_estimators=100, random_state=42)),
])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

full_pipeline.fit(X_train, y_train)
print(f"Accuracy: {full_pipeline.score(X_test, y_test):.2%}")

# Inspect what the preprocessor produces
X_transformed = preprocessor.fit_transform(X_train)
print(f"Original shape: {X_train.shape}")       # (4, 4)
print(f"Transformed shape: {X_transformed.shape}")  # (4, N) where N = 2 + num_one_hot_categories

ColumnTransformer is the standard sklearn way to handle mixed-type data. The remainder parameter controls what happens to columns you did not list: "drop" (default) discards them, "passthrough" keeps them unchanged.

#Unsupervised Learning with KMeans

Not all ML is supervised. KMeans finds clusters in data without labels.

pythonrunnable cell

from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs

# Generate data with 3 natural clusters
X_blobs, true_labels = make_blobs(n_samples=300, centers=3, random_state=42)

# KMeans does NOT get the labels -- it discovers clusters on its own
kmeans = KMeans(n_clusters=3, random_state=42, n_init=10)
kmeans.fit(X_blobs)

# Results
predicted_clusters = kmeans.labels_
centers = kmeans.cluster_centers_

print(f"Cluster centers:\n{centers}")
print(f"Inertia (lower = tighter clusters): {kmeans.inertia_:.2f}")

# Compare to true labels
from sklearn.metrics import adjusted_rand_score
ari = adjusted_rand_score(true_labels, predicted_clusters)
print(f"Adjusted Rand Index: {ari:.4f}")  # 1.0 = perfect match

#Choosing the Number of Clusters (Elbow Method)

pythonreference · read-only

inertias = []
K_range = range(1, 10)

for k in K_range:
    km = KMeans(n_clusters=k, random_state=42, n_init=10)
    km.fit(X_blobs)
    inertias.append(km.inertia_)

# The "elbow" in the plot indicates the best k
for k, inertia in zip(K_range, inertias):
    bar = "#" * int(inertia / 100)
    print(f"k={k}: inertia={inertia:>10.2f} {bar}")

# k=1: inertia=  13245.87 ################################################...
# k=2: inertia=   4935.21 ##################
# k=3: inertia=   1221.45 ####              <-- elbow! Adding more clusters helps less
# k=4: inertia=    981.12 ###
# k=5: inertia=    814.33 ##

Quick check

Inertia keeps falling as k rises — 13245, 4935, 1221, 981, 814. Why not simply pick the k with the lowest inertia?

#The Complete ML Pipeline

Here is a full, production-ready workflow combining everything:

pythonrunnable cell

import numpy as np
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report

# 1. Load data
wine = load_wine()
X, y = wine.data, wine.target
print(f"Dataset: {X.shape[0]} samples, {X.shape[1]} features, {len(wine.target_names)} classes")

# 2. Split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# 3. Build pipeline
pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("model", RandomForestClassifier(n_estimators=100, random_state=42)),
])

# 4. Cross-validation (more robust than a single train/test split)
cv_scores = cross_val_score(pipeline, X_train, y_train, cv=5, scoring="accuracy")
print(f"\nCross-validation scores: {cv_scores}")
print(f"Mean CV accuracy: {cv_scores.mean():.2%} (+/- {cv_scores.std():.2%})")

# 5. Train on full training set
pipeline.fit(X_train, y_train)

# 6. Final evaluation on test set
y_pred = pipeline.predict(X_test)
print(f"\nTest accuracy: {pipeline.score(X_test, y_test):.2%}")
print("\nDetailed Report:")
print(classification_report(y_test, y_pred, target_names=wine.target_names))

# 7. Feature importance (from the Random Forest inside the pipeline)
importances = pipeline.named_steps["model"].feature_importances_
feature_ranking = sorted(zip(wine.feature_names, importances), key=lambda x: -x[1])
print("Top 5 most important features:")
for name, importance in feature_ranking[:5]:
    bar = "#" * int(importance * 100)
    print(f"  {name:30s} {importance:.4f} {bar}")

#Code Playground

pythonplayground.py · Pyodide

# ML Pipeline Concepts in Python
# The sklearn workflow: generate, split, train, predict, evaluate
import random
import math
from collections import Counter

# Step 1: Generate synthetic data
def generate_data(n_samples: int, noise: float = 0.1):
  X, y = [], []
  for _ in range(n_samples):
      x = random.random() * 10
      label = 3 * x + 5 + (random.random() - 0.5) * noise * 20
      X.append([x])
      y.append(label)
  return X, y

# Step 2: Train/test split
def train_test_split(X, y, test_ratio=0.2):
  n = len(X)
  indices = list(range(n))
  random.shuffle(indices)
  test_size = int(n * test_ratio)
  test_idx, train_idx = indices[:test_size], indices[test_size:]
  return ([X[i] for i in train_idx], [X[i] for i in test_idx],
          [y[i] for i in train_idx], [y[i] for i in test_idx])

# Step 3: Simple Linear Regression (from scratch!)
class LinearRegression:
  def __init__(self):
      self.slope = 0.0
      self.intercept = 0.0

def fit(self, X, y):
      n = len(X)
      xs = [row[0] for row in X]
      sum_x = sum(xs)
      sum_y = sum(y)
      sum_xy = sum(x * yi for x, yi in zip(xs, y))
      sum_x2 = sum(x**2 for x in xs)
      self.slope = (n * sum_xy - sum_x * sum_y) / (n * sum_x2 - sum_x**2)
      self.intercept = (sum_y - self.slope * sum_x) / n
      return self

def predict(self, X):
      return [self.slope * row[0] + self.intercept for row in X]

def score(self, X, y):
      preds = self.predict(X)
      mean_y = sum(y) / len(y)
      ss_res = sum((yi - pi)**2 for yi, pi in zip(y, preds))
      ss_tot = sum((yi - mean_y)**2 for yi in y)
      return 1 - ss_res / ss_tot  # R^2

# Step 4: KNN Classifier (from scratch!)
class KNNClassifier:
  def __init__(self, k: int = 3):
      self.k = k
      self.X_train = []
      self.y_train = []

def fit(self, X, y):
      self.X_train, self.y_train = X, y
      return self

def predict(self, X):
      results = []
      for point in X:
          dists = [(math.dist(tp, point), label) for tp, label in zip(self.X_train, self.y_train)]
          dists.sort()
          k_nearest = [label for _, label in dists[:self.k]]
          results.append(Counter(k_nearest).most_common(1)[0][0])
      return results

def score(self, X, y):
      preds = self.predict(X)
      return sum(p == yi for p, yi in zip(preds, y)) / len(y)

# --- RUN THE PIPELINE ---
print("=== Linear Regression ===")
X, y = generate_data(200, 0.3)
X_train, X_test, y_train, y_test = train_test_split(X, y)

reg = LinearRegression()
reg.fit(X_train, y_train)
print(f"Learned: y = {reg.slope:.2f}x + {reg.intercept:.2f}")
print(f"True:    y = 3.00x + 5.00")
print(f"R^2 score: {reg.score(X_test, y_test):.4f}")

print("\n=== KNN Classification ===")
class_X = [[1, 2], [2, 3], [1.5, 1.5], [2, 2],
         [6, 5], [7, 7], [8, 6], [6.5, 6.5]]
class_y = [0, 0, 0, 0, 1, 1, 1, 1]

knn = KNNClassifier(k=3)
knn.fit(class_X, class_y)
print("Predict [3, 3]:", knn.predict([[3, 3]]))
print("Predict [7, 6]:", knn.predict([[7, 6]]))
print("Training accuracy:", knn.score(class_X, class_y))

# CHALLENGES:
# 1. Add a StandardScaler class that normalizes features (mean=0, std=1)
# 2. Chain StandardScaler + KNN into a Pipeline class with fit/predict
# 3. Implement cross-validation: split data into k folds, train/test k times
# 4. Add a confusion_matrix function for classification results

Tests · Build a complete ML pipeline from data loading to evaluation!

Quick check

A Pipeline chains a scaler and a model. What does it prevent that calling `scaler.fit_transform(X)` before `train_test_split` would not?

#Feature Scaling: The Step Everyone Forgets

Most ML algorithms perform better when features are on similar scales. House prices: bedrooms=3 vs square_feet=2000 — the large number dominates.

pythonreference · read-only

from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# ❌ WRONG — data leakage: scaler sees test data before predicting
scaler_wrong = StandardScaler()
X_all_scaled = scaler_wrong.fit_transform(X)  # fits on test data too!

# ✅ CORRECT — fit ONLY on training data
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # learn mean/std from train
X_test_scaled  = scaler.transform(X_test)        # apply SAME scale to test

#Key Takeaways

The sklearn workflow is always: load -> split -> train -> predict -> evaluate — this five-step pattern applies to every supervised learning problem. fit() trains, predict() generates outputs, score() measures accuracy
Pipelines chain preprocessing and modeling — Pipeline([("scaler", StandardScaler()), ("model", RandomForest())]) prevents data leakage by ensuring test data is never seen during training
The same API works for every model — swap DecisionTreeClassifier for RandomForestClassifier or SVM and the rest of your code stays identical. This is sklearn's greatest strength
KMeans does unsupervised learning — it discovers clusters without labels. Use the elbow method to choose the right number of clusters
Everything in Python connects to ML — lists are feature vectors, dictionaries are hyperparameters, classes are models, generators feed data batches, type hints catch pipeline bugs early

Quick Check1 / 5

What does model.fit(X_train, y_train) do?

Python for Machine Learning: sklearn & Beyond

Learning Objectives

After this lesson, you will be able to:

Follow the sklearn workflow: load, split, train, predict, evaluate
Use train_test_split to create training and testing sets
Build pipelines that combine preprocessing and modeling steps
Apply common models: LinearRegression, DecisionTreeClassifier, KMeans
Connect everything learned in Python to a real ML workflow

Prerequisites

Before You Start

Review

#The sklearn Workflow

Every sklearn project follows the same five-step pattern:

pythonrunnable cell

# Step 1: Load data
from sklearn.datasets import load_iris
import numpy as np

iris = load_iris()
X = iris.data        # features: sepal length, sepal width, petal length, petal width
y = iris.target      # labels: 0=setosa, 1=versicolor, 2=virginica

print(f"Features shape: {X.shape}")   # (150, 4) -- 150 samples, 4 features
print(f"Labels shape: {y.shape}")     # (150,) -- 150 labels
print(f"Feature names: {iris.feature_names}")
print(f"Classes: {iris.target_names}")  # ['setosa', 'versicolor', 'virginica']

# Step 2: Split into training and testing sets
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
print(f"Training set: {X_train.shape[0]} samples")  # 120
print(f"Test set: {X_test.shape[0]} samples")        # 30

# Step 3: Choose and create a model
from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier(max_depth=3, random_state=42)

# Step 4: Train the model
model.fit(X_train, y_train)

# Step 5: Predict and evaluate
y_pred = model.predict(X_test)

from sklearn.metrics import accuracy_score
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy: {accuracy:.2%}")  # ~96.67%

What Do You Think?

We trained five models on the same Iris dataset (120 training samples, 30 test samples). Which model do you predict will have the HIGHEST accuracy?

#The Universal API: fit(), predict(), score()

Every sklearn model follows the same interface:

pythonreference · read-only

# This works for ANY sklearn model -- just swap the class
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier

models = {
    "Decision Tree": DecisionTreeClassifier(max_depth=3, random_state=42),
    "Logistic Regression": LogisticRegression(max_iter=200, random_state=42),
    "Random Forest": RandomForestClassifier(n_estimators=100, random_state=42),
    "SVM": SVC(random_state=42),
    "KNN (k=5)": KNeighborsClassifier(n_neighbors=5),
}

print("Model comparison on Iris dataset:")
print("-" * 40)

for name, model in models.items():
    model.fit(X_train, y_train)
    accuracy = model.score(X_test, y_test)  # score() = predict() + accuracy
    print(f"  {name:25s} accuracy: {accuracy:.2%}")

# Decision Tree              accuracy: 96.67%
# Logistic Regression        accuracy: 100.00%
# Random Forest              accuracy: 96.67%
# SVM                        accuracy: 96.67%
# KNN (k=5)                  accuracy: 96.67%

Notice how every model has the same fit() / predict() / score() API. This is the power of sklearn's consistent design — you learned the pattern once, now you can use hundreds of algorithms.

The ML Workflow — From Data to PredictionInteractive

Loading visualization...

#Regression with sklearn

Classification predicts categories (spam/not spam). Regression predicts continuous values (price, temperature, score).

pythonreference · read-only

from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

# Generate synthetic regression data
X, y = make_regression(n_samples=200, n_features=1, noise=20, random_state=42)

# Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train
reg = LinearRegression()
reg.fit(X_train, y_train)

# Evaluate
y_pred = reg.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

print(f"Slope (coefficient): {reg.coef_[0]:.2f}")
print(f"Intercept: {reg.intercept_:.2f}")
print(f"Mean Squared Error: {mse:.2f}")
print(f"R^2 Score: {r2:.4f}")  # 1.0 = perfect, 0.0 = no better than mean

#Evaluation Metrics Explained

pythonrunnable cell

# For Classification — re-load Iris because the regression demo above
# reassigned X_train/y_train. Always re-establish your splits before reusing
# the names!
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import classification_report, confusion_matrix

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=42, stratify=iris.target
)

tree = DecisionTreeClassifier(max_depth=3, random_state=42)
tree.fit(X_train, y_train)
y_pred = tree.predict(X_test)

print("Confusion Matrix:")
print(confusion_matrix(y_test, y_pred))
# (Rows = true class, cols = predicted class — a diagonal-heavy matrix
#  is a good sign. Exact counts depend on the split.)

print("\nClassification Report:")
print(classification_report(y_test, y_pred, target_names=iris.target_names))
# Shows precision, recall, f1-score per class

# For Regression:
# MSE (Mean Squared Error) -- average of squared errors, penalizes large errors
# R^2 Score -- how much variance the model explains (1.0 = perfect)
# MAE (Mean Absolute Error) -- average of absolute errors, easier to interpret

What Do You Think?

A regression model scores R² = 0.95 on the training set and 0.42 on the test set. What has gone wrong?

#Pipelines

A pipeline chains preprocessing and modeling into a single object. This prevents data leakage and makes your workflow reproducible.

pythonreference · read-only

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

# Without pipeline (error-prone -- easy to forget a step)
# scaler = StandardScaler()
# X_train_scaled = scaler.fit_transform(X_train)
# X_test_scaled = scaler.transform(X_test)  # must use transform, not fit_transform!

# With pipeline (clean, safe, reproducible)
pipe = Pipeline([
    ("scaler", StandardScaler()),         # Step 1: normalize features
    ("pca", PCA(n_components=2)),         # Step 2: reduce to 2 dimensions
    ("classifier", LogisticRegression()), # Step 3: classify
])

# The pipeline handles everything correctly
pipe.fit(X_train, y_train)
accuracy = pipe.score(X_test, y_test)
print(f"Pipeline accuracy: {accuracy:.2%}")

# Under the hood, pipe.fit() does:
# 1. scaler.fit_transform(X_train) -- learn mean/std, then scale
# 2. pca.fit_transform(X_train_scaled) -- learn components, then project
# 3. classifier.fit(X_train_projected, y_train) -- learn decision boundary
#
# And pipe.predict() does:
# 1. scaler.transform(X_test) -- scale using training stats
# 2. pca.transform(X_test_scaled) -- project using training components
# 3. classifier.predict(X_test_projected) -- make predictions

Watch Out

Common Mistake

Mistake: using fit_transform() on test data instead of just transform().

pythonreference · read-only

# WRONG -- the scaler learns from test data statistics
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # OK
X_test_scaled = scaler.fit_transform(X_test)    # BUG: re-fits on test data!

# RIGHT -- test data uses the mean/std learned from training
X_train_scaled = scaler.fit_transform(X_train)  # learns mean/std
X_test_scaled = scaler.transform(X_test)         # applies the same mean/std

#Why Pipelines Matter: Preventing Data Leakage

pythonreference · read-only

# DATA LEAKAGE -- a subtle but devastating bug
# BAD: fitting the scaler on ALL data (including test)
# scaler.fit(X)  # <-- sees test data statistics!
# X_scaled = scaler.transform(X)
# X_train, X_test = split(X_scaled)  # test set is "contaminated"

# GOOD: pipeline ensures test data is never seen during fit
pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("model", RandomForestClassifier(n_estimators=100, random_state=42)),
])

# fit() only sees training data, predict() applies the same transformations
pipe.fit(X_train, y_train)
print(f"No leakage accuracy: {pipe.score(X_test, y_test):.2%}")

#ColumnTransformer: Different Preprocessing for Different Columns

pythonrunnable cell

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

# Example dataset: mix of numeric and categorical features
data = pd.DataFrame({
    "age":        [25, 35, 45, 28, 52, 31],
    "income":     [50000, 80000, 120000, 62000, 95000, 71000],
    "education":  ["bachelor", "master", "phd", "bachelor", "master", "bachelor"],
    "city":       ["NYC", "SF", "NYC", "Austin", "SF", "NYC"],
    "bought":     [0, 1, 1, 0, 1, 1],
})

X = data.drop("bought", axis=1)
y = data["bought"]

# Identify column types
numeric_features = ["age", "income"]
categorical_features = ["education", "city"]

# ColumnTransformer: apply the right transformer to each group
preprocessor = ColumnTransformer(transformers=[
    ("num", StandardScaler(), numeric_features),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical_features),
    # Any columns NOT listed are dropped by default (set remainder="passthrough" to keep)
])

# Full pipeline: preprocess + model
full_pipeline = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", RandomForestClassifier(n_estimators=100, random_state=42)),
])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

full_pipeline.fit(X_train, y_train)
print(f"Accuracy: {full_pipeline.score(X_test, y_test):.2%}")

# Inspect what the preprocessor produces
X_transformed = preprocessor.fit_transform(X_train)
print(f"Original shape: {X_train.shape}")       # (4, 4)
print(f"Transformed shape: {X_transformed.shape}")  # (4, N) where N = 2 + num_one_hot_categories

#Unsupervised Learning with KMeans

Not all ML is supervised. KMeans finds clusters in data without labels.

pythonrunnable cell

from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs

# Generate data with 3 natural clusters
X_blobs, true_labels = make_blobs(n_samples=300, centers=3, random_state=42)

# KMeans does NOT get the labels -- it discovers clusters on its own
kmeans = KMeans(n_clusters=3, random_state=42, n_init=10)
kmeans.fit(X_blobs)

# Results
predicted_clusters = kmeans.labels_
centers = kmeans.cluster_centers_

print(f"Cluster centers:\n{centers}")
print(f"Inertia (lower = tighter clusters): {kmeans.inertia_:.2f}")

# Compare to true labels
from sklearn.metrics import adjusted_rand_score
ari = adjusted_rand_score(true_labels, predicted_clusters)
print(f"Adjusted Rand Index: {ari:.4f}")  # 1.0 = perfect match

#Choosing the Number of Clusters (Elbow Method)

pythonreference · read-only

inertias = []
K_range = range(1, 10)

for k in K_range:
    km = KMeans(n_clusters=k, random_state=42, n_init=10)
    km.fit(X_blobs)
    inertias.append(km.inertia_)

# The "elbow" in the plot indicates the best k
for k, inertia in zip(K_range, inertias):
    bar = "#" * int(inertia / 100)
    print(f"k={k}: inertia={inertia:>10.2f} {bar}")

# k=1: inertia=  13245.87 ################################################...
# k=2: inertia=   4935.21 ##################
# k=3: inertia=   1221.45 ####              <-- elbow! Adding more clusters helps less
# k=4: inertia=    981.12 ###
# k=5: inertia=    814.33 ##

Quick check

Inertia keeps falling as k rises — 13245, 4935, 1221, 981, 814. Why not simply pick the k with the lowest inertia?

#The Complete ML Pipeline

Here is a full, production-ready workflow combining everything:

pythonrunnable cell

import numpy as np
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report

# 1. Load data
wine = load_wine()
X, y = wine.data, wine.target
print(f"Dataset: {X.shape[0]} samples, {X.shape[1]} features, {len(wine.target_names)} classes")

# 2. Split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# 3. Build pipeline
pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("model", RandomForestClassifier(n_estimators=100, random_state=42)),
])

# 4. Cross-validation (more robust than a single train/test split)
cv_scores = cross_val_score(pipeline, X_train, y_train, cv=5, scoring="accuracy")
print(f"\nCross-validation scores: {cv_scores}")
print(f"Mean CV accuracy: {cv_scores.mean():.2%} (+/- {cv_scores.std():.2%})")

# 5. Train on full training set
pipeline.fit(X_train, y_train)

# 6. Final evaluation on test set
y_pred = pipeline.predict(X_test)
print(f"\nTest accuracy: {pipeline.score(X_test, y_test):.2%}")
print("\nDetailed Report:")
print(classification_report(y_test, y_pred, target_names=wine.target_names))

# 7. Feature importance (from the Random Forest inside the pipeline)
importances = pipeline.named_steps["model"].feature_importances_
feature_ranking = sorted(zip(wine.feature_names, importances), key=lambda x: -x[1])
print("Top 5 most important features:")
for name, importance in feature_ranking[:5]:
    bar = "#" * int(importance * 100)
    print(f"  {name:30s} {importance:.4f} {bar}")