Track 03 · Classical ML · 13 min
The 80% of AI that isn't an LLM.
Linear regression, decision trees, XGBoost. They power your bank's fraud detection, your insurance's pricing, your Netflix queue. Modern AI didn't replace classical ML — it sits on top. Here's the field manual, with seven interactive demos you can play with as you read.
“In production ML, 80% of the time you'd be better off with logistic regression and good features than with the fanciest neural network in the world.”
#The hook
Random forests. Logistic regression. XGBoost. SHAP. These aren't obsolete relics — they're the silent workhorses paying the rent for almost every "data science team" in industry. Your credit score, your insurance premium, the ads you see, the products Amazon recommends — almost all classical ML.
#Why this matters in 2026 — the receipts
Classical ML by the numbers
Quietly running the world
80%
Production ML jobs that are tabular
Kaggle 2024 survey
70%
Kaggle wins still using XGBoost
Kaggle leaderboards
$1B+
Annual fraud blocked by classical ML
Visa, Mastercard 2025
50ms
Avg classical-model inference
industry benchmarks
#The four model families
The map
Four families. Pick one, ship a baseline.
1. Linear models
Logistic + linear regressionThe simplest model that ever works. Always your first try.
- Interpretable: each coefficient is a 'how much this feature matters'.
- Trains in milliseconds even on millions of rows.
- Surprisingly hard to beat on well-engineered tabular features.
2. Tree-based
Decision trees → forests → boostingThe 80% answer for tabular data. XGBoost is the deserved king.
- Random forests: ensemble of trees, each on a random subset. Hard to overfit.
- XGBoost / LightGBM / CatBoost: gradient-boosted trees. Owns Kaggle leaderboards.
- No feature scaling needed. Handles missing values. Reads like a flowchart.
3. SVMs and kernel methods
When you need nonlinear marginsMathematical elegance. Useful in 2026 for small text/image classification with limited data.
- Find the maximum-margin separator in a high-dimensional space.
- Kernels let you handle non-linear boundaries without explicit feature engineering.
- Less common in 2026, but elegant and worth understanding.
4. Unsupervised + clustering
When you don't have labelsFind structure in unlabeled data. K-means, DBSCAN, PCA, t-SNE.
- Customer segmentation, anomaly detection, dimension reduction for visualization.
- PCA / t-SNE / UMAP — the way you find clusters in an unlabeled embedding.
- Don't underestimate: 30% of real ML problems start unsupervised.
70%
of Kaggle competition winners use XGBoost
Year after year, the winning solutions on tabular Kaggle competitions are dominated by gradient-boosted trees — XGBoost, LightGBM, CatBoost. The neural-network exception happens almost only on image, text, or sequence problems where deep learning is structurally a better fit.
Kaggle 'State of Data Science' surveys 2020-2024
Vocabulary
Six classical-ML terms you'll see daily
Concept
Bias-variance
Underfit (high bias) vs overfit (high variance) — the central tradeoff.
Like: Aiming a dart at the bull's eye: bias is being off-center, variance is shaky hands.
e.g. Deeper tree → less bias, more variance
Concept
Cross-validation
Train and test on K different folds of the data — robust accuracy estimate.
Like: Don't grade on one test — average over five.
e.g. K=5 is the field default
Concept
Regularization
Penalize model complexity to prevent overfit.
Like: A deadline for an essay — forces you to pick what matters.
e.g. L1 (lasso) zeros out features; L2 (ridge) shrinks them
Concept
Hyperparameter tuning
Search over model 'knobs' (depth, lr, etc.) for the best setting.
Like: Tasting and adjusting seasoning.
e.g. Optuna, Bayesian search, random search
Concept
Feature importance
Which inputs the model relies on most.
Like: What evidence the detective trusts.
e.g. SHAP values, gain importance
Concept
Calibration
Do the model's '70% confident' answers actually win 70% of the time?
Like: Weather forecast: when it says 70% rain, does it rain 70% of those days?
e.g. Reliability diagrams, Platt scaling
#Linear models — interactive
The simplest learnable model: a line through the data. It's still the right answer surprisingly often.
#Decision trees and forests — the daily driver
For tabular data, gradient-boosted trees (XGBoost / LightGBM) are still the right answer in 2026 for a huge fraction of real problems. Here's the intuition:
Run a real XGBoost-like ensemble in your browser:
# Tabular ML done right — 80% of real ML problems look like this
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
import numpy as np
# Make a realistic tabular dataset
X, y = make_classification(n_samples=2000, n_features=20, n_informative=10,
n_redundant=4, random_state=42)
models = {
"Logistic regression": LogisticRegression(max_iter=2000),
"Random forest": RandomForestClassifier(n_estimators=100, random_state=42),
"Gradient boosting": GradientBoostingClassifier(n_estimators=100, random_state=42),
}
print(f"{'Model':25}{'CV accuracy':>15}")
print("-" * 40)
for name, model in models.items():
scores = cross_val_score(model, X, y, cv=5)
print(f"{name:25}{np.mean(scores):>15.3f} ± {np.std(scores):.3f}")#Clustering — for when there are no labels
#Diagnostics — the senior-engineer toolkit
#What's been built on classical ML
The boring half-trillion
Systems built on logistic regression, trees, and ensembles
Logistic regression
FICO Score
90%
US lending decisions
Your credit score is a logistic regression on ~50 features. Trained in 1989. Still rules.
Linear models
Tree-based ensemble
Visa fraud system
$1B+
Fraud blocked / yr
Real-time fraud scoring. XGBoost + custom features. Decision in <50ms per transaction.
XGBoost
GLM + GBM
Allstate pricing
100M+
Policies priced
Generalized linear models for the regulated layer; GBMs underneath for accuracy. Pure classical ML, billion-dollar margins.
Linear + GBM
Naive Bayes → SVMs
Spam filters
99%
Spam catch rate
From 1998 Bayesian filters to modern ensembled classifiers — one of the longest-running win-lists in applied ML.
Probabilistic
K-means + RFM
Customer segmentation
$5B+
Marketing spend driven
Every retailer with a CRM runs k-means on RFM features. Old, unglamorous, hugely valuable.
Unsupervised
Logistic regression
Google AdWords CTR
$200B+
Annual ad revenue
The original Google AdWords click prediction was logistic regression at planetary scale. Pure classical ML, billion-dollar feedback loops.
Linear at scale
#Where to go next
- Classical ML track — 18 lessons across the four families, with diagnostics and tuning.
- Math Foundations — gradient descent, calibration, probability — the foundation underneath.
- Data Foundations — feature engineering and CV are 50% of classical ML wins.
- Deep Learning — when tabular runs out of road, here's where you go next.
#Key takeaways
Key Takeaways
- Classical ML is still the right answer for ~80% of tabular-data problems in industry.
- Four families: linear models, tree ensembles (XGBoost), kernel/SVM, unsupervised + clustering.
- Always start with logistic regression as a baseline. If a more complex model can't beat it, the data is the problem.
- Random forests and gradient-boosted trees own tabular ML. XGBoost / LightGBM / CatBoost are the field standard.
- Diagnostics matter more than algorithms. Learning curves, calibration plots, SHAP explanations.
- Deep learning hasn't replaced classical ML. They live side-by-side; pick by problem shape and data size.
#References & further reading
- The Elements of Statistical Learning by Hastie, Tibshirani, Friedman (free PDF). The textbook.
- Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow by Aurélien Géron — the practical companion.
- XGBoost: A Scalable Tree Boosting System (Chen & Guestrin, 2016). The paper that defined modern tabular.
- A Unified Approach to Interpreting Model Predictions (Lundberg & Lee, 2017). Introduced SHAP.
- Kaggle Solutions Recap (kaggle.com/competitions) — see what actually wins on real data.