Experiment Tracking & Model Versioning

"Which run produced our best model?" is the most expensive question in ML. If you can't answer it in 30 seconds — code commit, data version, hyperparameters, environment — you've already lost weeks of work. Weights & Biases, MLflow, and DVC exist so that answer is one click away.

Learning Objectives

After this lesson, you will be able to:

Track every experiment's hyperparameters, metrics, code version, data version, and artifacts in a system of record (W&B, MLflow, Comet, Neptune)
Use a model registry to manage model versions, lifecycle stages, and approvals for production deployment
Apply DVC or LakeFS to version datasets alongside code so any experiment is fully reproducible months later
Recognize when 'it worked on my notebook' becomes 'I can't reproduce my best model from 6 months ago' — and the patterns that prevent it

Prerequisites

Before You Start

Review

Why

Why This Matters

Why "I can't reproduce my best model from 6 months ago" is the most expensive ML team mistake -- without experiment tracking, you've lost weeks of work permanently; W&B and MLflow exist precisely to prevent this
Why W&B became the dominant tracking platform in 2024-2026 -- $1B+ valuation, used by OpenAI, Anthropic, Stability, every major lab; the network effect of "everyone's models are tracked there"
Why model registries became regulatory requirements -- EU AI Act compliance requires lineage from data → training run → deployed model; without a registry, you can't prove which model handled a specific user request
Build this --> Wrap a PyTorch training loop with MLflow tracking; log hyperparameters, metrics, gradient norms, and the model itself; navigate the MLflow UI to compare 5 runs and pick the best

Layer	What	Why
Hyperparameters	LR, batch size, model size, dropout, optimizer	Reproducibility
Metrics	Train/val/test loss, accuracy, F1, custom domain metrics	Best-model selection
System metrics	GPU utilization, memory, FLOPs, time-per-step	Cost optimization
Code	Git commit SHA, branch, dirty-state warning	Reproducibility
Data	Dataset version (DVC hash), preprocessing config	Reproducibility
Artifacts	Model weights, configs, plots, sample outputs	Comparison + audit
Environment	Python version, package versions (`pip freeze`), CUDA version	Reproducibility

Tool	Hosted / OSS	Strengths	Weaknesses
W&B (Weights & Biases)	Hosted SaaS + self-host	Best UI; collaboration features (reports, sweeps, tables); huge ecosystem	Most expensive at scale; vendor lock-in
MLflow	OSS, you host	Free; widely integrated (Databricks built-in); model registry mature	UI dated; multi-team collaboration weaker
Comet	Hosted SaaS	Strong on artifacts and reproducibility reports; experiment diffing	Smaller community than W&B
Neptune	Hosted SaaS	Lightweight; very fast logging; clean Python API	Fewer integrations; less collaboration tooling
ClearML	OSS + hosted	Adds orchestration + agents (think W&B + light Airflow); good for self-hosted teams	Steeper learning curve

Tool	Best for
W&B (Weights & Biases)	Industry standard, best UI, expensive at scale
MLflow	Open-source, self-hostable, good baseline
Comet	Strong artifacts/reports, good for research teams
Neptune	Lightweight, niche but loyal users
TensorBoard	Free, good for local; struggles at team scale
ClearML	Open-source W&B alternative

Stage	Meaning
None	Just trained, no decision made
Staging	Candidate for production, in evaluation
Production	Currently serving real traffic
Archived	Retired, kept for audit/lineage

Experiment Tracking & Model Versioning

After this lesson, you will be able to:

Before You Start

Your Reflection

Continue Learning

Discussion

#What to Track

#The Major Players in Depth

#What to track at minimum

#Hyperparameter Sweeps and the Tie-In to Track 03

#Leaderboards and Collaboration Patterns

#Tools (2026)

#Model Registry: Lifecycle Stages

#Data Versioning with DVC

#Hands-On

#A Minimal Logging Stub (W&B and MLflow side-by-side)

#Key Takeaways

#Quick Check

Deep Dive