MLOps: CI/CD for ML

MLOps is DevOps for machine learning. The same git → CI → deploy → monitor cycle, but with models, data, and statistics joined to it. Companies that get this right ship 10x faster. Most don't.

Learning Objectives

After this lesson, you will be able to:

Explain why standard software CI/CD is not enough for ML and what extra automation you need
Set up experiment tracking (using MLflow or W&B) so you can always reproduce any result
Build a model registry workflow that safely promotes models from 'experimental' to 'serving real users'

Prerequisites

Before You Start

Review

Why

Why This Matters

How Netflix updates its recommendation AI every week without breaking anything — Netflix's MLOps pipeline automatically retrains models on fresh viewing data, runs quality checks (is the new model actually better?), deploys to production servers, and rolls back instantly if anything goes wrong — all without a single user noticing; this continuous automation is what makes their recommendations feel like they are always getting smarter
Why your hackathon AI project dies the day after the demo — it lives in a Jupyter notebook with no version control, no experiment tracking, and no deployment pipeline; nobody else can run it, nobody knows which hyperparameters produced the "good" result, and the model cannot be updated without re-running the whole notebook from scratch; MLOps is the infrastructure that turns a one-time demo into a living product
How companies run hundreds of AI experiments simultaneously — Netflix and Spotify use MLOps to A/B test dozens of model versions at the same time, automatically tracking which version gets more engagement and promoting the winner to 100% of traffic; without this infrastructure, "improving the model" means guessing and hoping
Build this → Set up a mini CI/CD pipeline for ML: version your dataset and model with git, track experiments with a free tool (MLflow or Weights & Biases), and write a script that automatically retrains, evaluates against a baseline, and only "deploys" the new model if it beats the old one

#Software Engineering Had the Same Problem — 20 Years Ago

If you have ever lost track of which version of your model is "the good one," or asked yourself "wait, which hyperparameters produced that result?" — you already understand the pain that MLOps solves. This lesson gives you the tools to never have that feeling again.

Intuition

Build Your Intuition

In the early 2000s, software teams deployed code by manually copying files to servers. Builds broke constantly. Nobody knew which version was in production. Deployments were terrifying Friday afternoon affairs.

Then CI/CD (Continuous Integration / Continuous Delivery) changed everything. Automated builds. Automated tests. Automated deployments. Version control. Code review. The result: teams could deploy 100 times a day with confidence.

ML engineering in 2024 is where software engineering was in 2005. Most ML teams still train models in notebooks, manually copy model files, and have no idea which model version is serving traffic. MLOps is the CI/CD revolution for machine learning — bringing the same automation, reproducibility, and confidence to ML workflows.

But ML has unique challenges that standard CI/CD was never designed to handle: data versions, experiment tracking, model registries, feature drift, and the fact that "code correctness" is not binary — a model can be "correct" in that it runs, but still terrible at its job.

#The Three Dimensions of ML

Traditional software has one axis of change: code. ML has three:

The Three Dimensions of ML

ML System

Code (logic, features, serving)

Data (training data, labels, feature definitions)

Model (architecture, hyperparams, weights)

Code CI (traditional): linting, unit tests, integration tests, code review. Data CI (new): data validation, schema checks, distribution tests, data versioning. Model CI (new): training, evaluation, comparison to baseline, behavioral tests, fairness checks.

A change in any of these three dimensions can break production. MLOps must track and test all three.

#MLOps Maturity Levels

#Level 0: Manual Process

Level 0: Manual Process

Training is manual (run a notebook)
No experiment tracking (which run produced this model?)
Model is deployed by copying files
No monitoring (is the model still working?)
Pain point: "I cannot reproduce last month's results"

#Level 1: ML Pipeline Automation

Training is automated and triggered (schedule or data arrival)
Experiment tracking records every run
Automated evaluation gates
Model registry stores versioned artifacts
Pain point: "Changing the pipeline itself requires manual work"

#Level 2: CI/CD for the ML Pipeline

The pipeline code itself is tested and deployed via CI/CD
Changes to features, training logic, or serving code go through automated testing
A/B testing infrastructure for model comparison
Full observability across all three dimensions
This is the goal, but few organizations achieve it fully

What Do You Think?

A team at MLOps Level 0 is frustrated that their model performance degraded over 3 months but nobody noticed. What Level 1 capability would have caught this?

Try it: Explore the CI/CD Pipeline for MLInteractive

Loading visualization...

#The ML CI/CD Pipeline, Step by Step

In software engineering, CI/CD automates the path from code change to production. In ML, the pipeline is more complex because data and models change too. Here is the full CI/CD cycle for an ML system:

#Step 1: Code Commit

A developer pushes changes to the feature pipeline, training code, or serving logic. This triggers the CI pipeline, just like traditional software. The change could be a new feature definition, a hyperparameter tweak, or a bug fix in data preprocessing.

#Step 2: CI — Lint + Unit Tests

Automated checks run immediately: code linting, type checking, unit tests for feature pipeline functions, integration tests for data connectors. These catch obvious bugs before any expensive training runs. Fast feedback (under 5 minutes) keeps iteration speed high.

#Step 3: Training Pipeline Runs

If CI passes, the training pipeline kicks off. The orchestrator (Airflow, Kubeflow) runs the full DAG: data validation, feature engineering, model training. Every run logs hyperparameters, metrics, and artifacts to the experiment tracker. GPU time is expensive, so this step only runs when CI confirms code correctness.

#Step 4: Model Evaluation vs Baseline

The newly trained model is evaluated against the current production model on a held-out test set, behavioral tests, and fairness checks. Automated gates enforce minimum thresholds: accuracy above 0.85, latency below 100ms, fairness gap below 0.05. If any gate fails, the pipeline halts.

#Step 5: If Better — Promote to Staging

Models that pass all evaluation gates are registered in the model registry and promoted to the Staging environment. In staging, the model is tested with production-like traffic patterns, load tests, and integration tests against live data sources.

#Step 6: A/B Test in Production

The staging model is deployed as a canary, receiving 5-10% of live production traffic. For 24-72 hours, the system compares the canary model's business metrics (engagement, conversion, revenue) against the control (current production model). Statistical significance is required before proceeding.

#Step 7: Full Rollout

If the A/B test shows statistically significant improvement (or at minimum no degradation), the new model is promoted to 100% of production traffic. The old model is archived but kept available for instant rollback. Monitoring continues to watch for delayed degradation.

#Experiment Tracking: The Lab Notebook of ML

Every training run should record:

What	Why	Tool
Hyperparameters	Reproduce the run	MLflow, W&B
Metrics over time	Compare runs	MLflow, W&B
Code version	Link to exact source	Git SHA, DVC
Data version	Know what data was used	DVC, lakeFS
Environment	Reproduce the environment	Docker, conda lock
Model artifacts	Deploy the exact model	MLflow, S3/GCS
System metrics	Diagnose training issues	W&B, GPU monitoring

Try it! Sign up for a free MLflow or Weights & Biases account (both have free tiers). Run a simple training script and log one metric (like accuracy). Then change a hyperparameter and run it again. Open the dashboard and compare the two runs side by side — this is the "aha moment" where you see why experiment tracking matters.

pythonplayground.py · Pyodide

# Simulating an experiment tracker
# In production, this would be MLflow or W&B
import random
import time

class ExperimentTracker:
  def __init__(self, experiment_name: str):
      self.experiment_name = experiment_name
      self.runs: list[dict] = []
      self.active_run: dict | None = None

def start_run(self, run_name: str) -> "ExperimentTracker":
      self.active_run = {
          "name": run_name, "start_time": time.time(),
          "params": {}, "metrics": {}, "artifacts": [], "tags": {},
      }
      print(f"Started run: {run_name}")
      return self

def log_param(self, key: str, value) -> "ExperimentTracker":
      self.active_run["params"][key] = value
      return self

def log_metric(self, key: str, value: float, step: int) -> "ExperimentTracker":
      self.active_run["metrics"].setdefault(key, []).append({"value": value, "step": step})
      return self

def log_artifact(self, path: str) -> "ExperimentTracker":
      self.active_run["artifacts"].append(path)
      return self

def end_run(self):
      self.active_run["end_time"] = time.time()
      self.active_run["duration"] = self.active_run["end_time"] - self.active_run["start_time"]
      self.runs.append(self.active_run)
      print(f"Ended run: {self.active_run['name']} ({self.active_run['duration']:.3f}s)")
      self.active_run = None

def compare_runs(self):
      print(f"\n=== Experiment: {self.experiment_name} ===")
      print(f"{'Run Name':<20}{'LR':<10}{'Epochs':<10}{'Best Acc':<10}")
      print("-" * 50)
      for run in self.runs:
          acc_hist = run["metrics"].get("accuracy", [])
          best_acc = max((m["value"] for m in acc_hist), default=None)
          best_str = f"{best_acc:.4f}" if best_acc is not None else "N/A"
          lr = run["params"].get("learning_rate", "?")
          epochs = run["params"].get("epochs", "?")
          print(f"{run['name']:<20}{str(lr):<10}{str(epochs):<10}{best_str:<10}")

# Simulate 3 experiment runs
tracker = ExperimentTracker("sentiment-classifier")

# Run 1: baseline
tracker.start_run("baseline-lr0.001")
tracker.log_param("learning_rate", 0.001).log_param("epochs", 10).log_param("model", "bert-base")
for i in range(1, 11):
  tracker.log_metric("accuracy", 0.5 + 0.04 * i + random.random() * 0.01, i)
  tracker.log_metric("loss", 1.0 - 0.08 * i + random.random() * 0.02, i)
tracker.log_artifact("models/baseline-v1.pt")
tracker.end_run()

# Run 2: higher LR
tracker.start_run("high-lr-0.01")
tracker.log_param("learning_rate", 0.01).log_param("epochs", 10).log_param("model", "bert-base")
for i in range(1, 11):
  tracker.log_metric("accuracy", 0.5 + 0.05 * i - (0.03 * (i - 7) if i > 7 else 0) + random.random() * 0.01, i)
  tracker.log_metric("loss", 1.0 - 0.09 * i + (0.05 * (i - 7) if i > 7 else 0) + random.random() * 0.02, i)
tracker.log_artifact("models/high-lr-v1.pt")
tracker.end_run()

# Run 3: larger model
tracker.start_run("bert-large")
tracker.log_param("learning_rate", 0.001).log_param("epochs", 10).log_param("model", "bert-large")
for i in range(1, 11):
  tracker.log_metric("accuracy", 0.55 + 0.04 * i + random.random() * 0.01, i)
  tracker.log_metric("loss", 0.9 - 0.07 * i + random.random() * 0.02, i)
tracker.log_artifact("models/bert-large-v1.pt")
tracker.end_run()

tracker.compare_runs()

print("\nCHALLENGE: Add a 'get_best_run(metric)' method that returns the run with the best final value for a given metric.")

Tests · Run all 3 experiments and verify the comparison table shows the right values. Implement getBestRun and verify it returns the bert-large run for accuracy.

#Model Registry: Version Control for Models

A model registry is to models what Git is to code — a centralized store of versioned model artifacts with metadata:

Model Registry: sentiment-classifier

Version	Stage	Accuracy	Created
Version 1	Archived	0.82	2024-01-15
Version 2	Production	0.87	2024-02-01
Version 3	Staging	0.89	2024-02-15
Version 4	Development	0.91	2024-03-01

Stage transitions are gated by automated checks:

Development -> Staging: Offline metrics exceed threshold
Staging -> Production: A/B test shows improvement on business metrics
Any -> Archived: Superseded by a better version

Modern model registries (2024-2026 landscape)

Tool	Strengths	When to pick it
MLflow Model Registry	Self-hosted, framework-agnostic, mature	Default for self-hosted stacks
Hugging Face Hub Enterprise	Massive ecosystem, private repos, inference endpoints	If your team already lives on HF
Vertex AI Model Registry	Tight integration with Vertex pipelines + endpoints	All-in on Google Cloud
SageMaker Model Registry	IAM, lineage, deploy hooks, multi-account	All-in on AWS
BentoML / Bento Cloud	Service-oriented packaging (Bentos), built-in serving	Want model + serving config as one artifact
KitOps (ModelKit) / Replicate	OCI-compliant model artifacts, push to any registry	GitOps-first teams
W&B Models	Tight loop with experiment tracking	Already on W&B for tracking

#GitOps for Models

The pattern that won for application deployment (declarative manifests in Git, an operator reconciles cluster state) is now standard for ML:

ModelKit / KitOps packages model + config + dataset references as an OCI image. The same docker push toolchain works for models.
Flux / ArgoCD with custom CRDs (e.g., KServe InferenceService) reconciles model deployments from Git manifests.
Promotion-by-PR: changing the image: my-model:v3 line in a manifest and merging the PR is the production deploy. Rollback is a git revert.
Canary + shadow modes are declared in the manifest (traffic split percentages), not configured in a UI.

This gives ML the same audit trail, peer review, and instant rollback that backend services have enjoyed for a decade.

Data lineage from the NLP pretraining-pipelines lesson is the upstream half of every model registry entry — without it, you cannot reproduce a model.

#Data Versioning: The Missing Piece

Code versioning (Git) is solved. Model versioning (registries) is solved. Data versioning is the hardest remaining piece:

Why it matters

"Which dataset trained the model that is currently in production?"
"The model degraded — did the data change?"
"Can I reproduce last month's training run exactly?"

Tools

DVC (Data Version Control): Git-like commands for data. Stores metadata in Git, large files in S3/GCS.
lakeFS: Git-like branching for data lakes. Create a branch of your entire data lake, experiment, merge.
Delta Lake / Iceberg: Table formats with time travel — query the exact state of a table at any past timestamp.

#CI/CD Pipeline Definition

A complete ML CI/CD pipeline in YAML (conceptual):

yaml

# ml-pipeline.yaml (conceptual -- actual syntax depends on your orchestrator)
stages:
  data-validation:
    trigger: on-data-arrival
    steps:
      - validate-schema
      - check-distribution-drift
      - verify-completeness
    on-failure: alert-data-team

  feature-engineering:
    trigger: after-data-validation
    steps:
      - compute-features
      - validate-feature-distributions
      - write-to-feature-store

  training:
    trigger: after-feature-engineering OR on-schedule(weekly)
    steps:
      - pull-features-from-store
      - train-model
      - log-to-experiment-tracker
    resources:
      gpu: a100
      timeout: 4h

  evaluation:
    trigger: after-training
    steps:
      - evaluate-on-test-set
      - run-behavioral-tests
      - check-fairness-metrics
      - compare-against-baseline
    gates:
      accuracy: ">= 0.85"
      latency_p99: "<= 100ms"
      fairness_gap: "<= 0.05"

  deployment:
    trigger: after-evaluation-passes
    strategy: canary
    steps:
      - register-model
      - deploy-canary-5-percent
      - monitor-24h
      - promote-or-rollback

Watch Out

Common Mistake

"We have MLflow, therefore we have MLOps." Experiment tracking is one piece of MLOps. Without automated pipelines, validation gates, monitoring, and deployment automation, you just have a fancy log viewer.

"Our ML pipeline is a single Python script." A monolithic script is not a pipeline. When it fails at step 7 of 10, you cannot restart from step 7 — you restart from scratch. Proper pipelines have isolated, idempotent steps with defined inputs and outputs.

"We retrain every day to stay fresh." Retraining frequency should match the rate of data drift, not an arbitrary schedule. Retraining daily when your data distribution changes monthly wastes compute. Retraining monthly when it changes daily causes model staleness. Monitor drift and retrain when needed.

Ignoring the "last mile" of ML: Many teams invest heavily in training infrastructure but have no automated deployment, A/B testing, or rollback capability. The model sits in a registry while an engineer manually deploys it. This is like having a CI pipeline that runs tests but requires manual deployment.

#Key Takeaways

MLOps extends CI/CD for the unique challenges of ML. In addition to code changes, ML systems must track data changes, model versions, hyperparameters, and metrics, requiring purpose-built tooling beyond traditional DevOps
Experiment tracking enables reproducibility. Tools like MLflow and Weights & Biases log every hyperparameter, metric, and artifact for every run, so you can always reproduce or compare past experiments
Model registries provide safe promotion workflows. Models progress through stages (development, staging, production) with approval gates, automated tests, and rollback capabilities at each transition
Data and model versioning must work together. A model version is meaningless without the specific data version it was trained on; DVC and similar tools version data alongside code in the same Git workflow

#Quick Check

Quick Check1 / 4

What are the 'three axes' that change in ML systems, requiring more than traditional CI/CD?

Training pipelines at MLOps Level 2+ depend on the parallelism strategies covered in the NLP track for any model larger than a single GPU.

#Exit Ticket

With MLOps foundations in place, you can automate the full ML lifecycle. Next up: Model Serving & Inference — the art and engineering of serving model predictions to users at scale.

MLOps: CI/CD for ML

Learning Objectives

After this lesson, you will be able to:

Explain why standard software CI/CD is not enough for ML and what extra automation you need
Set up experiment tracking (using MLflow or W&B) so you can always reproduce any result
Build a model registry workflow that safely promotes models from 'experimental' to 'serving real users'

Prerequisites

Before You Start

Review

Why

Why This Matters

How Netflix updates its recommendation AI every week without breaking anything — Netflix's MLOps pipeline automatically retrains models on fresh viewing data, runs quality checks (is the new model actually better?), deploys to production servers, and rolls back instantly if anything goes wrong — all without a single user noticing; this continuous automation is what makes their recommendations feel like they are always getting smarter
Why your hackathon AI project dies the day after the demo — it lives in a Jupyter notebook with no version control, no experiment tracking, and no deployment pipeline; nobody else can run it, nobody knows which hyperparameters produced the "good" result, and the model cannot be updated without re-running the whole notebook from scratch; MLOps is the infrastructure that turns a one-time demo into a living product
How companies run hundreds of AI experiments simultaneously — Netflix and Spotify use MLOps to A/B test dozens of model versions at the same time, automatically tracking which version gets more engagement and promoting the winner to 100% of traffic; without this infrastructure, "improving the model" means guessing and hoping
Build this → Set up a mini CI/CD pipeline for ML: version your dataset and model with git, track experiments with a free tool (MLflow or Weights & Biases), and write a script that automatically retrains, evaluates against a baseline, and only "deploys" the new model if it beats the old one

#Software Engineering Had the Same Problem — 20 Years Ago

Intuition

Build Your Intuition

#The Three Dimensions of ML

Traditional software has one axis of change: code. ML has three:

The Three Dimensions of ML

ML System

Code (logic, features, serving)

Data (training data, labels, feature definitions)

Model (architecture, hyperparams, weights)

A change in any of these three dimensions can break production. MLOps must track and test all three.

#MLOps Maturity Levels

#Level 0: Manual Process

Level 0: Manual Process

Training is manual (run a notebook)
No experiment tracking (which run produced this model?)
Model is deployed by copying files
No monitoring (is the model still working?)
Pain point: "I cannot reproduce last month's results"

#Level 1: ML Pipeline Automation

Level 1: ML Pipeline Automation

📨Trigger

🛡️Data Validation

🔧Feature Eng

🧠Train

📊Evaluate

🔌Deploy (if passing)

📨 queue🛡️ guardrail🔧 tool🧠 llm📊 evaluator🔌 api

Training is automated and triggered (schedule or data arrival)
Experiment tracking records every run
Automated evaluation gates
Model registry stores versioned artifacts
Pain point: "Changing the pipeline itself requires manual work"

#Level 2: CI/CD for the ML Pipeline

Level 2: CI/CD for the ML Pipeline

🔧Code Change

📊Test Pipeline

🔧Build Pipeline

🔌Deploy Pipeline

📊Monitor

🔧 tool📊 evaluator🔌 api

The pipeline code itself is tested and deployed via CI/CD
Changes to features, training logic, or serving code go through automated testing
A/B testing infrastructure for model comparison
Full observability across all three dimensions
This is the goal, but few organizations achieve it fully

What Do You Think?

A team at MLOps Level 0 is frustrated that their model performance degraded over 3 months but nobody noticed. What Level 1 capability would have caught this?

Try it: Explore the CI/CD Pipeline for MLInteractive

Loading visualization...

#The ML CI/CD Pipeline, Step by Step

#Step 1: Code Commit

#Step 2: CI — Lint + Unit Tests

#Step 3: Training Pipeline Runs

#Step 4: Model Evaluation vs Baseline

#Step 5: If Better — Promote to Staging

#Step 6: A/B Test in Production

#Step 7: Full Rollout

#Experiment Tracking: The Lab Notebook of ML

Every training run should record:

What	Why	Tool
Hyperparameters	Reproduce the run	MLflow, W&B
Metrics over time	Compare runs	MLflow, W&B
Code version	Link to exact source	Git SHA, DVC
Data version	Know what data was used	DVC, lakeFS
Environment	Reproduce the environment	Docker, conda lock
Model artifacts	Deploy the exact model	MLflow, S3/GCS
System metrics	Diagnose training issues	W&B, GPU monitoring

Try it! Sign up for a free MLflow or Weights & Biases account (both have free tiers). Run a simple training script and log one metric (like accuracy). Then change a hyperparameter and run it again. Open the dashboard and compare the two runs side by side — this is the "aha moment" where you see why experiment tracking matters.

pythonplayground.py · Pyodide

# Simulating an experiment tracker
# In production, this would be MLflow or W&B
import random
import time

class ExperimentTracker:
  def __init__(self, experiment_name: str):
      self.experiment_name = experiment_name
      self.runs: list[dict] = []
      self.active_run: dict | None = None

def log_param(self, key: str, value) -> "ExperimentTracker":
      self.active_run["params"][key] = value
      return self

def log_metric(self, key: str, value: float, step: int) -> "ExperimentTracker":
      self.active_run["metrics"].setdefault(key, []).append({"value": value, "step": step})
      return self

def log_artifact(self, path: str) -> "ExperimentTracker":
      self.active_run["artifacts"].append(path)
      return self

# Simulate 3 experiment runs
tracker = ExperimentTracker("sentiment-classifier")

tracker.compare_runs()

print("\nCHALLENGE: Add a 'get_best_run(metric)' method that returns the run with the best final value for a given metric.")

Tests · Run all 3 experiments and verify the comparison table shows the right values. Implement getBestRun and verify it returns the bert-large run for accuracy.

#Model Registry: Version Control for Models

A model registry is to models what Git is to code — a centralized store of versioned model artifacts with metadata:

Model Registry: sentiment-classifier

Version	Stage	Accuracy	Created
Version 1	Archived	0.82	2024-01-15
Version 2	Production	0.87	2024-02-01
Version 3	Staging	0.89	2024-02-15
Version 4	Development	0.91	2024-03-01

Stage transitions are gated by automated checks:

Development -> Staging: Offline metrics exceed threshold
Staging -> Production: A/B test shows improvement on business metrics
Any -> Archived: Superseded by a better version

Modern model registries (2024-2026 landscape)

Tool	Strengths	When to pick it
MLflow Model Registry	Self-hosted, framework-agnostic, mature	Default for self-hosted stacks
Hugging Face Hub Enterprise	Massive ecosystem, private repos, inference endpoints	If your team already lives on HF
Vertex AI Model Registry	Tight integration with Vertex pipelines + endpoints	All-in on Google Cloud
SageMaker Model Registry	IAM, lineage, deploy hooks, multi-account	All-in on AWS
BentoML / Bento Cloud	Service-oriented packaging (Bentos), built-in serving	Want model + serving config as one artifact
KitOps (ModelKit) / Replicate	OCI-compliant model artifacts, push to any registry	GitOps-first teams
W&B Models	Tight loop with experiment tracking	Already on W&B for tracking

#GitOps for Models

The pattern that won for application deployment (declarative manifests in Git, an operator reconciles cluster state) is now standard for ML:

ModelKit / KitOps packages model + config + dataset references as an OCI image. The same docker push toolchain works for models.
Flux / ArgoCD with custom CRDs (e.g., KServe InferenceService) reconciles model deployments from Git manifests.
Promotion-by-PR: changing the image: my-model:v3 line in a manifest and merging the PR is the production deploy. Rollback is a git revert.
Canary + shadow modes are declared in the manifest (traffic split percentages), not configured in a UI.

This gives ML the same audit trail, peer review, and instant rollback that backend services have enjoyed for a decade.

Data lineage from the NLP pretraining-pipelines lesson is the upstream half of every model registry entry — without it, you cannot reproduce a model.

#Data Versioning: The Missing Piece

Code versioning (Git) is solved. Model versioning (registries) is solved. Data versioning is the hardest remaining piece:

Why it matters

"Which dataset trained the model that is currently in production?"
"The model degraded — did the data change?"
"Can I reproduce last month's training run exactly?"

Tools

DVC (Data Version Control): Git-like commands for data. Stores metadata in Git, large files in S3/GCS.
lakeFS: Git-like branching for data lakes. Create a branch of your entire data lake, experiment, merge.
Delta Lake / Iceberg: Table formats with time travel — query the exact state of a table at any past timestamp.

#CI/CD Pipeline Definition

A complete ML CI/CD pipeline in YAML (conceptual):

yaml

# ml-pipeline.yaml (conceptual -- actual syntax depends on your orchestrator)
stages:
  data-validation:
    trigger: on-data-arrival
    steps:
      - validate-schema
      - check-distribution-drift
      - verify-completeness
    on-failure: alert-data-team

  feature-engineering:
    trigger: after-data-validation
    steps:
      - compute-features
      - validate-feature-distributions
      - write-to-feature-store

  training:
    trigger: after-feature-engineering OR on-schedule(weekly)
    steps:
      - pull-features-from-store
      - train-model
      - log-to-experiment-tracker
    resources:
      gpu: a100
      timeout: 4h

  evaluation:
    trigger: after-training
    steps:
      - evaluate-on-test-set
      - run-behavioral-tests
      - check-fairness-metrics
      - compare-against-baseline
    gates:
      accuracy: ">= 0.85"
      latency_p99: "<= 100ms"
      fairness_gap: "<= 0.05"

  deployment:
    trigger: after-evaluation-passes
    strategy: canary
    steps:
      - register-model
      - deploy-canary-5-percent
      - monitor-24h
      - promote-or-rollback

Watch Out

Common Mistake

#Key Takeaways

MLOps extends CI/CD for the unique challenges of ML. In addition to code changes, ML systems must track data changes, model versions, hyperparameters, and metrics, requiring purpose-built tooling beyond traditional DevOps
Experiment tracking enables reproducibility. Tools like MLflow and Weights & Biases log every hyperparameter, metric, and artifact for every run, so you can always reproduce or compare past experiments
Model registries provide safe promotion workflows. Models progress through stages (development, staging, production) with approval gates, automated tests, and rollback capabilities at each transition
Data and model versioning must work together. A model version is meaningless without the specific data version it was trained on; DVC and similar tools version data alongside code in the same Git workflow

#Quick Check

Quick Check1 / 4

What are the 'three axes' that change in ML systems, requiring more than traditional CI/CD?

Training pipelines at MLOps Level 2+ depend on the parallelism strategies covered in the NLP track for any model larger than a single GPU.

#Exit Ticket

With MLOps foundations in place, you can automate the full ML lifecycle. Next up: Model Serving & Inference — the art and engineering of serving model predictions to users at scale.

MLOps: CI/CD for ML

#Software Engineering Had the Same Problem — 20 Years Ago

#The Three Dimensions of ML

#MLOps Maturity Levels

#Level 0: Manual Process

#Level 1: ML Pipeline Automation

#Level 2: CI/CD for the ML Pipeline

#The ML CI/CD Pipeline, Step by Step

#Step 1: Code Commit

#Step 2: CI — Lint + Unit Tests

#Step 3: Training Pipeline Runs

#Step 4: Model Evaluation vs Baseline

#Step 5: If Better — Promote to Staging

#Step 6: A/B Test in Production

#Step 7: Full Rollout

#Experiment Tracking: The Lab Notebook of ML

#Model Registry: Version Control for Models

Modern model registries (2024-2026 landscape)

#GitOps for Models

#Data Versioning: The Missing Piece

Why it matters

Tools

#CI/CD Pipeline Definition

#Key Takeaways

#Quick Check

Deep Dive

Related Lessons

#Exit Ticket

Your Reflection

Continue Learning

Discussion

MLOps: CI/CD for ML

#Software Engineering Had the Same Problem — 20 Years Ago

#The Three Dimensions of ML

#MLOps Maturity Levels

#Level 0: Manual Process

#Level 1: ML Pipeline Automation

#Level 2: CI/CD for the ML Pipeline

#The ML CI/CD Pipeline, Step by Step

#Step 1: Code Commit

#Step 2: CI — Lint + Unit Tests

#Step 3: Training Pipeline Runs

#Step 4: Model Evaluation vs Baseline

#Step 5: If Better — Promote to Staging

#Step 6: A/B Test in Production

#Step 7: Full Rollout

#Experiment Tracking: The Lab Notebook of ML

#Model Registry: Version Control for Models

Modern model registries (2024-2026 landscape)

#GitOps for Models

#Data Versioning: The Missing Piece

Why it matters

Tools

#CI/CD Pipeline Definition

#Key Takeaways

#Quick Check

Deep Dive

Related Lessons

#Exit Ticket

Your Reflection

Continue Learning

Discussion