Track 10 · ML Engineering · 13 min
From notebook to production.
The hardest part of ML isn't training — it's keeping a model alive. Serving, scaling, monitoring, retraining, paying the GPU bill. This is the field manual for what happens after the model.fit() — and what makes a senior ML engineer worth their weight in H100s.
“Most of the time when you read a machine learning paper, the authors get a great result on a benchmark and then say 'and now we deploy it'. The 'and now we deploy it' is where most of the work is.”
#The hook
This is the field manual for that work — the part of "AI" nobody Instagrams about, but where the actual revenue lives.
#Why this matters in 2026 — the receipts
MLE by the numbers
The cost of doing AI in prod
87%
ML projects that never reach production
VentureBeat 2024
30%+
Of model TCO is GPU compute
Modal/Anyscale benchmarks
60%
Models in prod that drift in 6 months
MLflow community 2025
$250B+
MLOps market by 2030
IDC 2025
#The seven pillars
The job
Seven concerns of any production ML system
1. Pipelines
Reproducible trainingData in → model out, repeatably. Airflow / Dagster / Prefect / KubeFlow.
- Every step versioned: data version, code version, config version, model version.
- Reproducibility is the entry-level bar. Without it nothing else works.
- DAG-based orchestrators are the field standard.
2. Serving
Inference at scaleMake the model answer requests fast, cheap, reliable.
- vLLM, TensorRT-LLM, Triton, TGI — the serving stack.
- Quantization (INT8, INT4) cuts cost by 2-4x with minimal quality loss.
- Latency budgets matter. p99 < 200ms is a common SLO.
3. Monitoring
Catch drift before users doTrack input distributions, prediction distributions, ground-truth feedback.
- Data drift, concept drift, prediction drift — all monitored separately.
- Evidently, Arize, WhyLabs, custom Prometheus — the toolset.
- PSI, KS-test, JSD — the drift metrics.
4. Evaluation
Know what 'better' meansOffline benchmarks, online A/B tests, shadow deployments.
- Champion-challenger: deploy the new model in shadow, compare against prod.
- Statistical significance matters — don't roll out on noise.
- LLM-as-judge for generative models. Eppo, Statsig, custom — the platforms.
5. FinOps
Pay the GPU billGPU hours are the new AWS bill. Quantize, cache, batch, multiplex.
- vLLM continuous batching → 5-10x throughput.
- KV-cache reuse → ~30% latency reduction.
- Spot/preemptible GPUs on H100 + autoscaling = real cost wins.
6. Compliance & governance
Don't get suedModel cards, data lineage, audit logs. EU AI Act compliance.
- Model cards document intended use, limitations, training data.
- EU AI Act came into force 2025. Bias evaluations, transparency, human review.
- SOC 2, HIPAA, ISO 42001 — the compliance alphabet.
7. Incident response
When (not if)Rollback, blameless post-mortems, runbooks. Treat ML like any production system.
- Rollback to previous model version in <5 min.
- Runbooks for: latency spike, drift alert, bad prediction, hallucination report.
- Most ML incidents are data incidents — fix at the source.
87%
of ML projects never reach production
Year after year, the same number from VentureBeat, Gartner, and IDC: most trained models never serve a real user. The reasons are almost never about the model — they're about data pipelines, monitoring, integration, and the dozen non-glamorous concerns ML Engineering exists to handle.
VentureBeat 'AI Stack' 2024
Vocabulary
Six MLE terms you'll see daily
Concept
Feature store
Single store of features for training and serving — eliminates skew.
Like: One pantry for the home kitchen and food truck.
e.g. Tecton, Feast, Hopsworks
Concept
Drift
Input or relationship distributions shift over time.
Like: Yesterday's map of a changing city.
e.g. PSI, KS-test detect it
Concept
Shadow deployment
Run the new model alongside prod, compare outputs, don't serve users.
Like: Junior pilot in the cockpit before flying solo.
e.g. Champion-challenger pattern
Concept
KV cache
Reuse the attention-key/value cache across LLM tokens for speed.
Like: Don't redo addition you already did.
e.g. Cuts latency 30%
Concept
Continuous batching
Stitch in-flight requests together for higher GPU utilization.
Like: Carpooling — fill empty seats.
e.g. vLLM's signature trick
Concept
Model card
A doc describing intended use, limitations, training data, metrics.
Like: Nutrition label for an ML model.
e.g. Required by EU AI Act in regulated sectors
#A real serving + monitoring loop — runnable
# A miniature serving + drift monitoring loop
import numpy as np
import time
# Pretend "model" — predicts whether a number is large
def model_predict(x):
return 1 if x > 0.5 else 0
# Simulated production traffic (slowly drifting)
def get_traffic_batch(epoch):
if epoch < 3:
# Early: requests roughly uniform
return np.random.uniform(0, 1, size=200)
else:
# Drift: the world has shifted toward larger values
return np.random.beta(5, 2, size=200)
# Population Stability Index
def psi(reference, current, bins=10):
cuts = np.percentile(reference, np.linspace(0, 100, bins + 1))
cuts[0], cuts[-1] = -np.inf, np.inf
ref_pct = np.histogram(reference, cuts)[0] / len(reference) + 1e-6
cur_pct = np.histogram(current, cuts)[0] / len(current) + 1e-6
return float(np.sum((cur_pct - ref_pct) * np.log(cur_pct / ref_pct)))
# Reference distribution from training
training_distribution = np.random.uniform(0, 1, size=2000)
# Serve traffic + monitor
print(f"{'Epoch':>6}{'Latency':>10}{'Predictions':>14}{'Drift':>10}{'Status':>15}")
print("-" * 55)
for epoch in range(6):
batch = get_traffic_batch(epoch)
start = time.time()
predictions = [model_predict(x) for x in batch]
latency_ms = (time.time() - start) * 1000
drift = psi(training_distribution, batch)
status = "OK" if drift < 0.1 else "DRIFT!" if drift < 0.25 else "RETRAIN!"
print(f"{epoch:>6}{latency_ms:>9.1f}ms{sum(predictions):>14}{drift:>10.3f}{status:>15}")That's a tiny but real serving + monitoring loop. You can see the drift kick in around epoch 3, and the monitoring catch it.
#What's been built with ML Engineering
MLE in production
Where the serious GPU bills get paid
LLM serving
ChatGPT (serving)
1B+
Daily messages
Multi-region inference clusters. Continuous batching. KV-cache reuse. Custom inference stack atop NVIDIA Triton / TensorRT-LLM.
Serving
Serverless GPU
Modal Labs
$100M+
ARR run rate
Container-fast cold starts. ML teams use it instead of EC2/SageMaker. The new default for many.
FinOps
Distributed compute
Anyscale / Ray
50K+
Distributed clusters
Open-source distributed framework that powers ChatGPT training. Now standard for ML clusters.
Pipelines
Experiment tracking
Weights & Biases
1M+
Experiments tracked daily
Acquired by CoreWeave (2024). The default 'observability for training runs' platform.
Tracking
Drift monitoring
Evidently / Arize
30%+
Drift catches before users
Open-source (Evidently) and SaaS (Arize) for production drift detection. The early-warning system.
Monitoring
Feature stores
Tecton / Feast
60%+
Of unicorns running feature stores
Same feature for offline training and online serving — eliminates train/serve skew.
Feature stores
#The 2026 frontier
#Where to go next
- ML Engineering track — 19 lessons: pipelines, serving, monitoring, FinOps, EU AI Act, incident response.
- Data Foundations — most production ML failures are data failures.
- Claude Code Mastery — the AI engineer's daily-driver workflow.
- AI Agents — agent platforms now run on top of ML Engineering infrastructure.
#Key takeaways
Key Takeaways
- ML Engineering is what bridges 'great model' and 'shipped product'. Most teams need it more than they need a better model.
- Seven pillars: pipelines, serving, monitoring, evaluation, FinOps, compliance, incident response.
- Production ML is 90% monitoring. Drift is the silent killer.
- GPU costs dominate TCO. Continuous batching, quantization, KV-cache reuse — the levers.
- EU AI Act came into force 2025. Bias evaluations, transparency, human review are now mandatory in regulated sectors.
- Treat ML like any production system: version everything, alert on SLOs, runbook for every failure mode.
#References & further reading
- Designing Machine Learning Systems by Chip Huyen — the production ML bible.
- Made With ML by Goku Mohandas — practical, free, recently updated.
- vLLM blog (vllm.ai/blog) — best engineering blog on LLM serving.
- Hugging Face Inference Endpoints docs — practical production LLM patterns.
- EU AI Act official text + summaries — required reading for any regulated AI work.
- Modal/Anyscale/Banana engineering blogs — modern serving infra.