Without observability, your AI system is a black box you pray to. With it, you debug in minutes. Companies that ship AI products at scale all converge on the same stack: traces, metrics, logs, evals — plus AI-specific monitoring for hallucination rate, retrieval quality, and drift.
Learning Objectives
After this lesson, you will be able to:
Design a monitoring system that catches data changes, model quality drops, and performance issues before your users notice
Set up tracing for LLM applications so you can see exactly what happened inside a complex AI request
Build alerts and dashboards that tell you what actually matters -- without drowning you in noise
If you have ever shipped a feature and assumed "no errors = everything is fine," this lesson will change how you think about production systems forever. ML models can silently serve terrible results for weeks without a single error log. Let's learn how to catch that.
The three pillars of observability answer different questions. Metrics are aggregated numbers over time — latency p99, throughput, GPU utilisation — and tell you WHAT is happening. Logs are individual timestamped events and tell you WHAT HAPPENED in one specific case. Traces follow a single request across every service and component it touched, and tell you WHERE the time went. You need all three: metrics reveal that something broke, logs reveal what, traces reveal where.
The three pillars: metrics, logs, traces
Traditional observability has three pillars: logs, metrics, and traces. ML observability extends each:
Compare binned distributions of training vs production data
Numerical features
KS Test (Kolmogorov-Smirnov)
Maximum distance between two CDFs
Numerical features
Chi-squared Test
Compare categorical frequency distributions
Categorical features
Jensen-Shannon Divergence
Symmetric KL divergence between distributions
Any distribution
Embedding drift
Distance between embedding centroids over time
Text, image inputs
Slide the production distribution away from the training baseline to see how drift detection surfaces the shift.
Loading visualization...
Try it! Simulate drift yourself: generate two sets of random numbers, one from random.gauss(0, 1) and another from random.gauss(0.5, 1) (a small mean shift). Compute the mean of each set and compare them. That 0.5 shift is data drift in action -- small enough to miss by eye, large enough to degrade a model.
pythonplayground.py · Pyodide
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
Tests · Verify PSI is near 0 for identical distributions, moderate for slight shifts, and large for significant shifts.
The relationship between inputs and outputs changes, even if the input distribution stays the same.
Example: A fraud detection model trained on 2019 data. In 2020, fraud patterns changed dramatically (COVID drove online shopping, new fraud vectors emerged). The inputs (transaction features) looked similar, but the meaning of "suspicious" changed.
Detection
Monitor ground-truth feedback: if labeled outcomes become available (even delayed), compare model predictions to actuals
Track prediction accuracy over rolling time windows
Use adaptive models that explicitly model temporal dynamics
The model's output distribution changes -- even without data or concept drift.
Causes
Model version change (silent deployment of wrong model)
Feature pipeline bug (upstream change)
Infrastructure issue (GPU returning wrong results -- rare but real)
What Do You Think?
Your production model's prediction confidence has been steadily declining over 2 weeks, even though prediction accuracy has not changed yet. What is happening and what should you do?
Declining confidence is often the leading indicator of drift. The model is encountering inputs that are further from its training distribution, so it becomes less certain. Accuracy may still look fine if the model is "correctly uncertain," but eventually it will start making wrong predictions. This is your early warning system -- investigate drift now, before accuracy drops.
Try it: ML Monitoring DashboardInteractive
Simulate data drift and latency spikes to see how monitoring dashboards detect model degradation in real time.
OpenLLMetry (Traceloop) — Vendor-neutral OpenTelemetry semantic conventions for LLMs. Emit OTel spans, ship to any backend (Datadog, Honeycomb, Grafana Tempo).
Cloudflare AI Gateway. Edge-deployed observability + caching + rate limiting for any LLM provider.
Braintrust / Patronus / Galileo. Evaluation-first platforms that also do tracing.
The convergence trend matters: production LLM observability is moving onto OpenTelemetry. If you instrument with OTel semantic conventions for GenAI, you can swap backends without rewriting code. Lock-in is a real risk with closed proprietary trace formats.
ML systems fail silently. Unlike traditional software that crashes with errors, a degrading model continues serving predictions with increasing confidence in wrong answers, making monitoring essential
Monitor for data drift, concept drift, and model degradation. Data drift means input distributions changed, concept drift means the relationship between inputs and outputs changed, and both cause model quality to silently erode
LLM tracing tools debug complex chains. Langfuse, Arize, and similar tools trace token flow through multi-step agent pipelines, making it possible to identify which step in a chain caused a bad output
Good alerting distinguishes signal from noise. Alert on actionable metrics (prediction quality, feature distribution shifts, latency spikes) rather than noisy signals, and build runbooks for each alert type
With monitoring in place, your models stay healthy in production. Next up: Cost Architecture -- how to optimize GPU selection, token usage, and multi-model routing to build AI systems that are both powerful and economically viable.