Incident Response & Disaster Recovery
At 3am your model accuracy collapses, your agent burns $50K in an hour-long runaway loop, or a prompt-injection attack drains your token budget. With runbooks, you fix it in 15 minutes. Without them, it's a four-hour blameful heroic and a public post-mortem. Here's how Anthropic, OpenAI, and Stripe handle AI incidents.
Learning Objectives
After this lesson, you will be able to:
- Build runbooks for the most common ML/LLM production incidents — model degradation, cost spikes, prompt injection, downstream service outages
- Design rollback strategies that can revert to a known-good model version within 60 seconds when production accuracy collapses
- Apply on-call rotations and incident-response playbooks adapted for ML-specific failure modes (silent quality regressions, drift, hallucinations)
- Conduct post-mortems that turn incidents into systemic improvements rather than blame
Prerequisites
Before You Start
#SLO and SLI for ML Services
Site Reliability Engineering brought a vocabulary that maps cleanly onto ML serving — once you adapt the metrics. The three terms:
- SLI (Service Level Indicator): a measurable property of the service. For ML: p95 latency, success rate, prediction quality vs. baseline.
- SLO (Service Level Objective): a target for an SLI. "95% of inference requests respond in under 300ms over a 28-day window."
- SLA (Service Level Agreement): contractual SLO with consequences if missed. Usually customer-facing.
A typical ML serving SLO bundle in 2026 looks like:
| SLI | Target SLO |
|---|---|
| Inference p95 latency | < 400 ms |
| Inference success rate | > 99.5% |
| Eval-set accuracy vs. baseline | within -2% |
| Cost per 1,000 inferences | < $X |
| Time to rollback after incident | < 5 minutes |
The accuracy SLO is the ML-specific addition — it's why ML SRE differs from regular SRE. A web service that returns 200 OK with the wrong answer would be considered "succeeding" by traditional SLOs; an ML service has to layer a quality SLI on top, typically driven by a continuous eval pipeline that re-scores a holdout set against the live model and pages if accuracy falls below threshold.
The error budget concept also adapts: you have a budget of "wrong answers" per quarter, separate from the budget of HTTP 5xx errors. Burning through the accuracy budget triggers an investigation just as a 5xx-rate burn would. This is what "ML observability" platforms (Arize, Fiddler, WhyLabs) productize.
What Do You Think?
A team sets a single SLO for their LLM chatbot: 'p95 latency under 2 seconds.' Six months later, latency is good, but the bot is hallucinating product features that don't exist. What was the SLO missing?
#ML-Specific Incident Classes
| Incident | Detection | Response |
|---|---|---|
| Sudden accuracy drop | Eval pipeline regression alert | Roll back model to previous version |
| Silent quality regression | Drift detector + human-eval sample | Investigate: data drift / model drift / upstream change |
| Cost spike | Token / GPU-hour cost dashboard alert | Rate limit; investigate runaway loop or attack |
| Prompt injection / jailbreak | Output classifier + abuse detection | Block offending prompts; tighten input filter |
| Downstream tool failure | Tool-call success-rate alert | Failover to backup tool; notify users |
| GPU / infra outage | Cloud provider status + heartbeat | Failover to multi-region; degraded mode |
| Model server OOM | Memory alerts + OOM kills | Scale up memory; reduce batch size; investigate leak |
| Embedding drift | Vector similarity to baseline | Re-embed corpus; investigate model change |
#The 5-Minute Triage Protocol
For ANY ML incident:
1. Identify (1 min): which service? which model? when did it start?
2. Bound (1 min): how many users affected? how severe? trending?
3. Mitigate (3 min): roll back / disable / degrade gracefully
4. Communicate: status page update, stakeholder notification
5. Investigate: only AFTER mitigation
The priority is stop the bleeding, not understand the cause. Investigation comes after.
#Rollback Patterns
| Pattern | Implementation | Recovery Time |
|---|---|---|
| Model registry version revert | MLflow registry: re-promote previous version | < 60s |
| Blue-green deployment | Two identical environments; switch traffic | < 30s |
| Canary rollback | Progressive deploy hit issue at 5%; roll back before 50% | Automatic |
| Feature flag | Flip a flag to disable new model code | < 10s |
| Shadow mode | New model only logs predictions; safe to abandon | Instant |
Production rule: every model deploy must have a rollback plan documented BEFORE deployment. Run rollback drills quarterly.
#Disaster Recovery Patterns
For larger-scale failures (region outage, major model corruption):
1. Multi-region deployments: replicate model serving to 2+ regions; auto-failover via DNS or load balancer.
2. Backup model versions: keep last 5 known-good model versions warm in S3/GCS for instant restore.
3. Degraded mode: when primary fails, fall back to a simpler model (e.g., rules-based, or a smaller LLM) rather than full outage.
4. Cached responses: serve last-known-good response from cache during brief outages.
5. Graceful degradation: feature flags that disable risky features (autonomous agent, deep research) while keeping core functionality alive.
#ML Incident Playbooks: The Five Most Common Classes
A runbook for each of the top-five incident classes, in compressed form. Each follows the 5-minute triage protocol but adds class-specific diagnostics.
#1. Model Drift Incident
Signal: input feature distributions (PSI/KL) diverge from the training distribution, OR a continuous-eval pipeline detects a quality drop.
text
Triage:
- Quantify: which features drifted? By how much (PSI > 0.2 is significant)?
- Bound: is the quality drop user-visible? Compare to error budget.
- Mitigate: depends on severity
* mild + recoverable: deploy a hotfix data filter, plan retrain
* severe: rollback to previous model version
Investigation:
- Upstream change: did the data producer release new code? Check change log.
- Distribution shift: real-world change (seasonality, viral event, regulatory)?
- Data pipeline bug: is the schema still as expected?
Resolution:
- Retrain on the shifted distribution OR
- Add drift-resilient features OR
- Move to online/continual learning
#2. Prediction Latency Spike
Signal: p95/p99 latency jumps above SLO. Often correlates with traffic spike or cold pod.
text
Triage:
- Cold start vs steady state: check GPU utilization and pod-age dashboards
- Sudden vs gradual: sudden = infra event, gradual = leak or accumulating queue
- Mitigate:
* cold-start storm: scale up replicas, force warm pool
* queue backup: drop the oldest requests; degrade to smaller model
* memory leak: rolling restart
Investigation:
- Recent deploy: did latency regress at exact deploy time?
- GPU thermal throttling: check DCGM metrics
- KV cache OOM (LLM-specific): is max_seq_len too high for available HBM?
Resolution:
- Right-size the pod or pre-warm pool
- Cap max sequence length / batch
- Add a slow-query timeout to shed load gracefully
#3. Data Pipeline Failure
Signal: feature freshness alert, or downstream model gets stale features.
text
Triage:
- Identify the failing DAG/task
- Bound: how many feature groups affected? How long stale?
- Mitigate:
* short outage: serve last-known-good features from cache; alert
* long outage: failover model to a feature-light variant
Investigation:
- Upstream producer outage: check their status page
- Schema change: validate against contract; check DLQ for rejected records
- Resource exhaustion: cluster autoscaling, quota limits
Resolution:
- Restart the failed task; retry from checkpoint
- Backfill missing windows
- Add the failure mode to runbook for next time
#4. Hallucination Outbreak
Signal: spike in user reports of incorrect outputs, or a fact-check classifier's positive rate exceeds threshold.
text
Triage:
- Sample 10-20 hallucinated outputs; identify pattern (factual? structural? persona drift?)
- Bound: which queries trigger it? Specific topics? Specific users?
- Mitigate:
* jailbreak-induced: tighten input filter
* prompt-template regression: revert template
* model regression: rollback to previous model
* RAG retrieval gap: increase recall, broaden corpus
Investigation:
- Recent prompt template change?
- Recent model swap (e.g., went from Claude-3 to a faster Haiku variant)?
- Coverage gap: are users asking about topics absent from the corpus?
Resolution:
- Restore last-good prompt template
- Add eval cases that catch this pattern
- For RAG: expand corpus or improve retrieval recall
#5. Cost Spike / Runaway Agent Loop
Signal: hourly spend breaches budget threshold; per-customer token usage anomaly.
text
Triage:
- Identify: which model, which customer, which endpoint?
- Pattern: malicious DDOS, runaway agent, legitimate high usage?
- Mitigate:
* suspected attack: block IP/user immediately
* runaway loop: kill all jobs for that customer; investigate prompt
* legitimate burst: apply rate limit; communicate
Investigation:
- Agent stuck in tool-thrashing loop? Check trace for repeated tool calls.
- Missing rate limit on a new endpoint?
- Prompt injection that exhausts token budget intentionally?
Resolution:
- Add per-customer and per-endpoint rate limits + hard daily cost caps
- Add a circuit breaker on agent loops (> 20 iterations -> halt)
- Add a max-cost-per-conversation guard
#Post-Mortem Template
A blameless post-mortem template that maps to SRE practice with ML-specific fields:
markdown
# Incident: [short name]
## Summary
- **Date**: 2026-05-12
- **Duration**: 47 minutes (T+0 detection at 14:32 UTC, mitigated 15:19)
- **Severity**: P1
- **User impact**: ~12,500 users received degraded predictions; estimated revenue loss $X
## Timeline (UTC)
| Time | Event |
|---|---|
| 14:32 | Continuous eval pipeline alert: val_acc dropped 0.92 -> 0.78 |
| 14:34 | On-call acknowledged in PagerDuty |
| 14:36 | #incidents-ml notified |
| 14:40 | Bounded scope: only checkout-recommendation model affected |
| 14:48 | Decision: rollback to v3.2.1 (previous Production version) |
| 14:52 | Rollback executed via MLflow registry |
| 15:06 | Verified val_acc returned to 0.91 |
| 15:19 | Incident closed; status page updated |
## Root cause (5-whys)
1. Why did accuracy drop? -> Model v3.2.2 was trained on corrupted data
2. Why was the data corrupted? -> Upstream events service deployed a schema change that renamed `purchased_at` -> `bought_at`
3. Why did training succeed despite the rename? -> Our pipeline silently null-filled missing columns
4. Why did the pipeline null-fill? -> No data contract validation at ingest
5. Why no contract? -> The team hadn't adopted contracts yet
Root cause: missing data contract at the events-service boundary.
## Action items
- [ ] **[Owner: data-platform]** Add Pydantic contract for events ingest by 2026-05-20
- [ ] **[Owner: ml-platform]** Eval pipeline should refuse to promote models with > 5% feature null-rate change by 2026-05-26
- [ ] **[Owner: incident-response]** Update model-accuracy-regression runbook with 'check for upstream schema change' step by 2026-05-15
## What went well
- Detection: continuous eval caught it within 5 minutes of deploy
- Mitigation: rollback completed in 4 minutes
- Communication: stakeholders informed before any social-media noise
## What didn't
- Data contract was a known gap, not yet implemented
- The training pipeline silently null-filled instead of failing loudly
## What we'll change
- All upstream boundaries get Pydantic contracts by end of Q2 2026
- Pipeline never silently null-fills; missing-column = hard failure with alert
The discipline matters: blameless framing (no naming the person who deployed the rename), 5-whys to systemic cause (not just "the model broke"), action items with owners and dates (not vague "we'll do better"), explicit "what went well" so good responses get reinforced.
#On-Call Rotation Patterns for ML Teams
ML on-call differs from web-service on-call in three ways:
- Alerts can be slow. Model quality regressions take minutes-to-hours to surface (eval pipelines run on cadence). On-call windows must accept longer detection windows for quality issues than for latency issues.
- Mitigation needs ML context. Rolling back a model registry version, checking drift dashboards, or interpreting prediction-distribution shifts requires domain familiarity. Rotation pools must include people who know the models.
- The on-call is often a small team. ML platform teams are typically 5-15 people, not 50. Standard 24/7 follow-the-sun rotation doesn't scale. Common patterns:
| Pattern | When |
|---|---|
| Single rotation, business-hours-only | Pre-revenue products; non-critical ML |
| Single rotation, 24/7 | Critical ML systems, small team |
| Primary + secondary, weekly | Mid-size team (10-15 ML engineers) |
| Follow-the-sun across regions | Large org with EU + US + APAC engineers |
| Dedicated "ML SRE" subteam | Hyperscale ML platforms (FAANG, top AI labs) |
The escalation chain inside each rotation: primary on-call (acknowledge in < 5 min), secondary (page if primary doesn't respond in 10 min), tech lead (page on P0 or 30 min unresolved), engineering manager (P0 or 1 hour unresolved). Document this on the team's runbook page so nobody has to figure it out at 3am.
#Real-World ML Incidents: Three 2024-2026 Case Studies
Case studies make the abstract concrete. Three high-visibility incidents that motivated industry-wide practice changes:
#Air Canada chatbot: bereavement-fare hallucination (Feb 2024)
An Air Canada customer asked the airline's website chatbot about bereavement fares. The bot confidently described a refund policy that did not exist. The customer relied on it, booked the flight, and was denied the refund. The Civil Resolution Tribunal of British Columbia ordered Air Canada to pay damages and ruled that the chatbot's statements were the airline's responsibility.
Lessons
- Output classifiers and fact-check layers for any LLM that makes policy-binding statements
- Continuous eval against a curated FAQ test set; pages if hallucination rate exceeds threshold
- Legal/compliance review of LLM outputs is now standard for customer-facing deployments
- "The model said it" is not a defense; you own your bot's outputs
#Klarna AI rollback: agentic customer service (Spring 2025)
Klarna had publicly celebrated replacing ~700 customer-service agents with an AI assistant. After ~12 months in production, NPS dropped, complex tickets escalated to humans without context, and the company partially rolled back — re-hiring human agents for tickets above a complexity threshold.
Lessons
- Replacing humans 1:1 with a single LLM is usually wrong; the right model is human + AI co-pilot, with clear escalation criteria
- Quality SLOs (CSAT, NPS, escalation rate) are first-class production signals, not just latency
- A 12-month "we're saving X" press release ages badly if you haven't built the rollback path into the system from day one
#A 2025 token-cost runaway incident (anonymized)
A team deployed a coding agent that, under specific prompts, entered a tool-thrashing loop calling its own browser tool with progressively longer URLs. The loop ran for ~6 hours overnight before the daily cost alert fired. Total bill: mid-six figures.
Lessons
- Hard per-conversation token caps (
max_tokens_total=200000, kill at limit) at the application layer, not just per-request - Per-tenant hourly spend caps with circuit breakers (auto-halt new requests when budget hit)
- Continuous integration tests for the agent-loop layer: known prompts that historically caused thrashing should be in the eval suite
- On-call rotations need cost-alert routing as a first-class incident class, not a billing email
Quick check
An on-call engineer is paged at 3am: 'agent loop runaway, $200/min burn rate'. Which action comes first?
#Post-Mortem Discipline
After every incident, run a blameless post-mortem:
- Timeline (with timestamps): what happened when
- Root cause (use 5-whys): not just "the model broke" but "why was it allowed to be deployed without eval gate?"
- Action items (ownership + due date): systemic fixes, not just patches
- Process improvements: what monitoring was missing? what playbook needs to be written?
The goal: every incident makes the system more resilient. Same incident type happening twice means the post-mortem failed.
#Hands-On
pythonplayground.py · Pyodide
Tests · Verify the runbook covers all 5 steps clearly. Verify escalation path is documented. Pretend you're on-call at 2am — can you follow this without prior context?
#Key Takeaways
- Mitigate before you investigate. Every incident protocol prioritizes stopping the bleeding
- Runbooks for top 10 incident classes. Drift, cost spike, hallucination, RAG failure, prompt injection, infra outage, OOM, embedding drift, agent runaway, downstream API outage
- Rollback in < 60 seconds. Model registry promote-previous, blue-green deploy, feature flags
- Chaos engineering for ML. Quarterly drills; first time your rollback is tested should NOT be in production
- Blameless post-mortems. Every incident becomes systemic improvement; same incident twice = post-mortem failed
#Quick Check
Quick Check1 / 2
Why is 'mitigate before investigate' the standard incident-response order?
What Do You Think?
Your team has runbooks for top-10 incident classes but has never run a chaos drill. A real P0 hits — the runbook for the affected class is 18 months old. Most defensible response?
With incident response in place, the production stack is complete. Next: the capstone that synthesizes everything into a production-ready ML/LLM system.