Incident Response & Disaster Recovery

#SLO and SLI for ML Services

Site Reliability Engineering brought a vocabulary that maps cleanly onto ML serving — once you adapt the metrics. The three terms:

• SLI (Service Level Indicator): a measurable property of the service. For ML: p95 latency, success rate, prediction quality vs. baseline.
• SLO (Service Level Objective): a target for an SLI. "95% of inference requests respond in under 300ms over a 28-day window."
• SLA (Service Level Agreement): contractual SLO with consequences if missed. Usually customer-facing.

A typical ML serving SLO bundle in 2026 looks like:

The error budget concept also adapts: you have a budget of "wrong answers" per quarter, separate from the budget of HTTP 5xx errors. Burning through the accuracy budget triggers an investigation just as a 5xx-rate burn would. This is what "ML observability" platforms (Arize, Fiddler, WhyLabs) productize.

#ML-Specific Incident Classes

#The 5-Minute Triage Protocol

For ANY ML incident:

1. Identify (1 min): which service? which model? when did it start?
2. Bound (1 min): how many users affected? how severe? trending?
3. Mitigate (3 min): roll back / disable / degrade gracefully
4. Communicate: status page update, stakeholder notification
5. Investigate: only AFTER mitigation

#Rollback Patterns

#Disaster Recovery Patterns

For larger-scale failures (region outage, major model corruption):

#ML Incident Playbooks: The Five Most Common Classes

A runbook for each of the top-five incident classes, in compressed form. Each follows the 5-minute triage protocol but adds class-specific diagnostics.

#1. Model Drift Incident

Triage:
  - Quantify: which features drifted? By how much (PSI > 0.2 is significant)?
  - Bound: is the quality drop user-visible? Compare to error budget.
  - Mitigate: depends on severity
      * mild + recoverable: deploy a hotfix data filter, plan retrain
      * severe: rollback to previous model version
Investigation:
  - Upstream change: did the data producer release new code? Check change log.
  - Distribution shift: real-world change (seasonality, viral event, regulatory)?
  - Data pipeline bug: is the schema still as expected?
Resolution:
  - Retrain on the shifted distribution OR
  - Add drift-resilient features OR
  - Move to online/continual learning

#2. Prediction Latency Spike

Triage:
  - Cold start vs steady state: check GPU utilization and pod-age dashboards
  - Sudden vs gradual: sudden = infra event, gradual = leak or accumulating queue
  - Mitigate:
      * cold-start storm: scale up replicas, force warm pool
      * queue backup: drop the oldest requests; degrade to smaller model
      * memory leak: rolling restart
Investigation:
  - Recent deploy: did latency regress at exact deploy time?
  - GPU thermal throttling: check DCGM metrics
  - KV cache OOM (LLM-specific): is max_seq_len too high for available HBM?
Resolution:
  - Right-size the pod or pre-warm pool
  - Cap max sequence length / batch
  - Add a slow-query timeout to shed load gracefully

#3. Data Pipeline Failure

Triage:
  - Identify the failing DAG/task
  - Bound: how many feature groups affected? How long stale?
  - Mitigate:
      * short outage: serve last-known-good features from cache; alert
      * long outage: failover model to a feature-light variant
Investigation:
  - Upstream producer outage: check their status page
  - Schema change: validate against contract; check DLQ for rejected records
  - Resource exhaustion: cluster autoscaling, quota limits
Resolution:
  - Restart the failed task; retry from checkpoint
  - Backfill missing windows
  - Add the failure mode to runbook for next time

#4. Hallucination Outbreak

Triage:
  - Sample 10-20 hallucinated outputs; identify pattern (factual? structural? persona drift?)
  - Bound: which queries trigger it? Specific topics? Specific users?
  - Mitigate:
      * jailbreak-induced: tighten input filter
      * prompt-template regression: revert template
      * model regression: rollback to previous model
      * RAG retrieval gap: increase recall, broaden corpus
Investigation:
  - Recent prompt template change?
  - Recent model swap (e.g., went from Claude-3 to a faster Haiku variant)?
  - Coverage gap: are users asking about topics absent from the corpus?
Resolution:
  - Restore last-good prompt template
  - Add eval cases that catch this pattern
  - For RAG: expand corpus or improve retrieval recall

#5. Cost Spike / Runaway Agent Loop

Triage:
  - Identify: which model, which customer, which endpoint?
  - Pattern: malicious DDOS, runaway agent, legitimate high usage?
  - Mitigate:
      * suspected attack: block IP/user immediately
      * runaway loop: kill all jobs for that customer; investigate prompt
      * legitimate burst: apply rate limit; communicate
Investigation:
  - Agent stuck in tool-thrashing loop? Check trace for repeated tool calls.
  - Missing rate limit on a new endpoint?
  - Prompt injection that exhausts token budget intentionally?
Resolution:
  - Add per-customer and per-endpoint rate limits + hard daily cost caps
  - Add a circuit breaker on agent loops (> 20 iterations -> halt)
  - Add a max-cost-per-conversation guard

#Post-Mortem Template

A blameless post-mortem template that maps to SRE practice with ML-specific fields:

# Incident: [short name]

## Summary
- **Date**: 2026-05-12
- **Duration**: 47 minutes (T+0 detection at 14:32 UTC, mitigated 15:19)
- **Severity**: P1
- **User impact**: ~12,500 users received degraded predictions; estimated revenue loss $X

## Timeline (UTC)
| Time | Event |
|---|---|
| 14:32 | Continuous eval pipeline alert: val_acc dropped 0.92 -> 0.78 |
| 14:34 | On-call acknowledged in PagerDuty |
| 14:36 | #incidents-ml notified |
| 14:40 | Bounded scope: only checkout-recommendation model affected |
| 14:48 | Decision: rollback to v3.2.1 (previous Production version) |
| 14:52 | Rollback executed via MLflow registry |
| 15:06 | Verified val_acc returned to 0.91 |
| 15:19 | Incident closed; status page updated |

## Root cause (5-whys)
1. Why did accuracy drop? -> Model v3.2.2 was trained on corrupted data
2. Why was the data corrupted? -> Upstream events service deployed a schema change that renamed `purchased_at` -> `bought_at`
3. Why did training succeed despite the rename? -> Our pipeline silently null-filled missing columns
4. Why did the pipeline null-fill? -> No data contract validation at ingest
5. Why no contract? -> The team hadn't adopted contracts yet

Root cause: missing data contract at the events-service boundary.

## Action items
- [ ] **[Owner: data-platform]** Add Pydantic contract for events ingest by 2026-05-20
- [ ] **[Owner: ml-platform]** Eval pipeline should refuse to promote models with > 5% feature null-rate change by 2026-05-26
- [ ] **[Owner: incident-response]** Update model-accuracy-regression runbook with 'check for upstream schema change' step by 2026-05-15

## What went well
- Detection: continuous eval caught it within 5 minutes of deploy
- Mitigation: rollback completed in 4 minutes
- Communication: stakeholders informed before any social-media noise

## What didn't
- Data contract was a known gap, not yet implemented
- The training pipeline silently null-filled instead of failing loudly

## What we'll change
- All upstream boundaries get Pydantic contracts by end of Q2 2026
- Pipeline never silently null-fills; missing-column = hard failure with alert

The discipline matters: blameless framing (no naming the person who deployed the rename), 5-whys to systemic cause (not just "the model broke"), action items with owners and dates (not vague "we'll do better"), explicit "what went well" so good responses get reinforced.

#On-Call Rotation Patterns for ML Teams

ML on-call differs from web-service on-call in three ways:

1. Alerts can be slow. Model quality regressions take minutes-to-hours to surface (eval pipelines run on cadence). On-call windows must accept longer detection windows for quality issues than for latency issues.
2. Mitigation needs ML context. Rolling back a model registry version, checking drift dashboards, or interpreting prediction-distribution shifts requires domain familiarity. Rotation pools must include people who know the models.
3. The on-call is often a small team. ML platform teams are typically 5-15 people, not 50. Standard 24/7 follow-the-sun rotation doesn't scale. Common patterns:

The escalation chain inside each rotation: primary on-call (acknowledge in < 5 min), secondary (page if primary doesn't respond in 10 min), tech lead (page on P0 or 30 min unresolved), engineering manager (P0 or 1 hour unresolved). Document this on the team's runbook page so nobody has to figure it out at 3am.

#Real-World ML Incidents: Three 2024-2026 Case Studies

Case studies make the abstract concrete. Three high-visibility incidents that motivated industry-wide practice changes:

#Air Canada chatbot: bereavement-fare hallucination (Feb 2024)

An Air Canada customer asked the airline's website chatbot about bereavement fares. The bot confidently described a refund policy that did not exist. The customer relied on it, booked the flight, and was denied the refund. The Civil Resolution Tribunal of British Columbia ordered Air Canada to pay damages and ruled that the chatbot's statements were the airline's responsibility.

Lessons

• Output classifiers and fact-check layers for any LLM that makes policy-binding statements
• Continuous eval against a curated FAQ test set; pages if hallucination rate exceeds threshold
• Legal/compliance review of LLM outputs is now standard for customer-facing deployments
• "The model said it" is not a defense; you own your bot's outputs

#Klarna AI rollback: agentic customer service (Spring 2025)

Klarna had publicly celebrated replacing ~700 customer-service agents with an AI assistant. After ~12 months in production, NPS dropped, complex tickets escalated to humans without context, and the company partially rolled back — re-hiring human agents for tickets above a complexity threshold.

Lessons

• Replacing humans 1:1 with a single LLM is usually wrong; the right model is human + AI co-pilot, with clear escalation criteria
• Quality SLOs (CSAT, NPS, escalation rate) are first-class production signals — not just latency
• A 12-month "we're saving X" press release ages badly if you haven't built the rollback path into the system from day one

#A 2025 token-cost runaway incident (anonymized)

A team deployed a coding agent that, under specific prompts, entered a tool-thrashing loop calling its own browser tool with progressively longer URLs. The loop ran for ~6 hours overnight before the daily cost alert fired. Total bill: mid-six figures.

Lessons

• Hard per-conversation token caps (max_tokens_total=200000, kill at limit) at the application layer, not just per-request
• Per-tenant hourly spend caps with circuit breakers (auto-halt new requests when budget hit)
• Continuous integration tests for the agent-loop layer: known prompts that historically caused thrashing should be in the eval suite
• On-call rotations need cost-alert routing as a first-class incident class, not a billing email

#Post-Mortem Discipline

After every incident, run a blameless post-mortem:

1. Timeline (with timestamps): what happened when
2. Root cause (use 5-whys): not just "the model broke" but "why was it allowed to be deployed without eval gate?"
3. Action items (ownership + due date): systemic fixes, not just patches
4. Process improvements: what monitoring was missing? what playbook needs to be written?

The goal: every incident makes the system more resilient. Same incident type happening twice means the post-mortem failed.

#Hands-On

#Key Takeaways

• Mitigate before you investigate. Every incident protocol prioritizes stopping the bleeding
• Runbooks for top 10 incident classes. Drift, cost spike, hallucination, RAG failure, prompt injection, infra outage, OOM, embedding drift, agent runaway, downstream API outage
• Rollback in < 60 seconds. Model registry promote-previous, blue-green deploy, feature flags
• Chaos engineering for ML. Quarterly drills; first time your rollback is tested should NOT be in production
• Blameless post-mortems. Every incident becomes systemic improvement; same incident twice = post-mortem failed

#Quick Check