System Design Case Studies

ML system design interviews aren't about training models — they're about candidate generation, ranking, online vs offline features, latency budgets, and the trade-offs that determine whether your design survives 10x traffic. Here's how to think like a staff engineer at Netflix, OpenAI, or Stripe.

Learning Objectives

After this lesson, you will be able to:

Break down a big AI system design problem into clear steps: requirements, architecture, components, and trade-offs
Design the full ML architecture for a recommendation system, a ChatGPT-scale chat app, and an autonomous coding agent
Weigh real engineering trade-offs: speed vs volume, accuracy vs cost, complexity vs maintainability

Prerequisites

Before You Start

Review

Why

Why This Matters

How Netflix, ChatGPT, and GitHub Copilot are actually built under the hood -- Netflix's recommendation system saves $1 billion per year, not because of a single fancy model, but because of the system around it: data pipelines ingesting viewing behavior, feature stores computing user preferences in milliseconds, A/B testing hundreds of model variants simultaneously, and monitoring to catch when recommendations go stale; the model is 5% of the magic -- system design is the other 95%
This is the skill that gets you hired at top AI companies -- Google, OpenAI, and Netflix do not ask you to train a model in interviews; they ask you to design the full system: "Design TikTok's recommendation engine" means sketching the data pipeline, candidate generation, ranking model, serving infrastructure, and monitoring -- and explaining the trade-offs at each layer
ChatGPT is not just a good model -- it is an engineering marvel -- serving 100 million users means thousands of GPUs with load balancing, PagedAttention for memory efficiency, continuous batching to keep GPUs saturated, multi-model routing for cost optimization, and safety filters on every response; understanding how these pieces fit together is what this lesson teaches
Build this --> Create a system design sketch for a "Personalized Playlist Generator" like Spotify: diagram the data pipeline (listening history ingestion), feature engineering (genre preferences, skip patterns), model (collaborative filtering + content embeddings), serving (real-time recommendations via API), and monitoring (skip rate, engagement tracking) -- present it like you would in a system design interview

Decision	Option A	Option B	Netflix's Likely Choice
Recompute frequency	Real-time scoring	Batch pre-computation	Hybrid: batch candidates, real-time ranking
Model complexity	Simple (logistic regression)	Complex (deep network)	Deep network (worth the complexity at scale)
Cold start	Show popular items	Ask preferences	Both: popular items + onboarding quiz
Latency vs accuracy	Simpler model, faster	Better model, slower	Pre-compute to get both

Decision	Option A	Option B	Recommended
Model size	Smaller (cheaper, faster)	Larger (smarter)	Multi-model routing by difficulty
Context handling	Truncate old messages	Summarize old messages	Summarization (preserves context)
Safety filtering	Pre-generation (block input)	Post-generation (filter output)	Both (defense in depth)
Streaming	Full response at once	Token-by-token	Streaming (better UX)
Tool execution	Parallel (fast)	Sequential (safe)	Parallel with dependency analysis

Decision	Option A	Option B	Recommended
Context strategy	Read entire codebase	Selective retrieval	Selective (cost, context limit)
Planning depth	Plan all steps upfront	Plan one step at a time	Hybrid: high-level plan, detailed per-step
Error handling	Give up after N failures	Always retry with feedback	Retry with escalation (ask human after N)
Model choice	One model for everything	Specialized models per task	One frontier model (simplicity)
Verification	Trust model's assessment	Run tests automatically	Always run tests (trust but verify)

System Design Case Studies

Your Reflection

Continue Learning

Discussion

#How to Approach AI System Design

#Case Study 1: Design Netflix-Style Recommendations

#Step 1: Requirements

Functional

Non-functional

Business metrics

#Step 2: Architecture

#Step 3: ML Components

Candidate generation (broad retrieval)

Ranking model (precise scoring)

Business rules layer

#Step 4: Trade-offs

#Case Study 2: Design a ChatGPT Clone

#Step 1: Requirements

Functional

Non-functional

Safety requirements

#Step 2: Architecture

#Step 3: ML Components

The LLM cluster

Session management

Tool use system

Safety system

#Step 4: Trade-offs

#Case Study 3: Design an Autonomous Coding Agent

#Step 1: Requirements

Functional

Non-functional

Safety requirements

#Step 2: Architecture

#Step 3: ML Components

The reasoning engine

Codebase understanding

Planning and execution

Tool definitions

#Step 4: Trade-offs

#The System Design Template

#Key Takeaways

#Quick Check

Deep Dive

Related Lessons