Model Serving & Inference

#From Training to Serving: A Different World

You have made it to one of the most practical lessons in ML engineering. If you have ever wondered how AI products actually work in real life (not just in notebooks), this is where it all clicks. Let's dive in.

#Serving Protocols: REST vs gRPC vs Streaming

#REST (HTTP/JSON)

The most common serving pattern. Simple, universal, works with any client:

Try it! Open a terminal and use curl to send a JSON request to a free public prediction API (or a local Flask server running a simple model). Watch the request-response cycle in action. Try sending 5 requests in rapid succession and see how response time changes under load — this is exactly the problem that batching solves.

#gRPC (HTTP/2 + Protocol Buffers)

Binary protocol with schema enforcement. 2-10x faster than REST for serialization:

#Server-Sent Events / WebSocket Streaming

Essential for LLMs that generate tokens one at a time:

Select a serving protocol above and step through the inference request lifecycle. Compare REST, gRPC, and streaming side by side. Adjust the concurrency slider to see how throughput and latency change under different loads.

#Traditional Model Serving

For non-LLM models (XGBoost, PyTorch classifiers, embedding models):

Serving frameworks

• TorchServe: PyTorch models. Handles batching, model versioning, multi-model serving.
• TensorFlow Serving: TF models. gRPC + REST. Used extensively at Google.
• Triton Inference Server (NVIDIA): Multi-framework (PyTorch, TF, ONNX, TensorRT). GPU optimized.
• BentoML: Framework-agnostic. Python-native. Easy to package and deploy.
• ONNX Runtime: Convert any framework to ONNX format for optimized cross-platform inference.

Key optimizations

• Dynamic batching: Collect individual requests into batches for GPU efficiency. Wait up to N ms or until batch is full.
• Model compilation: Convert PyTorch models to TorchScript or TensorRT for faster inference.
• Caching: Cache frequent predictions (embedding lookups, common queries).
• Multi-model serving: Load multiple models on one GPU, route requests based on model ID.

#LLM Serving: A Different Beast

Large language models have unique serving challenges:

1. Memory bound: A 70B parameter model needs ~140GB in FP16 just for weights (plus KV cache)
2. Sequential generation: Each token depends on all previous tokens — hard to parallelize
3. Variable-length outputs: Cannot predict how long generation will take
4. KV cache management: The key-value cache grows linearly with sequence length

#vLLM: PagedAttention

#Continuous Batching

Traditional batching waits for a batch to complete before starting new requests. Continuous batching (iteration-level scheduling) is smarter:

When request C finishes, request D immediately takes its slot in the batch. No request waits for the slowest request to finish. This keeps the GPU saturated at all times.

#Speculative Decoding

Use a small "draft" model to generate N candidate tokens quickly, then verify all N in parallel with the large model:

Draft model (fast): generates 5 tokens speculatively
Large model (slow): verifies all 5 in one forward pass
If first 3 are correct: accept 3 tokens in one step (3x speedup!)
If all 5 are correct: accept all 5 (5x speedup!)

This exploits the fact that many tokens are predictable (common phrases, code patterns, structured output). The speedup is proportional to the acceptance rate.

#LLM Serving Systems Comparison

#Inference Disaggregation (2024-2026)

Production systems implementing this pattern:

• Mooncake (Moonshot AI / Kimi) — separates prefill and decode clusters, shares a global KV-cache pool over RDMA.
• DistServe (Peking University, 2024) — disaggregated serving with per-stage goodput optimization, 4-7x more requests per GPU at the same SLO.
• Splitwise (Microsoft) — phase splitting on different hardware classes (compute-heavy prefill on A100, memory-heavy decode on cheaper H100-MIG slices).
• SGLang v0.3+. Adds disaggregated mode plus chunked prefill that interleaves long-prompt processing with active decode batches.

The win is that prefill and decode have wildly different optimal batch sizes and KV-cache footprints. Co-locating them caps throughput at the lower of the two.

#Multi-LoRA and Multi-Tenant GPU Serving

• vLLM --enable-lora — supports hundreds of concurrent LoRA adapters with negligible overhead.
• S-LoRA (Stanford, 2023) — unified paging for adapters and KV cache; 4x more adapters per GPU than naive swapping.
• Punica. Kernel-level fusion of multiple LoRAs in a single CUDA call (SGMV — Segmented Gather Matrix-Vector).
• TGI multi-LoRA. Same pattern for HuggingFace stacks.

This pattern is what makes per-customer fine-tuning economically feasible: 500 enterprise customers can each have a custom adapter served from one shared base-model fleet.

#The Inference Request Lifecycle

Every model prediction follows the same path from request to response. Understanding each step reveals where latency hides and where optimization pays off:

#Batching Strategies Deep Dive

#Inference Optimization Techniques

#Model Compilation

Convert dynamic computation graphs to static, optimized ones:

• TorchScript / torch.compile: PyTorch graph optimization, kernel fusion
• ONNX Runtime: Cross-platform optimized inference
• TensorRT: NVIDIA GPU-specific optimization. Kernel fusion, precision calibration.
• OpenVINO: Intel CPU/GPU optimization

#Quantization at Serving Time

Reduce precision for faster inference (covered in depth in the next lesson):

• FP32 -> FP16: 2x memory reduction, ~2x speedup, minimal quality loss
• FP16 -> INT8: 2x further reduction, requires calibration
• INT8 -> INT4: Another 2x, primarily for LLMs (GPTQ, AWQ)

#KV Cache Optimization

For LLMs, the KV cache is often the memory bottleneck:

• Multi-Query Attention (MQA): Share K/V heads across attention heads (Llama 2 uses GQA, a middle ground)
• Sliding window attention: Only cache the last N tokens (Mistral)
• KV cache quantization: Quantize cached keys/values to INT8 or FP8
• Prefix caching: Hash and reuse the KV cache for common prompt prefixes (system prompts, RAG templates) — vLLM and SGLang ship this out of the box; cache hit rates of 30-60% are typical in chat workloads.
• Cross-request offload: Page cold KV-cache blocks to host memory or NVMe (LMCache, Mooncake) so popular conversations stay resident.

#Sizing Your Serving Fleet: Run the Math

Before signing off on hardware, work the numbers. The cell below estimates KV-cache memory per request, total memory headroom, and the maximum concurrent batch size for a vLLM-style replica — the exact math your platform team needs in their capacity plan.

#Key Takeaways

• Training and serving are fundamentally different workloads. Training optimizes for throughput (process data as fast as possible), while serving optimizes for latency (respond to each request in milliseconds)
• LLM serving uses specialized techniques for efficiency. PagedAttention manages KV cache memory dynamically, continuous batching maximizes GPU utilization, and speculative decoding predicts future tokens to reduce latency
• Choose your serving pattern based on requirements. REST APIs for simple request-response, gRPC for low-latency microservices, and streaming (SSE/WebSocket) for LLM token-by-token generation
• Cost, latency, throughput, and availability form a quadrilateral tradeoff. You cannot optimize all four simultaneously; production systems must make explicit choices about which dimensions matter most for their use case

#Quick Check

#Exit Ticket

Before moving on, you should be able to answer these in your own words. If any feel shaky, scroll back.