RugvAI Labs

Model Serving & Inference

Training is a one-time cost. Serving is forever. ChatGPT costs OpenAI more in inference per month than the original GPT-4 training run. vLLM, TGI, Triton, and continuous batching are the difference between a $10K/month and a $1M/month bill at the same QPS — same model, same accuracy.

Learning Objectives

After this lesson, you will be able to:

Compare three ways to send predictions (REST, gRPC, and streaming) and pick the best one for your use case
Explain how LLM serving systems like vLLM keep GPUs busy using batching, memory management, and speculative decoding
Design a serving setup that balances speed, volume, cost, and reliability

Prerequisites

Before You Start

Review

Why

Why This Matters

How ChatGPT responds to 100 million users at the same time without crashing -- behind every ChatGPT response is a system of continuous batching (grouping requests together so the GPU stays busy), PagedAttention (managing memory like an operating system manages RAM), and load balancing across hundreds of GPUs; without these techniques, answering one question would block everyone else in line
Serving costs more than training -- way more -- training GPT-4 cost around $100 million (a one-time expense); but serving it to users costs OpenAI billions per year because every single response requires expensive GPU computation; the engineering that makes inference fast and cheap is actually more valuable than the engineering that trained the model in the first place
Why your AI demo crashes the moment 10 people try it at once -- running a model on your laptop for one user is easy; serving it to 100 concurrent users means requests queue up, the GPU runs out of memory, and latency spikes to 30 seconds; batching, caching, streaming, and load balancing are the techniques that bridge the gap from "cool demo" to "real product"
Build this --> Deploy a simple text classification model as a REST API endpoint, hit it with 50 concurrent requests, measure the latency and throughput, then add request batching and compare -- see exactly how much faster batching makes it

#From Training to Serving: A Different World

You have made it to one of the most practical lessons in ML engineering. If you have ever wondered how AI products actually work in real life (not just in notebooks), this is where it all clicks. Let's dive in.

Intuition

Build Your Intuition

Training and serving are fundamentally different workloads. Training is like studying for an exam -- you can take your time, process data in large batches, and use as much compute as you need. Serving is like taking the exam -- you have seconds (or milliseconds) to produce an answer, thousands of people are asking simultaneously, and every millisecond of delay costs money or users.

Think of a restaurant kitchen. Training is recipe development: experiment freely, take all day, waste ingredients. Serving is dinner service: every order must be prepared correctly, quickly, and consistently while handling 200 orders per hour. The skills overlap, but the constraints are entirely different.

This is why many ML engineers who are excellent at training models struggle with serving: it requires a completely different mindset -- systems engineering, not data science.

#Serving Protocols: REST vs gRPC vs Streaming

#REST (HTTP/JSON)

The most common serving pattern. Simple, universal, works with any client:

REST (HTTP/JSON) Flow

Pros: Universal compatibility, easy debugging (curl), human-readable, load balancer support. Cons: JSON serialization overhead, no streaming, higher latency than binary protocols. Best for: Web applications, mobile apps, low-throughput APIs, prototyping.

Try it! Open a terminal and use curl to send a JSON request to a free public prediction API (or a local Flask server running a simple model). Watch the request-response cycle in action. Try sending 5 requests in rapid succession and see how response time changes under load -- this is exactly the problem that batching solves.

#gRPC (HTTP/2 + Protocol Buffers)

Binary protocol with schema enforcement. 2-10x faster than REST for serialization:

gRPC (HTTP/2 + Protobuf) Flow

Pros: Low latency (binary serialization), streaming support, strong typing, code generation. Cons: Harder to debug, requires client library generation, less firewall-friendly. Best for: Internal microservices, high-throughput systems, latency-sensitive workloads.

#Server-Sent Events / WebSocket Streaming

Essential for LLMs that generate tokens one at a time:

Server-Sent Events Streaming Flow

Pros: User sees output as it generates (time-to-first-token matters), better perceived latency. Cons: Connection management complexity, harder to load balance, client must handle partial responses. Best for: LLM chat interfaces, real-time generation, any autoregressive model.

What Do You Think?

A user sends a prompt to a ChatGPT-style service. The full response takes 5 seconds to generate. With streaming, the first token appears in 200ms. Without streaming, the user waits 5 seconds for the full response. Which feels faster to the user?

Time-to-first-token (TTFT) is often more important than total generation time for user experience. A system that streams the first token in 200ms feels responsive, even if the full response takes 5 seconds. This is the same psychology behind progressive loading in web applications.

Try it: Compare serving patternsInteractive

Loading visualization...

Select a serving protocol above and step through the inference request lifecycle. Compare REST, gRPC, and streaming side by side. Adjust the concurrency slider to see how throughput and latency change under different loads.

#Traditional Model Serving

For non-LLM models (XGBoost, PyTorch classifiers, embedding models):

Serving frameworks

TorchServe: PyTorch models. Handles batching, model versioning, multi-model serving.
TensorFlow Serving: TF models. gRPC + REST. Used extensively at Google.
Triton Inference Server (NVIDIA): Multi-framework (PyTorch, TF, ONNX, TensorRT). GPU optimized.
BentoML: Framework-agnostic. Python-native. Easy to package and deploy.
ONNX Runtime: Convert any framework to ONNX format for optimized cross-platform inference.

Key optimizations

Dynamic batching: Collect individual requests into batches for GPU efficiency. Wait up to N ms or until batch is full.
Model compilation: Convert PyTorch models to TorchScript or TensorRT for faster inference.
Caching: Cache frequent predictions (embedding lookups, common queries).
Multi-model serving: Load multiple models on one GPU, route requests based on model ID.

#LLM Serving: A Different Beast

Large language models have unique serving challenges:

Memory bound: A 70B parameter model needs ~140GB in FP16 just for weights (plus KV cache)
Sequential generation: Each token depends on all previous tokens -- hard to parallelize
Variable-length outputs: Cannot predict how long generation will take
KV cache management: The key-value cache grows linearly with sequence length

#vLLM: PagedAttention

Step through this visualization to watch the KV cache grow token by token and see how PagedAttention reclaims the memory that naive contiguous allocation leaves stranded.

Loading visualization...

#Continuous Batching

Traditional batching waits for a batch to complete before starting new requests. Continuous batching (iteration-level scheduling) is smarter:

Continuous Batching Timeline

Step	Slot 1	Slot 2	Slot 3
Step 1	Req A token 5	Req B token 3	Req C token 8
Step 2	Req A token 6	Req B token 4	Req C DONE -> Req D token 1
Step 3	Req A token 7	Req B token 5	Req D token 2

When request C finishes, request D immediately takes its slot in the batch. No request waits for the slowest request to finish. This keeps the GPU saturated at all times.

#Speculative Decoding

Use a small "draft" model to generate N candidate tokens quickly, then verify all N in parallel with the large model:

Draft model (fast): generates 5 tokens speculatively
Large model (slow): verifies all 5 in one forward pass
If first 3 are correct: accept 3 tokens in one step (3x speedup!)
If all 5 are correct: accept all 5 (5x speedup!)

This exploits the fact that many tokens are predictable (common phrases, code patterns, structured output). The speedup is proportional to the acceptance rate.

#LLM Serving Systems Comparison

System	Key Feature	Best For
vLLM	PagedAttention, continuous batching, multi-LoRA	General LLM serving, highest throughput
TGI (Text Generation Inference)	HuggingFace integration, tensor parallelism	HuggingFace models, quick deployment
Triton + TensorRT-LLM	NVIDIA optimization, multi-GPU, in-flight batching	Maximum single-request latency optimization
Ollama	Local models, easy setup	Development, single-user desktop
llama.cpp	CPU inference, quantization (Q4/Q5/Q8)	Edge deployment, no GPU available

#Inference Disaggregation (2024-2026)

The newest production architectures split the prefill stage (processing the input prompt — compute-bound, parallel) from the decode stage (generating tokens one at a time — memory-bound, sequential). Running both on the same GPU forces one to wait for the other; separating them onto different pools lets each scale independently.

Production systems implementing this pattern:

Mooncake (Moonshot AI / Kimi) — separates prefill and decode clusters, shares a global KV-cache pool over RDMA.
DistServe (Peking University, 2024) — disaggregated serving with per-stage goodput optimization, 4-7x more requests per GPU at the same SLO.
Splitwise (Microsoft) — phase splitting on different hardware classes (compute-heavy prefill on A100, memory-heavy decode on cheaper H100-MIG slices).
SGLang v0.3+ — adds disaggregated mode plus chunked prefill that interleaves long-prompt processing with active decode batches.

The win is that prefill and decode have wildly different optimal batch sizes and KV-cache footprints. Co-locating them caps throughput at the lower of the two.

What Do You Think?

A vLLM replica receives a mix of short chat turns (50-token prompts) and long document-summarization requests (8K-token prompts). Without disaggregation, what is the most common symptom users will notice?

#Multi-LoRA and Multi-Tenant GPU Serving

Serving N fine-tuned variants of the same base model used to mean N replicas. Modern systems load the base model once and swap LoRA adapters per request, paying only the cost of the rank-r matrices (typically 50-200 MB per adapter vs. 13 GB per full model).

vLLM --enable-lora — supports hundreds of concurrent LoRA adapters with negligible overhead.
S-LoRA (Stanford, 2023) — unified paging for adapters and KV cache; 4x more adapters per GPU than naive swapping.
Punica. Kernel-level fusion of multiple LoRAs in a single CUDA call (SGMV — Segmented Gather Matrix-Vector).
TGI multi-LoRA. Same pattern for HuggingFace stacks.

This pattern is what makes per-customer fine-tuning economically feasible: 500 enterprise customers can each have a custom adapter served from one shared base-model fleet.

Inference efficiency techniques from the NLP track — KV-cache management, paged attention, speculative decoding — are the foundation for every production serving system covered here. Tensor and pipeline parallelism, originally developed for training, are now standard in inference for 70B+ models.

#The Inference Request Lifecycle

Every model prediction follows the same path from request to response. Understanding each step reveals where latency hides and where optimization pays off:

#Step 1: Model Artifact

The journey starts with a trained model artifact stored in a model registry -- a file containing the architecture definition, learned weights, and any preprocessing artifacts (tokenizers, scalers). This artifact was produced by the training pipeline and promoted through evaluation gates.

#Step 2: Load into Serving Framework

At startup, the serving framework (vLLM, TorchServe, Triton) loads the model weights into GPU memory. For a 70B model in FP16, this means transferring 140 GB from disk to GPU -- a cold start that takes 30-60 seconds. Replicas stay warm to avoid this latency on user requests.

#Step 3: REST/gRPC Endpoint

The model is exposed as a network endpoint. REST (HTTP/JSON) for universality, gRPC (HTTP/2 + Protobuf) for low-latency microservices, or SSE/WebSocket for streaming LLM tokens. A load balancer distributes incoming requests across model replicas.

#Step 4: Request Arrives

A user request hits the API gateway, which handles authentication, rate limiting, and routing. The request is validated (correct schema, input size limits) and placed in a queue. For dynamic batching, the request waits up to N milliseconds for other requests to form a batch.

#Batching Strategies Deep Dive

pythonplayground.py · Pyodide

# Simulating different batching strategies for inference
import random

def simulate_inference(arrival_ms: float, processing_ms: float) -> dict:
  return {"arrival_ms": arrival_ms, "processing_ms": processing_ms}

# Strategy 1: No batching (process one at a time)
def no_batching(requests: list[dict]) -> dict:
  current_time = 0.0
  total_latency = 0.0
  for req in requests:
      wait_time = max(0, current_time - req["arrival_ms"])
      current_time = max(current_time, req["arrival_ms"]) + req["processing_ms"]
      total_latency += wait_time + req["processing_ms"]
  return {
      "strategy": "No Batching",
      "avg_latency": f"{total_latency / len(requests):.1f}",
      "throughput": f"{len(requests) / current_time * 1000:.1f}",
  }

# Strategy 2: Static batching (wait for batch_size requests)
def static_batching(requests: list[dict], batch_size: int) -> dict:
  current_time = 0.0
  total_latency = 0.0
  for i in range(0, len(requests), batch_size):
      batch = requests[i:i + batch_size]
      batch_start = max(current_time, batch[-1]["arrival_ms"])
      batch_process = max(r["processing_ms"] for r in batch)
      current_time = batch_start + batch_process
      for r in batch:
          total_latency += current_time - r["arrival_ms"]
  return {
      "strategy": f"Static Batch ({batch_size})",
      "avg_latency": f"{total_latency / len(requests):.1f}",
      "throughput": f"{len(requests) / current_time * 1000:.1f}",
  }

# Generate 20 requests arriving over 2 seconds
requests = [
  simulate_inference(i * 100 + random.random() * 50, 80 + random.random() * 40)
  for i in range(20)
]

print("=== Batching Strategy Comparison (20 requests) ===\n")
results = [
  no_batching(requests),
  static_batching(requests, 2),
  static_batching(requests, 4),
  static_batching(requests, 8),
]

print(f"{'Strategy':<25}{'Avg Latency(ms)':<18}{'Throughput(req/s)'}")
print("-" * 60)
for r in results:
  print(f"{r['strategy']:<25}{r['avg_latency']:<18}{r['throughput']}")

print("\nCHALLENGE: Implement 'dynamic batching' that waits up to max_wait_ms")
print("OR until batch_size requests accumulate, whichever comes first.")

Tests · Compare no-batching, static, and dynamic strategies. Verify that dynamic batching provides better throughput than no batching while keeping latency lower than large static batches.

#Inference Optimization Techniques

#Model Compilation

Convert dynamic computation graphs to static, optimized ones:

TorchScript / torch.compile: PyTorch graph optimization, kernel fusion
ONNX Runtime: Cross-platform optimized inference
TensorRT: NVIDIA GPU-specific optimization. Kernel fusion, precision calibration.
OpenVINO: Intel CPU/GPU optimization

#Quantization at Serving Time

Reduce precision for faster inference (covered in depth in the next lesson):

FP32 -> FP16: 2x memory reduction, ~2x speedup, minimal quality loss
FP16 -> INT8: 2x further reduction, requires calibration
INT8 -> INT4: Another 2x, primarily for LLMs (GPTQ, AWQ)

#KV Cache Optimization

For LLMs, the KV cache is often the memory bottleneck:

Multi-Query Attention (MQA): Share K/V heads across attention heads (Llama 2 uses GQA, a middle ground)
Sliding window attention: Only cache the last N tokens (Mistral)
KV cache quantization: Quantize cached keys/values to INT8 or FP8
Prefix caching: Hash and reuse the KV cache for common prompt prefixes (system prompts, RAG templates) — vLLM and SGLang ship this out of the box; cache hit rates of 30-60% are typical in chat workloads.
Cross-request offload: Page cold KV-cache blocks to host memory or NVMe (LMCache, Mooncake) so popular conversations stay resident.

#Sizing Your Serving Fleet: Run the Math

Before signing off on hardware, work the numbers. The cell below estimates KV-cache memory per request, total memory headroom, and the maximum concurrent batch size for a vLLM-style replica — the exact math your platform team needs in their capacity plan.

Loading visualization...

Watch Out

Common Mistake

Optimizing the wrong bottleneck. Before optimizing, profile! Many teams spend weeks optimizing model computation when the actual bottleneck is data preprocessing, network latency, or serialization. Use profiling tools (PyTorch Profiler, NVIDIA Nsight, cProfile) to identify the real bottleneck first.

Ignoring cold start latency. Loading a 70B model into GPU memory takes 30-60 seconds. If your autoscaler spins up a new replica for burst traffic, the first requests will time out. Solution: pre-warm replicas, keep a minimum replica count, or use model-in-memory techniques.

Not testing under realistic load. A model that serves one request in 50ms might take 500ms under 100 concurrent requests due to GPU memory contention and batching delays. Always load test with realistic traffic patterns before declaring "it is fast enough."

Watch Out

Common Mistake

Not setting max_tokens and ignoring KV cache management in production. Every unfinished LLM request holds KV cache memory proportional to the number of tokens generated so far. Without a hard max_tokens cap, a single runaway generation (a user who prompts for "write me a 10,000-word essay") can exhaust the KV cache pool and degrade latency for every other concurrent request. vLLM's PagedAttention helps, but it does not eliminate the problem — it defers it.

Set aggressive max_tokens defaults tuned to your p99 use case. Implement per-user and per-request token budgets at the gateway layer, not just in the model config. Monitor KV cache utilization alongside GPU memory: a cache that runs at 95% occupancy will cause paging, increased latency, and — under continuous batching — request stalls. Treat KV cache as the primary resource to manage in LLM serving, not an afterthought.

#Key Takeaways

Training and serving are fundamentally different workloads. Training optimizes for throughput (process data as fast as possible), while serving optimizes for latency (respond to each request in milliseconds)
LLM serving uses specialized techniques for efficiency. PagedAttention manages KV cache memory dynamically, continuous batching maximizes GPU utilization, and speculative decoding predicts future tokens to reduce latency
Choose your serving pattern based on requirements. REST APIs for simple request-response, gRPC for low-latency microservices, and streaming (SSE/WebSocket) for LLM token-by-token generation
Cost, latency, throughput, and availability form a quadrilateral tradeoff. You cannot optimize all four simultaneously; production systems must make explicit choices about which dimensions matter most for their use case

#Quick Check

Quick Check1 / 4

What is PagedAttention (used in vLLM) and why does it matter?

#Exit Ticket

Before moving on, you should be able to answer these in your own words. If any feel shaky, scroll back.

You now know how to serve models at scale with modern inference systems. Next up: Model Compression & Efficiency -- how quantization, pruning, and distillation make models smaller and faster without (much) quality loss.

Your Reflection

Continue Learning

Discussion

Model Serving & Inference

#From Training to Serving: A Different World

#Serving Protocols: REST vs gRPC vs Streaming

#REST (HTTP/JSON)

#gRPC (HTTP/2 + Protocol Buffers)

#Server-Sent Events / WebSocket Streaming

#Traditional Model Serving

Serving frameworks

Key optimizations

#LLM Serving: A Different Beast

#vLLM: PagedAttention

#Continuous Batching

#Speculative Decoding

#LLM Serving Systems Comparison

#Inference Disaggregation (2024-2026)

#Multi-LoRA and Multi-Tenant GPU Serving

#The Inference Request Lifecycle

#Step 1: Model Artifact

#Step 2: Load into Serving Framework

#Step 3: REST/gRPC Endpoint

#Step 4: Request Arrives

#Batching Strategies Deep Dive

#Inference Optimization Techniques

#Model Compilation

#Quantization at Serving Time

#KV Cache Optimization

#Sizing Your Serving Fleet: Run the Math

#Key Takeaways

#Quick Check

Deep Dive

Related Lessons

#Exit Ticket

#Step 5: Preprocess Input

#Step 6: Model Inference

#Step 7: Postprocess Output

#Step 8: Return Response