Multi-Model Serving & GPU Infrastructure

A single-model GPU endpoint typically runs at 5-15% utilization — you're paying for a $30K H100 to idle. Multi-model serving with Triton, vLLM, MIG partitioning, and BentoML pushes utilization to 60-80%. Same models, same SLA, one-fifth the bill.

Learning Objectives

After this lesson, you will be able to:

Architect multi-model serving with shared GPU infrastructure (Triton, vLLM, BentoML) to maximize utilization vs single-model deployment
Apply GPU sharing techniques (MIG, MPS, virtual GPUs) to run multiple workloads on one physical GPU
Use spot/preemptible instances + autoscaling to cut serving costs 40-70% for non-critical workloads
Pick the right serving topology: dedicated per-model, multi-model endpoints, model orchestrator + routing

Prerequisites

Before You Start

Review

#The GPU Utilization Problem

Single-model serving wastes GPUs. A typical workload:

LLM inference is memory-bandwidth-bound, not compute-bound
Burst traffic patterns leave GPUs idle 80%+ of the time
Each model gets one GPU minimum, even if it only needs 10% of one

The fix: serve multiple models on the same GPU(s).

\text{Cost ratio} = \frac{\sum_i \text{dedicated GPU cost}_i}{\text{shared GPU cost}} \quad \text{typically 3-7× savings}

#Serving Topology: Three Architectures

Before picking a tool, decide which topology you're building. The three production-grade options span a clear cost / isolation trade space:

Topology	What it means	Cost	Isolation
Dedicated per-model	One serving deployment per model; one GPU minimum	Highest	Strongest — each model has its own process and memory
Multi-model endpoint	One inference server, many models loaded; model selected per request	3-7x cheaper	Medium — same process; load/unload at request boundaries
Orchestrator + routing	A thin router fronts a heterogeneous fleet of dedicated/MME backends; classifies request -> route	Middle	Configurable — mix dedicated for hot models, MME for the long tail

For most teams, the right answer is the third: a hot core of high-traffic models on dedicated infra, a long tail of niche models on a shared multi-model endpoint, and a router that classifies the request and dispatches.

text

                          inbound request
                                  │
                                  ▼
                         +--------+--------+
                         | classifier /     |
                         |  router          |
                         +--------+--------+
                         /        |         \
                        ▼         ▼          ▼
              hot model 1    hot model 2    multi-model endpoint
              (dedicated)    (dedicated)    (50+ niche models on 1 GPU)
              200 QPS        180 QPS        cold + warm tier

#Multi-Model Serving Architectures

#1. Triton Inference Server (NVIDIA)

The production standard. Supports:

Dynamic batching: combine requests across users for GPU efficiency
Model ensembling: chain models in pipelines
Multi-framework: TensorRT, PyTorch, TensorFlow, ONNX, vLLM
Concurrent execution: multiple models on one GPU via streams

#2. vLLM (LLM-specific)

For LLM serving specifically:

Continuous batching: requests of varying lengths share GPU efficiently
PagedAttention: KV cache as a paging system (24× throughput vs naive)
Multi-LoRA: serve dozens of fine-tuned LoRA adapters on one base model

pythonreference · read-only

# Multi-LoRA serving
from vllm import LLM
llm = LLM(model="meta-llama/Llama-3-70b", enable_lora=True, max_loras=8)
# Now serve 8 different LoRA-fine-tuned variants from ONE 70B base model in memory

#Model Swapping at Request Boundaries (2026 frontier)

The 2026 production frontier for vLLM and Triton is request-boundary model swapping: the server holds a hot working set in GPU memory and swaps less-frequently-used models out to CPU/host memory (or NVMe), reloading them on demand. With NVLink-attached host memory and PCIe Gen5, the swap cost has dropped to 200-800ms for a 7B model — acceptable for the long tail of low-QPS variants. vLLM's experimental sleep-mode and Triton's instance group swapping both implement this. The win: a single H100 can effectively "serve" 30-100 models, with the hot 5-10 always resident and the rest paged in as traffic arrives. Cold starts go from "minutes to load checkpoint from S3" to "sub-second to page from host RAM."

#3. BentoML / SageMaker MME / Vertex AI Multi-Model Endpoints

Higher-level abstractions: define your models, the platform handles GPU sharing.

Mechanism	What it does	Use when
MIG (Multi-Instance GPU, NVIDIA H100/A100)	Hardware-level partitioning into 7 isolated instances	Production isolation needed
MPS (Multi-Process Service)	Software-level GPU time-slicing	Trusted workloads, no hard isolation
vGPU (NVIDIA virtualization)	Hypervisor-level sharing	Multi-tenant cloud environments
CUDA streams	Single-process concurrent kernels	Same model, multiple inference threads

For most production: MIG on H100 (hardware isolation, predictable performance) for isolated workloads, vLLM with continuous batching for LLM serving.

#Autoscaling Patterns

Strategy	When
Reactive (scale on QPS)	General-purpose
Predictive (ML-based forecasting)	Predictable diurnal patterns
Scheduled	Known traffic patterns (e.g., business hours)
Scale-to-zero	Long idle periods (Modal, Anyscale, Cloud Run)

For LLM serving specifically: scale-to-zero is risky because cold starts on H100 take 30-90s. Use warm pools or aggressive prediction.

#Cold-Start Mitigation

Cold starts are the single most user-visible failure of cost-optimized serving. The three mitigation tiers, in increasing cost:

Tier	Technique	Cold start	Cost
0	Pre-loaded model in GPU memory (always-warm replica)	0 ms	Highest — pay for idle GPU
1	Model on host RAM, page in on first request	200-800 ms	Low — host RAM is cheap
2	Model on local NVMe, mmap-load on first request	1-3 s	Lower — NVMe is cheap, no RAM cost
3	Model on S3/GCS, download on first request	30-90 s	Lowest — pay only for S3 storage

Production pattern: always keep tier-0 for hot models, tier-1 for warm models, tier-2 for cold but expected models, and tier-3 only for experiments. The cost difference between tier 0 and tier 3 is roughly 50-100x at typical traffic mixes. A 60-second cold start in user-facing serving is also a tail-latency disaster — p99 explodes.

Canary deployments and A/B traffic splitting add their own pre-warming needs: before flipping traffic to the new version, send shadow traffic for 5-10 minutes so the new replicas are warm before they take real load. Triton, KServe, and SageMaker all expose explicit warmup hooks.

#Canary + A/B Traffic Splitting

Pattern	Use	Risk
Blue/green	Two identical environments; instant traffic flip	Doubles infra for the cutover window
Canary 1% -> 5% -> 25% -> 100%	Progressive rollout; auto-rollback on metric regression	Slower to fully deploy; needs solid metric pipeline
Shadow traffic	Send copy of prod traffic to new replica; compare outputs offline	Doubles inference cost during shadow window
Multi-armed bandit A/B	Traffic split adapts based on observed quality	Requires real-time quality signal

Canary is the default for ML. Pre-warm new replicas via shadow traffic for ~10 min, ramp from 1% to 100% over 30-60 minutes with auto-rollback on latency/error regressions.

What Do You Think?

Your team runs a 70B model on a 4-GPU H100 pod. You add three new LoRA adapters per week. After two months, the pod is at 99% memory and you're seeing OOM kills. Most defensible response?

#Spot / Preemptible Strategy

Spot instances are 50-90% cheaper but can be reclaimed with 30s-2min notice. For ML serving:

Spot for inference replicas: most replicas on spot, ~10-20% on-demand for stability
Auto-failover: if spot reclaimed, traffic shifts to on-demand replicas seamlessly
NEVER spot for training: long-running jobs lose hours of progress on preemption (unless you checkpoint aggressively)

Combined: spot replicas + autoscaling + multi-model endpoints = 70-90% cost reduction vs naive dedicated on-demand.

Cost reasoning for multi-model serving is fiddly because the answer depends on traffic mix, baseline utilization, sharing mechanism, and spot/on-demand strategy. Below is a parametric calculator you can twiddle to see how each lever moves the bill.

A Pyodide-backed scratchpad for math lessons.

Loading visualization...

The shape of the answer matters more than the absolute number: dedicated serving cost scales linearly with model count, MIG-shared scales as ceil(N / 7), and multi-LoRA scales as roughly constant (one H100, dozens of adapters) until you hit the per-GPU memory ceiling. Knowing where each curve sits lets you predict the cost crossover points without rerunning the calculator: above ~5-8 models, multi-LoRA wins decisively; below that, MIG or even dedicated may be simpler.

Quick check

A startup is serving one production model on a dedicated H100, 24/7, with bursty traffic and 8% average GPU utilization. Their CFO wants a 50% cost cut. Lowest-risk single change?

#Hands-On

pythonplayground.py · Pyodide

# Compare dedicated vs shared serving cost calculator (back-of-envelope)
# Run: python cost_calculator.py

# 2026 H100 pricing (approximate)
H100_HOURLY_ONDEMAND = 4.00  # USD/hour, AWS p5
H100_HOURLY_SPOT = 1.50      # USD/hour, AWS p5 spot
H100_RESERVED_3YR = 1.30     # USD/hour, 3-year commit

# MIG partitions on H100 SXM (80GB): 7×10GB instances
MIG_PARTITIONS = 7

class ServingScenario:
  def __init__(self, name, gpus, hourly_cost, utilization, replicas=1):
      self.name = name
      self.gpus = gpus
      self.hourly_cost = hourly_cost
      self.utilization = utilization
      self.replicas = replicas

def monthly_cost(self):
      return self.gpus * self.hourly_cost * 24 * 30 * self.replicas

# --- Scenario: serve 5 medium-size LLMs (Llama-3-8B each) ---
print("=== Serving 5 Llama-3-8B models, ~16GB each ===\n")

scenarios = [
  ServingScenario("Dedicated H100 each (on-demand)", gpus=5, hourly_cost=H100_HOURLY_ONDEMAND, utilization=0.10),
  ServingScenario("Dedicated H100 each (spot)", gpus=5, hourly_cost=H100_HOURLY_SPOT, utilization=0.10),
  ServingScenario("Single H100 with MIG (5 of 7 partitions)", gpus=1, hourly_cost=H100_HOURLY_ONDEMAND, utilization=0.65),
  ServingScenario("Single H100 with vLLM + multi-LoRA (5 LoRAs on 1 base)", gpus=1, hourly_cost=H100_HOURLY_ONDEMAND, utilization=0.75),
]

print(f"{'Scenario':<55} {'$/month':>12}  {'GPU util':>10}")
print("-" * 80)
for s in scenarios:
  print(f"{s.name:<55} ${s.monthly_cost():>10,.0f}  {s.utilization * 100:>8.0f}%")

baseline = scenarios[0].monthly_cost()
print(f"\nSavings vs dedicated on-demand baseline (${baseline:,.0f}/month):")
for s in scenarios[1:]:
  saving = (baseline - s.monthly_cost()) / baseline * 100
  print(f"  {s.name}: {saving:.0f}% savings (${baseline - s.monthly_cost():,.0f}/month)")

print("\nCHALLENGE: Add an autoscaling scenario where 2 of 5 model")
print("families have diurnal traffic — peak 4 hours/day, idle the rest.")
print("Compute savings from scale-to-zero during idle hours.")

Tests · Verify the calculator shows multi-model + MIG savings of 80%+ vs dedicated on-demand. Verify spot pricing is ~63% cheaper. Verify multi-LoRA on single H100 is the cheapest at fixed accuracy.

#Key Takeaways

Multi-model serving is 3-7× cheaper than dedicated. Single biggest cost lever in ML production
Triton + vLLM are the production standards. Triton for general multi-model, vLLM for LLM-specific
MIG provides hardware-isolated GPU sharing. 7 partitions per H100, predictable performance
Spot instances cut 50-90% off serving costs. With mixed on-demand for stability
vLLM's multi-LoRA serves dozens of fine-tunes from one base model in memory. Game-changing for fine-tune-heavy workloads

#Quick Check

Quick Check1 / 2

Why does multi-model serving on shared GPUs typically save 3-7× on cost vs dedicated per-model deployment?

Multi-model serving is the highest-leverage cost optimization in production ML. Next: model security and adversarial robustness for the deployed system.

Multi-Model Serving & GPU Infrastructure

Learning Objectives

After this lesson, you will be able to:

Architect multi-model serving with shared GPU infrastructure (Triton, vLLM, BentoML) to maximize utilization vs single-model deployment
Apply GPU sharing techniques (MIG, MPS, virtual GPUs) to run multiple workloads on one physical GPU
Use spot/preemptible instances + autoscaling to cut serving costs 40-70% for non-critical workloads
Pick the right serving topology: dedicated per-model, multi-model endpoints, model orchestrator + routing

Prerequisites

Before You Start

Review

#The GPU Utilization Problem

Single-model serving wastes GPUs. A typical workload:

LLM inference is memory-bandwidth-bound, not compute-bound
Burst traffic patterns leave GPUs idle 80%+ of the time
Each model gets one GPU minimum, even if it only needs 10% of one

The fix: serve multiple models on the same GPU(s).

\text{Cost ratio} = \frac{\sum_i \text{dedicated GPU cost}_i}{\text{shared GPU cost}} \quad \text{typically 3-7× savings}

#Serving Topology: Three Architectures

Before picking a tool, decide which topology you're building. The three production-grade options span a clear cost / isolation trade space:

Topology	What it means	Cost	Isolation
Dedicated per-model	One serving deployment per model; one GPU minimum	Highest	Strongest — each model has its own process and memory
Multi-model endpoint	One inference server, many models loaded; model selected per request	3-7x cheaper	Medium — same process; load/unload at request boundaries
Orchestrator + routing	A thin router fronts a heterogeneous fleet of dedicated/MME backends; classifies request -> route	Middle	Configurable — mix dedicated for hot models, MME for the long tail

text

                          inbound request
                                  │
                                  ▼
                         +--------+--------+
                         | classifier /     |
                         |  router          |
                         +--------+--------+
                         /        |         \
                        ▼         ▼          ▼
              hot model 1    hot model 2    multi-model endpoint
              (dedicated)    (dedicated)    (50+ niche models on 1 GPU)
              200 QPS        180 QPS        cold + warm tier

#Multi-Model Serving Architectures

#1. Triton Inference Server (NVIDIA)

The production standard. Supports:

Dynamic batching: combine requests across users for GPU efficiency
Model ensembling: chain models in pipelines
Multi-framework: TensorRT, PyTorch, TensorFlow, ONNX, vLLM
Concurrent execution: multiple models on one GPU via streams

#2. vLLM (LLM-specific)

For LLM serving specifically:

Continuous batching: requests of varying lengths share GPU efficiently
PagedAttention: KV cache as a paging system (24× throughput vs naive)
Multi-LoRA: serve dozens of fine-tuned LoRA adapters on one base model

pythonreference · read-only

# Multi-LoRA serving
from vllm import LLM
llm = LLM(model="meta-llama/Llama-3-70b", enable_lora=True, max_loras=8)
# Now serve 8 different LoRA-fine-tuned variants from ONE 70B base model in memory

#Model Swapping at Request Boundaries (2026 frontier)

#3. BentoML / SageMaker MME / Vertex AI Multi-Model Endpoints

Higher-level abstractions: define your models, the platform handles GPU sharing.

Mechanism	What it does	Use when
MIG (Multi-Instance GPU, NVIDIA H100/A100)	Hardware-level partitioning into 7 isolated instances	Production isolation needed
MPS (Multi-Process Service)	Software-level GPU time-slicing	Trusted workloads, no hard isolation
vGPU (NVIDIA virtualization)	Hypervisor-level sharing	Multi-tenant cloud environments
CUDA streams	Single-process concurrent kernels	Same model, multiple inference threads

For most production: MIG on H100 (hardware isolation, predictable performance) for isolated workloads, vLLM with continuous batching for LLM serving.

#Autoscaling Patterns

Strategy	When
Reactive (scale on QPS)	General-purpose
Predictive (ML-based forecasting)	Predictable diurnal patterns
Scheduled	Known traffic patterns (e.g., business hours)
Scale-to-zero	Long idle periods (Modal, Anyscale, Cloud Run)

For LLM serving specifically: scale-to-zero is risky because cold starts on H100 take 30-90s. Use warm pools or aggressive prediction.

#Cold-Start Mitigation

Cold starts are the single most user-visible failure of cost-optimized serving. The three mitigation tiers, in increasing cost:

Tier	Technique	Cold start	Cost
0	Pre-loaded model in GPU memory (always-warm replica)	0 ms	Highest — pay for idle GPU
1	Model on host RAM, page in on first request	200-800 ms	Low — host RAM is cheap
2	Model on local NVMe, mmap-load on first request	1-3 s	Lower — NVMe is cheap, no RAM cost
3	Model on S3/GCS, download on first request	30-90 s	Lowest — pay only for S3 storage

#Canary + A/B Traffic Splitting

Pattern	Use	Risk
Blue/green	Two identical environments; instant traffic flip	Doubles infra for the cutover window
Canary 1% -> 5% -> 25% -> 100%	Progressive rollout; auto-rollback on metric regression	Slower to fully deploy; needs solid metric pipeline
Shadow traffic	Send copy of prod traffic to new replica; compare outputs offline	Doubles inference cost during shadow window
Multi-armed bandit A/B	Traffic split adapts based on observed quality	Requires real-time quality signal

Canary is the default for ML. Pre-warm new replicas via shadow traffic for ~10 min, ramp from 1% to 100% over 30-60 minutes with auto-rollback on latency/error regressions.

What Do You Think?

Your team runs a 70B model on a 4-GPU H100 pod. You add three new LoRA adapters per week. After two months, the pod is at 99% memory and you're seeing OOM kills. Most defensible response?

#Spot / Preemptible Strategy

Spot instances are 50-90% cheaper but can be reclaimed with 30s-2min notice. For ML serving:

Spot for inference replicas: most replicas on spot, ~10-20% on-demand for stability
Auto-failover: if spot reclaimed, traffic shifts to on-demand replicas seamlessly
NEVER spot for training: long-running jobs lose hours of progress on preemption (unless you checkpoint aggressively)

Combined: spot replicas + autoscaling + multi-model endpoints = 70-90% cost reduction vs naive dedicated on-demand.

A Pyodide-backed scratchpad for math lessons.

Loading visualization...

Quick check

A startup is serving one production model on a dedicated H100, 24/7, with bursty traffic and 8% average GPU utilization. Their CFO wants a 50% cost cut. Lowest-risk single change?

#Hands-On

pythonplayground.py · Pyodide

# Compare dedicated vs shared serving cost calculator (back-of-envelope)
# Run: python cost_calculator.py

# 2026 H100 pricing (approximate)
H100_HOURLY_ONDEMAND = 4.00  # USD/hour, AWS p5
H100_HOURLY_SPOT = 1.50      # USD/hour, AWS p5 spot
H100_RESERVED_3YR = 1.30     # USD/hour, 3-year commit

# MIG partitions on H100 SXM (80GB): 7×10GB instances
MIG_PARTITIONS = 7

def monthly_cost(self):
      return self.gpus * self.hourly_cost * 24 * 30 * self.replicas

# --- Scenario: serve 5 medium-size LLMs (Llama-3-8B each) ---
print("=== Serving 5 Llama-3-8B models, ~16GB each ===\n")

print(f"{'Scenario':<55} {'$/month':>12}  {'GPU util':>10}")
print("-" * 80)
for s in scenarios:
  print(f"{s.name:<55} ${s.monthly_cost():>10,.0f}  {s.utilization * 100:>8.0f}%")

Tests · Verify the calculator shows multi-model + MIG savings of 80%+ vs dedicated on-demand. Verify spot pricing is ~63% cheaper. Verify multi-LoRA on single H100 is the cheapest at fixed accuracy.

#Key Takeaways

Multi-model serving is 3-7× cheaper than dedicated. Single biggest cost lever in ML production
Triton + vLLM are the production standards. Triton for general multi-model, vLLM for LLM-specific
MIG provides hardware-isolated GPU sharing. 7 partitions per H100, predictable performance
Spot instances cut 50-90% off serving costs. With mixed on-demand for stability
vLLM's multi-LoRA serves dozens of fine-tunes from one base model in memory. Game-changing for fine-tune-heavy workloads

#Quick Check

Quick Check1 / 2

Why does multi-model serving on shared GPUs typically save 3-7× on cost vs dedicated per-model deployment?

Multi-model serving is the highest-leverage cost optimization in production ML. Next: model security and adversarial robustness for the deployed system.

Multi-Model Serving & GPU Infrastructure

#The GPU Utilization Problem

#Serving Topology: Three Architectures

#Multi-Model Serving Architectures

#1. Triton Inference Server (NVIDIA)

#2. vLLM (LLM-specific)

#Model Swapping at Request Boundaries (2026 frontier)

#3. BentoML / SageMaker MME / Vertex AI Multi-Model Endpoints

#GPU Sharing Mechanisms

#Autoscaling Patterns

#Cold-Start Mitigation

#Canary + A/B Traffic Splitting

#Spot / Preemptible Strategy

#Multi-Model GPU Sharing Cost Calculator

#Hands-On

#Key Takeaways

#Quick Check

Deep Dive

Your Reflection

Continue Learning

Discussion

Multi-Model Serving & GPU Infrastructure

#The GPU Utilization Problem

#Serving Topology: Three Architectures

#Multi-Model Serving Architectures

#1. Triton Inference Server (NVIDIA)

#2. vLLM (LLM-specific)

#Model Swapping at Request Boundaries (2026 frontier)

#3. BentoML / SageMaker MME / Vertex AI Multi-Model Endpoints

#GPU Sharing Mechanisms

#Autoscaling Patterns

#Cold-Start Mitigation

#Canary + A/B Traffic Splitting

#Spot / Preemptible Strategy

#Multi-Model GPU Sharing Cost Calculator

#Hands-On

#Key Takeaways

#Quick Check

Deep Dive

Your Reflection

Continue Learning

Discussion