A single-model GPU endpoint typically runs at 5-15% utilization — you're paying for a $30K H100 to idle. Multi-model serving with Triton, vLLM, MIG partitioning, and BentoML pushes utilization to 60-80%. Same models, same SLA, one-fifth the bill.
Learning Objectives
After this lesson, you will be able to:
Architect multi-model serving with shared GPU infrastructure — Triton, vLLM, BentoML — to maximize utilization vs single-model deployment
Apply GPU sharing techniques (MIG, MPS, virtual GPUs) to run multiple workloads on one physical GPU
Use spot/preemptible instances + autoscaling to cut serving costs 40-70% for non-critical workloads
Pick the right serving topology: dedicated per-model, multi-model endpoints, model orchestrator + routing
Before picking a tool, decide which topology you're building. The three production-grade options span a clear cost / isolation trade space:
Topology
What it means
Cost
Isolation
Dedicated per-model
One serving deployment per model; one GPU minimum
Highest
Strongest — each model has its own process and memory
Multi-model endpoint
One inference server, many models loaded; model selected per request
3-7x cheaper
Medium — same process; load/unload at request boundaries
Orchestrator + routing
A thin router fronts a heterogeneous fleet of dedicated/MME backends; classifies request -> route
Middle
Configurable — mix dedicated for hot models, MME for the long tail
For most teams, the right answer is the third: a hot core of high-traffic models on dedicated infra, a long tail of niche models on a shared multi-model endpoint, and a router that classifies the request and dispatches.
text
inbound request
│
▼
+--------+--------+
| classifier / |
| router |
+--------+--------+
/ | \
▼ ▼ ▼
hot model 1 hot model 2 multi-model endpoint
(dedicated) (dedicated) (50+ niche models on 1 GPU)
200 QPS 180 QPS cold + warm tier
Continuous batching: requests of varying lengths share GPU efficiently
PagedAttention: KV cache as a paging system (24× throughput vs naive)
Multi-LoRA: serve dozens of fine-tuned LoRA adapters on one base model
pythonreference · read-only
1
2
3
4
# Multi-LoRA serving
from vllm import LLM
llm = LLM(model="meta-llama/Llama-3-70b", enable_lora=True, max_loras=8)
# Now serve 8 different LoRA-fine-tuned variants from ONE 70B base model in memory
#Model Swapping at Request Boundaries (2026 frontier)
The 2026 production frontier for vLLM and Triton is request-boundary model swapping: the server holds a hot working set in GPU memory and swaps less-frequently-used models out to CPU/host memory (or NVMe), reloading them on demand. With NVLink-attached host memory and PCIe Gen5, the swap cost has dropped to 200-800ms for a 7B model — acceptable for the long tail of low-QPS variants. vLLM's experimental sleep-mode and Triton's instance group swapping both implement this. The win: a single H100 can effectively "serve" 30-100 models, with the hot 5-10 always resident and the rest paged in as traffic arrives. Cold starts go from "minutes to load checkpoint from S3" to "sub-second to page from host RAM."
#3. BentoML / SageMaker MME / Vertex AI Multi-Model Endpoints
Higher-level abstractions: define your models, the platform handles GPU sharing.
Cold starts are the single most user-visible failure of cost-optimized serving. The three mitigation tiers, in increasing cost:
Tier
Technique
Cold start
Cost
0
Pre-loaded model in GPU memory (always-warm replica)
0 ms
Highest — pay for idle GPU
1
Model on host RAM, page in on first request
200-800 ms
Low — host RAM is cheap
2
Model on local NVMe, mmap-load on first request
1-3 s
Lower — NVMe is cheap, no RAM cost
3
Model on S3/GCS, download on first request
30-90 s
Lowest — pay only for S3 storage
Production pattern: always keep tier-0 for hot models, tier-1 for warm models, tier-2 for cold but expected models, and tier-3 only for experiments. The cost difference between tier 0 and tier 3 is roughly 50-100x at typical traffic mixes. A 60-second cold start in user-facing serving is also a tail-latency disaster — p99 explodes.
Canary deployments and A/B traffic splitting add their own pre-warming needs: before flipping traffic to the new version, send shadow traffic for 5-10 minutes so the new replicas are warm before they take real load. Triton, KServe, and SageMaker all expose explicit warmup hooks.
Progressive rollout; auto-rollback on metric regression
Slower to fully deploy; needs solid metric pipeline
Shadow traffic
Send copy of prod traffic to new replica; compare outputs offline
Doubles inference cost during shadow window
Multi-armed bandit A/B
Traffic split adapts based on observed quality
Requires real-time quality signal
Canary is the default for ML. Pre-warm new replicas via shadow traffic for ~10 min, ramp from 1% to 100% over 30-60 minutes with auto-rollback on latency/error regressions.
What Do You Think?
Your team runs a 70B model on a 4-GPU H100 pod. You add three new LoRA adapters per week. After two months, the pod is at 99% memory and you're seeing OOM kills. Most defensible response?
Cost reasoning for multi-model serving is fiddly because the answer depends on traffic mix, baseline utilization, sharing mechanism, and spot/on-demand strategy. Below is a parametric calculator you can twiddle to see how each lever moves the bill.
Loading visualization...
The shape of the answer matters more than the absolute number: dedicated serving cost scales linearly with model count, MIG-shared scales as ceil(N / 7), and multi-LoRA scales as roughly constant (one H100, dozens of adapters) until you hit the per-GPU memory ceiling. Knowing where each curve sits lets you predict the cost crossover points without rerunning the calculator: above ~5-8 models, multi-LoRA wins decisively; below that, MIG or even dedicated may be simpler.
Quick check
A startup is serving one production model on a dedicated H100, 24/7, with bursty traffic and 8% average GPU utilization. Their CFO wants a 50% cost cut. Lowest-risk single change?
Tests · Verify the calculator shows multi-model + MIG savings of 80%+ vs dedicated on-demand. Verify spot pricing is ~63% cheaper. Verify multi-LoRA on single H100 is the cheapest at fixed accuracy.
Why does multi-model serving on shared GPUs typically save 3-7× on cost vs dedicated per-model deployment?
Multi-model serving is the highest-leverage cost optimization in production ML. Next: model security and adversarial robustness for the deployed system.