Training a frontier model costs tens of millions of dollars — once. Serving it costs the same every month, forever. By 2025, inference cost is what decides whether your LLM product survives — and the tricks that win (int4 quantization, speculative decoding, paged KV cache) are why Llama 3.3 70B runs on a single H100 and Claude Haiku 4.5 serves answers for fractions of a cent per token.
Learning Objectives
After this lesson, you will be able to:
Quantize an LLM from fp16 down to int4 with GPTQ or AWQ — and know when int4 is free, when it costs 1-3% quality, and when (int2 or below) it falls off the cliff
Use speculative decoding to get a 2-3x latency speedup with no quality loss — and predict the speedup from the draft-acceptance rate
Choose between vLLM, llama.cpp, and TGI for production serving based on whether you're throughput-bound, latency-bound, or running on consumer hardware
Diagnose whether your inference workload is prefill-bound (compute) or decode-bound (memory bandwidth) and pick the right intervention for each
Don't worry if "quantization" sounds like a separate field -- it's mostly four ideas (per-tensor scaling, per-group scaling, calibration data, outlier handling), and the libraries do the heavy lifting once you understand the tradeoffs.
fp32: 4 bytes per parameter, the original training format.
fp16 / bf16: 2 bytes per parameter, the modern training default.
int8: 1 byte per parameter, "essentially free" quality-wise on most models.
int4: 0.5 bytes per parameter, costs 1-3% quality on most benchmarks.
int2 / int1 (BitNet): research territory, 5-15% quality loss without specialized training.
The math is brutal: a 70B-parameter model in fp16 is 140GB (won't fit in any single GPU). The same model in int4 is 35GB and fits on one A100 80GB or two consumer 24GB GPUs.
Per-tensor scaling (one s for the whole layer) is naive and loses too much precision. Per-channel and per-group quantization preserve more of the dynamic range. Modern formats (GGUF, AWQ) use group sizes of 32-128 weights.
Naive rounding is locally optimal but globally suboptimal. GPTQ (Frantar 2023) quantizes one column at a time, computing the optimal correction to the remaining un-quantized weights via the inverse Hessian.
Δw=−[H−1]qqH−1ϵ
GPTQ is the de-facto standard for serving (Hugging Face Transformers, vLLM, TGI all support GPTQ-quantized models out of the box). AutoGPTQ and gptqmodel are the standard Python libraries.
AWQ (Lin 2024) makes the observation that a small fraction of weights (~1%) carry most of the salient information, and these "salient" weights correlate with high-magnitude activations. By scaling salient weight channels up before quantization, you protect them from rounding error at the cost of slightly compressing the rest.
The result: AWQ-quantized models often match GPTQ quality but are 10-30% faster at inference because they don't require runtime dequantization for the salient channels (they're already at higher effective precision).
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch
# 4-bit NF4 quantization with double quantization (QLoRA-style)
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3-8B-Instruct",
quantization_config=bnb_config,
device_map="auto",
)
# Memory: ~5.5 GB instead of 16 GB for fp16
# Quality drop: usually under 2% on MMLU/HumanEval
What Do You Think?
You serve a 70B model with very long input context (50k tokens) and short outputs (200 tokens). Where is the bottleneck?
Prefill is compute-bound -- it processes all 50k input tokens in a single forward pass, which is FLOPs-heavy. Decode is memory-bandwidth-bound -- each token requires reading the entire model from VRAM, but does relatively little compute per token. With short outputs and very long inputs, prefill dominates total time. Modern serving systems (Mooncake, Splitwise) literally split prefill and decode onto different GPU pools because they have different optimal hardware profiles.
Autoregressive generation is sequential by definition: you can't predict token N+1 without knowing token N. This means decode latency is dominated by the per-token forward pass through the full model.
Vanilla speculative (Leviathan 2023, Chen 2023): a separate small model is the draft.
Medusa (Cai 2024): instead of a separate model, attach multiple parallel "medusa heads" to the main model that predict tokens k=1, k=2, k=3 ahead.
EAGLE (Li 2024): an autoregressive prediction head trained on the verifier's hidden states; achieves higher acceptance rate than Medusa.
Lookahead decoding (Fu 2024): no draft model needed; uses Jacobi-like fixed-point iteration on a sliding window of future tokens.
In production, vLLM and TGI both support vanilla speculative; EAGLE is the current research frontier with the highest acceptance rates (~0.85+ on common workloads).
The naive serving loop is wasteful: launch a request, run it to completion, GPU sits idle waiting for the next request. Continuous batching (Yu 2022, Orca) and vLLM's PagedAttention (Kwon 2023) fix this.
KV cache memory per request=2⋅L⋅d⋅seq_len⋅bytes_per_param
PagedAttention also enables prefix caching -- if two requests share a system prompt, they can point to the same physical blocks for that prefix. For chatbot workloads with fixed system prompts, this can cut compute by 30-50%.
pythonplayground.py · Pyodide
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
Tests · Verify int4 model loads in <8 GB on a T4 (16 GB), generates coherent text, and uses ~30% less memory than fp16.
#Inference economics: what does a token actually cost?
Throughput numbers like "80 tokens per second" feel abstract until you turn them into dollars. The conversion is mechanical: pick a model and quantization, look up the prefill and decode rates on a target GPU, divide GPU rental cost by tokens produced, and you have $/token. Let us run the arithmetic on a concrete workload — a customer-support chatbot answering one-thousand-token prompts with two-hundred-token replies — and compare self-hosted Llama against a metered API.
A solid 2026 baseline: Llama 3.3 70B quantized to int4 running on a single H100 SXM (80 GB) under vLLM. Published benchmarks land around 500 tokens/sec prefill and 80 tokens/sec decode at batch 1. The H100 rents for roughly $2.50/hour on spot (RunPod, Lambda, Together) and ~$4/hour on-demand. For one request: prefill is 1000 / 500 = 2.0 s and decode is 200 / 80 = 2.5 s, so total compute time is ~4.5 s. At $2.50/hr that is 4.5 × 2.50 / 3600 = $0.0031 per request, or $3.10 per 1,000 requests. Stack a batch of 16 concurrent requests via continuous batching and the per-request cost can drop another 5–10×.
The headline API alternative is GPT-4o-mini at roughly $0.15 / M input tokens and $0.60 / M output tokens (OpenAI 2025 pricing). The same 1k-input / 200-output request costs 1000 × 0.15/1e6 + 200 × 0.60/1e6 = $0.00015 + $0.00012 = $0.00027, or $0.27 per 1,000 requests. The API is ~10× cheaper here because (a) you only pay for what you use — no idle GPU, (b) the API runs a much smaller model than 70B, and (c) the provider amortizes one H100 across thousands of tenants.
So when does self-hosted win? When your GPU is already saturated. If you keep an H100 busy 80% of the time and average ~200 concurrent decodes per second, you produce roughly 50M tokens per day for ~$60 of GPU — about $1.20 / M tokens all-in. The API at $0.60/M output tokens would cost $30 for the same output volume, but for output-heavy workloads (long generations) the line crosses around 40–80M tokens/day of steady load. Below that, the API wins on total cost of ownership; above it, self-hosting wins by a wide margin and only widens at scale.
Two practical caveats. First, prefill is roughly 6× cheaper per token than decode on these numbers (500 vs 80 tok/s), which is why caching prompts (prefix caching, OpenAI's prompt cache, Anthropic's prompt caching) is the single biggest cost lever for repetitive workloads — the chatbot system prompt gets billed once instead of every turn. Second, latency is not the same as throughput-cost: an H100 doing batch-1 decode at 80 tok/s gives 12.5 ms per token (fast) but is wildly more expensive than the same H100 in batch-32 mode, which might give 35 ms per token but serve 32 users at once. SLO targets, not raw throughput, dictate which mode you run in.
The right inference engine depends on what you are optimizing for. The table below is the short version of "which one do I install on Monday morning."
Engine
Best at
Hardware
When to pick it
vLLM
High-throughput multi-tenant serving
Nvidia (A100/H100/L40S)
Default for production APIs. PagedAttention + continuous batching + GPTQ/AWQ.
SGLang
Structured outputs, constrained gen
Nvidia
Heavy JSON-mode/grammar workloads or RAG with shared prefixes — RadixAttention shines.
TGI
HF-ecosystem enterprise
Nvidia / AMD
You already live on Hugging Face Hub, need gRPC + tensor parallel + safetensors.
TensorRT-LLM
Lowest latency on Nvidia
Nvidia only
You are willing to compile per-model and lock in to Nvidia for max tok/s.
llama.cpp
Consumer / edge / Apple Silicon
CPU, Metal, CUDA, ROCm
Mac mini, single-user laptop, or air-gapped deployments. GGUF q4 / q5 / q6.
For multi-tenant production with mixed-length requests, vLLM is the safe default. For constrained-output workloads (JSON, regex, function-calling at high QPS), SGLang often wins by 1.5–3×. For long-context retrieval workloads where many users share the same retrieved chunks, prefix caching in vLLM or RadixAttention in SGLang is worth more than any quantization tweak. For local / single-user / Apple Silicon, nothing beats llama.cpp.
The playground below turns the cost equation into a slider toy: dial in model size, GPU count, hourly cost, and throughput, and watch $/M tokens move.
int8 quantization is essentially free; int4 costs 1-3% quality. Modern formats (GPTQ, AWQ, NF4 with bitsandbytes) all deliver excellent int4 quality with proper per-group scaling. int2 and below remain research territory.
GPTQ vs AWQ: GPTQ is more flexible, AWQ is faster. Both achieve similar quality. AWQ's activation-aware salient-channel handling makes it 10-30% faster at inference, and it's the modern default for new deployments. GPTQ has wider tooling support.
Speculative decoding gives 2-3x latency speedup with zero quality loss. A small draft model generates k tokens, the big verifier accepts or rejects them in parallel. Net speedup depends on the draft acceptance rate (α): higher α = more accepted draft tokens per verify step.
vLLM's PagedAttention is the production default for high-throughput serving. Continuous batching plus block-level KV cache management raises GPU utilization from 20-40% (naive) to 80-95%. Prefix caching is a free additional win for chatbot workloads with shared system prompts.
Prefill and decode have different bottlenecks. Prefill is compute-bound (FLOPs); decode is memory-bandwidth-bound (reading model weights from VRAM). Modern serving systems are starting to disaggregate the two onto separate GPU pools optimized for each.
You quantize a 70B model from fp16 to int4. What's the approximate memory reduction?
Modern LLM serving is a sequence of well-understood tricks — quantize, speculate, batch — composed into 10-50x throughput improvements. Next up: Mixture of Experts, the architecture trick that scales total parameters faster than active parameters per token, and which only made commercial sense once these inference-efficiency tools matured.