A 70B model in FP16 needs 140GB of VRAM — two H100s. The same model quantized to INT4 fits in 35GB — one consumer GPU. Quantization (GPTQ, AWQ, GGUF), pruning, and distillation are the techniques that turn $30K/month inference bills into $300/month, and that make local LLMs possible at all.
Learning Objectives
After this lesson, you will be able to:
Understand quantization: why making numbers less precise (from 16-bit to 8-bit or 4-bit) still keeps a model smart enough to be useful
Compare the main quantization methods (GPTQ, AWQ, GGUF, bitsandbytes) and know when to use each
Combine multiple compression techniques (quantization, pruning, distillation) to hit a specific speed and memory goal
If you have ever wanted to run a powerful AI model on your own computer (or wondered how AI works on your phone), this lesson shows you exactly how it is done. Compression is one of the most practically useful skills in ML engineering -- and one of the most satisfying to see in action.
Every parameter in a neural network is a number. The precision of that number determines how much memory it uses:
Format
Bits
Range
Memory per param
Use case
FP32
32
~10^38
4 bytes
Training (master copy)
BF16
16
~10^38 (less precision)
2 bytes
Training (compute)
FP16
16
~65504
2 bytes
Inference default
INT8
8
-128 to 127
1 byte
Quantized inference
INT4
4
-8 to 7
0.5 bytes
Aggressive quantization
NF4
4
Normal-float 4
0.5 bytes
QLoRA training
The key insight: Most model weights cluster around zero in a roughly normal distribution. You do not need 32 bits of precision to represent a weight that is 0.0023. Eight or even four bits are usually enough.
What Do You Think?
A 7B parameter model in FP16 uses 14 GB. How much memory would the same model use in INT4 quantization?
FP16 uses 2 bytes per parameter: 7B x 2 = 14 GB. INT4 uses 0.5 bytes per parameter: 7B x 0.5 = 3.5 GB. That is a 4x memory reduction -- enough to run a 7B model on a consumer GPU with 4 GB of VRAM.
Round-to-nearest (RTN): Simply round each weight to the nearest quantized value.
Fast, simple, no calibration data needed
Works well for INT8, degrades significantly at INT4
Baseline method, rarely used in production
GPTQ (GPT Quantization)
Uses a small calibration dataset (~128 samples) to minimize quantization error
Quantizes one layer at a time, compensating for errors in subsequent layers
State-of-the-art for INT4 quantization of LLMs
One-time cost: ~1-4 hours for a 70B model
Result: near-FP16 quality at 4-bit precision
AWQ (Activation-Aware Weight Quantization)
Observes that ~1% of weights are disproportionately important (based on activation magnitudes)
Protects those critical weights while aggressively quantizing the rest
Slightly better quality than GPTQ at the same bit width
Faster quantization process than GPTQ
GGUF (GPT-Generated Unified Format)
Quantization format optimized for CPU inference (llama.cpp)
Multiple quantization levels: Q2_K through Q8_0
Supports mixed precision: important layers at higher precision
Popular for local/edge deployment
Try it! Open a Python REPL and try quantizing a number yourself: original = 0.0023; quantized = round(original * 127) / 127; print(f"Original: {original}, Quantized: {quantized}, Error: {abs(original - quantized):.6f}"). Notice how the error is tiny. Now imagine doing this for 7 billion parameters -- the total error stays small because the errors are random and tend to cancel out.
pythonplayground.py · Pyodide
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
Tests · Compare global vs per-channel quantization error at 8-bit and 4-bit. Per-channel should always have lower error.
Quantization-Aware Training (QAT)
Simulate quantization during training so the model learns to be robust to low precision:
Insert "fake quantization" nodes that round weights during forward pass
Gradients flow through using straight-through estimator
Higher quality than PTQ, especially at very low bit widths (2-3 bits)
Cost: Full training run required. For a 70B model, this is expensive.
Used by: Meta for official Llama quantized checkpoints
The frontier moved past simple INT8/INT4 in the last two years. New floating-point formats hit hardware:
Format
Bits
Hardware
Notes
FP8 (E4M3 / E5M2)
8
H100, H200, MI300X
Floating-point with exponent — preserves dynamic range better than INT8. Used for both training and inference at Meta, NVIDIA.
MXFP4 / MXFP6 (OCP Microscaling)
4 / 6
Blackwell B100/B200
Block-scaled floating-point. ~2x faster than FP16 with quality close to BF16. The de-facto inference format on Blackwell.
NF4 (Normal Float 4)
4
Any GPU via bitsandbytes
Information-theoretically optimal for normally distributed weights. Foundation of QLoRA.
INT4 GPTQ / AWQ
4
Any GPU
PTQ algorithms. AWQ preserves activation outliers; widely used by vLLM, TensorRT-LLM.
GGUF Q4_K_M
~4.5
CPU / Apple Silicon / GPU
Mixed-precision blocks. The llama.cpp ecosystem's default.
INT4-LLM (SmoothQuant + GPTQ)
4
Any GPU
Smooth activations into weights before quantization to recover lost accuracy.
KV-cache quantization is the underrated companion to weight quantization. KV cache often dominates GPU memory in long-context serving. INT8 KV cache halves cache memory at near-zero quality cost; FP8 is now standard in TensorRT-LLM and vLLM v0.6+.
#The Compression Pipeline: From Full Model to Edge Deployment
Here is the end-to-end journey of compressing a large model for efficient deployment. Each step reduces size and increases speed, with carefully measured quality trade-offs:
You start with a fully trained model in FP32 (32-bit floating point). A 1B parameter model at 4 bytes per parameter requires 4 GB just for weights. This is the highest-quality version -- the "master copy" that every compressed variant derives from.
Convert each weight from 32-bit float to 8-bit integer using a calibration dataset (128 samples is enough). This 4x memory reduction -- from 4 GB to 1 GB -- typically costs less than 0.5% accuracy. The calibration step finds optimal scale factors per layer to minimize quantization error.
Identify weights close to zero (contributing little to the output) and set them to exactly zero. Structured pruning removes entire neurons or attention heads for immediate speedup on standard hardware. At 50% sparsity, you remove half the computation with typically 1-2% quality loss.
4
#Step 4: Knowledge Distillation -- Train a Small Student
Train a much smaller model (the "student") to mimic the larger model's (the "teacher's") full probability distributions. The student learns from the teacher's soft labels, which contain richer information than hard labels alone. A 3B student can achieve 90%+ of a 70B teacher's quality.
After combining quantization, pruning, and distillation, the original 4 GB model is now 500 MB -- an 8x reduction. Benchmark on your specific evaluation set to verify quality is acceptable. Different tasks tolerate different compression levels.
Convert the compressed model to a platform-specific format: GGUF for CPU (llama.cpp), ONNX for cross-platform, Core ML for Apple devices, or TFLite for Android. The compressed model runs on a smartphone with 4 GB RAM, a laptop without a GPU, or an IoT device at the edge -- bringing AI capabilities to devices that could never run the original model.
Set individual weights to zero based on magnitude. A weight close to zero contributes little.
Lottery Ticket Hypothesis (2019): Dense neural networks contain sparse subnetworks ("winning tickets") that, when trained in isolation from the same initialization, match the full network's performance. This suggests most weights are not needed -- if only we could find the right ones to keep.
Can achieve 90%+ sparsity (90% of weights are zero)
Requires sparse matrix libraries for actual speedup (hardware support is improving)
Works well for smaller models, less explored for LLMs
Quantization reduces model size with minimal quality loss. Converting from FP16 to INT8 halves memory and doubles throughput; INT4 quarters memory with typically less than 2% quality degradation for LLMs
Different quantization methods suit different scenarios. GPTQ for offline batch processing, AWQ for accuracy-sensitive serving, GGUF for CPU/edge deployment, and bitsandbytes for quick experimentation
Pruning removes unimportant weights. Structured pruning removes entire neurons or attention heads, while unstructured pruning zeros out individual weights; both reduce compute at the cost of some accuracy
Knowledge distillation trains a small model to mimic a large one. The student learns from the teacher's soft probability distributions (which contain richer information than hard labels), often achieving 90%+ of the teacher's quality at a fraction of the size
With compression techniques mastered, you can run powerful models on modest hardware. Next up: Evaluation & Testing at Scale -- how to systematically measure model quality beyond simple accuracy metrics.