Model Compression & Efficiency

#The Impossible Size of Modern Models

If you have ever wanted to run a powerful AI model on your own computer (or wondered how AI works on your phone), this lesson shows you exactly how it is done. Compression is one of the most practically useful skills in ML engineering, and one of the most satisfying to see in action.

#The Three Pillars of Compression

Each technique can be used independently or combined. In practice, quantization is the most impactful and widely used for LLMs.

#Quantization: Fewer Bits, Same Brain

#Number Representation Primer

FP16 uses 2 bytes per parameter: 7B x 2 = 14 GB. INT4 uses 0.5 bytes per parameter: 7B x 0.5 = 3.5 GB. That is a 4x memory reduction — enough to run a 7B model on a consumer GPU with 4 GB of VRAM.

#Quantization Methods

Post-Training Quantization (PTQ)

Quantize a pre-trained model without retraining:

• Fast, simple, no calibration data needed
• Works well for INT8, degrades significantly at INT4
• Baseline method, rarely used in production

GPTQ (GPT Quantization)

• Uses a small calibration dataset (~128 samples) to minimize quantization error
• Quantizes one layer at a time, compensating for errors in subsequent layers
• State-of-the-art for INT4 quantization of LLMs
• One-time cost: ~1-4 hours for a 70B model
• Result: near-FP16 quality at 4-bit precision

AWQ (Activation-Aware Weight Quantization)

• Observes that ~1% of weights are disproportionately important (based on activation magnitudes)
• Protects those critical weights while aggressively quantizing the rest
• Slightly better quality than GPTQ at the same bit width
• Faster quantization process than GPTQ

GGUF (GPT-Generated Unified Format)

• Quantization format optimized for CPU inference (llama.cpp)
• Multiple quantization levels: Q2_K through Q8_0
• Supports mixed precision: important layers at higher precision
• Popular for local/edge deployment

Try it! Open a Python REPL and try quantizing a number yourself: original = 0.0023; quantized = round(original * 127) / 127; print(f"Original: {original}, Quantized: {quantized}, Error: {abs(original - quantized):.6f}"). Notice how the error is tiny. Now imagine doing this for 7 billion parameters — the total error stays small because the errors are random and tend to cancel out.

Quantization-Aware Training (QAT)

Simulate quantization during training so the model learns to be robust to low precision:

• Insert "fake quantization" nodes that round weights during forward pass
• Gradients flow through using straight-through estimator
• Higher quality than PTQ, especially at very low bit widths (2-3 bits)
• Cost: Full training run required. For a 70B model, this is expensive.
• Used by: Meta for official Llama quantized checkpoints

#Modern Quantization Landscape (2024-2026)

The frontier moved past simple INT8/INT4 in the last two years. New floating-point formats hit hardware:

#Memory Math: What Fits Where?

The number you actually need at hardware-selection time is "can this checkpoint + KV cache fit?". Work it out.

#Quantization Decision Guide

#The Compression Pipeline: From Full Model to Edge Deployment

Here is the end-to-end journey of compressing a large model for efficient deployment. Each step reduces size and increases speed, with carefully measured quality trade-offs:

#Pruning: Removing What Does Not Matter

Pruning removes weights or structures from a neural network:

#Unstructured Pruning

Set individual weights to zero based on magnitude. A weight close to zero contributes little.

• Can achieve 90%+ sparsity (90% of weights are zero)
• Requires sparse matrix libraries for actual speedup (hardware support is improving)
• Works well for smaller models, less explored for LLMs

#Structured Pruning

Remove entire neurons, attention heads, or layers:

• Attention head pruning: remove heads that contribute least to output
• Layer pruning: remove entire transformer layers (surprisingly effective — removing 25% of layers often loses under 5% quality)
• Width pruning: reduce hidden dimension

#Distillation: Teaching a Small Model

#Distillation for LLMs

• Standard distillation: Train a smaller model on the larger model's outputs
• On-policy distillation: Student generates its own outputs, teacher provides scores
• Synthetic data generation: Use the teacher to generate training data for the student

Real-world examples

• GPT-4 -> GPT-4o mini (rumored distillation)
• Llama 70B -> Llama 8B (Meta's approach combines distillation with architecture search)
• Gemma 27B -> Gemma 2B

#Key Takeaways

• Quantization reduces model size with minimal quality loss. Converting from FP16 to INT8 halves memory and doubles throughput; INT4 quarters memory with typically less than 2% quality degradation for LLMs
• Different quantization methods suit different scenarios. GPTQ for offline batch processing, AWQ for accuracy-sensitive serving, GGUF for CPU/edge deployment, and bitsandbytes for quick experimentation
• Pruning removes unimportant weights. Structured pruning removes entire neurons or attention heads, while unstructured pruning zeros out individual weights; both reduce compute at the cost of some accuracy
• Knowledge distillation trains a small model to mimic a large one. The student learns from the teacher's soft probability distributions (which contain richer information than hard labels), often achieving 90%+ of the teacher's quality at a fraction of the size

#Quick Check

#Exit Ticket