GPT-4 is rumored to be ~1.8T parameters total but only ~280B active per token. DeepSeek-V3 is 671B parameters total, 37B active. Mixtral 8x7B beats a dense 70B with one-third the compute. The trick that makes all of this possible is mixture of experts — and once you see the routing math, you will understand why every frontier model from 2024 onward is sparse, not dense.
Learning Objectives
After this lesson, you will be able to:
Explain the core MoE trick — replace a dense FFN with a bank of E specialist experts plus a router that picks the top-k for each token — so total parameters can grow far faster than active compute per token
Walk through the routing math — softmax over expert logits, top-k selection, weighted combination — and code a simple MoE block from scratch in PyTorch
Diagnose the load-balancing failure mode (one expert hogs every token) and apply the auxiliary load-balancing loss that fixes it
Pick MoE vs dense based on your real constraint — training FLOPs budget, GPU memory, inference batch size — and recognize when MoE's communication overhead actually loses you money
Don't worry if "mixture of experts" sounds exotic — once you see the routing as a simple softmax-pick-top-k, the rest is plumbing. The hard parts are infrastructure (load balancing, communication) not math.
Every machine learning architecture before MoE assumed a fixed compute budget per token, and the only way to spend more was to make the dense matrix bigger. MoE breaks that link. Total parameters and per-token compute become two separate dials.
The router is a tiny linear layer plus softmax. For each token's hidden state x, it produces a probability over experts; the top-k experts run; their outputs are combined weighted by those probabilities.
P(e∣x)=softmax(Wrx)e∈RE
y=e∈TopK(P(⋅∣x))∑P(e∣x)⋅FFNe(x)
The router is trained jointly with the experts via the same backprop signal that flows through the rest of the network. There are no expert-specialization labels — each expert discovers its niche through gradient descent.
Top-1 (Switch Transformer): each token routes to exactly one expert. Simpler, cheaper, slightly worse quality. Good for very large E.
Top-2 (GShard, Mixtral): each token routes to two experts; their outputs are blended. More compute per token, generally higher quality. The default in modern open-source MoE.
Top-k for k > 2 is rare — diminishing returns and growing complexity.
Without intervention, MoE training collapses. Early in training, one or two experts get slightly better at predicting common patterns. The router learns this and routes more tokens their way. Those experts get more gradient updates and improve faster. Within a few thousand steps, one expert handles 90% of tokens and the rest sit idle, getting almost no learning signal.
This is the collapse mode of MoE, and the standard fix is an auxiliary loss that nudges the router toward balanced expert utilization.
Laux=α⋅E⋅e=1∑Efe⋅pe
This single auxiliary term prevents the collapse and makes MoE trainable in practice. Without it, you get extremely unstable training and one-expert-takes-all behavior. With it (and proper gradient clipping), MoE trains nearly as smoothly as dense models.
Mixtral 8x7B has 8 experts and uses top-2 routing per token. Roughly what fraction of total FFN parameters is active per forward pass?
Top-2 of 8 means 2/8 = 25% of FFN parameters are active per token. The router itself is tiny (a single linear layer producing 8 logits) so it does not change the picture. Mixtral's reported "13B active out of 47B" reflects this 1/4 ratio in the FFN layers plus the shared attention/embedding components that always run.
Tests · Verify the MoE block produces output of the same shape as input, that experts have the right number of params, and that top-k routing actually selects the right number of experts per token.
Training compute budget is the hard wall, want to maximize quality
MoE
GPU memory is the hard wall (fits dense in VRAM but not MoE)
Dense
Inference latency at batch=1 is critical
Dense
Inference throughput at high batch is critical
MoE with expert parallelism
Model needs to specialize across very different domains (code + math + dialogue)
MoE (experts can specialize)
You want max quality-per-active-parameter (model card looks good for inference cost)
MoE
You want max quality-per-total-parameter (memory-constrained edge)
Dense
The frontier-lab math: at fixed training-compute budget, MoE consistently beats dense on quality benchmarks. That is why every 2024-2025 frontier release (GPT-4, Gemini 1.5 Pro, Mixtral, DeepSeek-V3) chose MoE. The cost is operational complexity at serving time.
MoE decouples total parameters from active compute. Replace each FFN with E experts plus a top-k router. Total params grow with E; per-token compute grows with k. The k/E ratio is the lever that gives you more knowledge at the same FLOPs budget.
The router is a tiny softmax-over-experts plus top-k selection. Trained jointly with the rest of the network via standard backprop. No expert-specialization labels needed; experts discover their niches through gradient descent.
Load balancing is the central training problem. Without an auxiliary loss, one expert hogs every token and the rest atrophy. The Switch Transformer aux loss (α·E·Σf_e·p_e with α≈0.01) is the standard fix and makes MoE trainable in practice.
MoE wins at training-FLOPs budget; dense often wins at inference latency. Frontier labs use MoE because they are training-compute-constrained. Single-user edge serving still favors dense models because MoE's all-to-all communication overhead doesn't amortize at batch=1.
Modern MoE stacks shared experts plus fine-grained routed experts. DeepSeek-V3 uses 256 routed experts plus a few always-active shared experts, with top-6 or top-8 routing. The architecture trick that lets a 671B-total / 37B-active model match GPT-4-class behavior on academic-budget compute.
What is the primary purpose of the auxiliary load-balancing loss in an MoE training run?
MoE turns "scale total params" and "scale per-token compute" into two separate dials. Next: state space models like Mamba — a completely different bet on sequence modeling that drops attention entirely in favor of linear-time recurrence with selective updates.