One attention head learns one pattern. Multi-head attention runs 32, 64, sometimes 128 heads in parallel — each free to specialize in syntax, coreference, factual recall, or something nobody has named yet. GPT-4 uses ~96 heads per layer. Claude Sonnet 4.6 and Llama 3.3 use grouped-query variants that share keys/values across heads to save memory at inference. The jump in capability is what made the transformer eat the world.
Learning Objectives
After this lesson, you will be able to:
Understand why one attention head is not enough -- and how multiple heads fix this
See how each head focuses on a different type of pattern (grammar, meaning, position, etc.)
Grasp the dimension budget: the total size stays the same, you just slice it into more pieces
Follow how head outputs get combined back into a single representation
Compare MHA, MQA, and GQA -- the efficiency tricks that make real LLMs affordable
Reason about the tradeoff: more heads vs. larger heads
A single attention head computes one set of attention weights. It can capture one type of pattern -- maybe positional proximity, or maybe subject-verb agreement. But language (and data in general) has many simultaneous relationships happening at once. The word "it" in "The animal didn't cross the street because it was too tired" needs to track coreference (it = animal), syntax (it = subject), and semantics (tired = animate) all at the same time. One head is not enough.
Try it! Read the sentence: "She told him that she would meet him there after she finished." How many different relationships can you spot? Pronouns referring back to people, time sequencing ("after"), spatial reference ("there"). Your brain tracks all of these simultaneously -- that is what multi-head attention does for the model.
Take the model dimension d_model (e.g., 512) and split it across h heads (e.g., 8). Each head works with d_k = d_model / h = 64 dimensions. This is the dimension budget -- no extra computation, just slicing the existing space differently.
Each head has its own W_Q, W_K, W_V weight matrices. Head 1 projects input into its own 64-dimensional subspace. Head 2 projects into a different 64-dimensional subspace. The projections are learned independently during training, so each head can specialize.
Run scaled dot-product attention independently on each head, in parallel. Head 1 computes its own attention pattern. Head 2 computes a different pattern. All 8 heads execute simultaneously on the GPU -- this is embarrassingly parallel.
Take the output from all heads (each of shape [seq_len x 64]) and concatenate them along the last dimension. Result: [seq_len x 512]. We are back to the full model dimension, but now each token carries information from 8 different attention perspectives.
In practice nobody implements the head loop explicitly. Production transformer code expresses the QKV projection and the attention matmul as a single einsum contraction over the head, sequence, and feature axes — see tensor notation and einsum for the index-by-index walkthrough that turns bhqd, bhkd -> bhqk into the dot-product scores you saw above.
Here is the key constraint that keeps multi-head attention efficient:
dmodel=h×dk
The total parameter count stays roughly the same regardless of how many heads you use (ignoring the output projection). With 8 heads of dimension 64 or 16 heads of dimension 32, you are projecting the same 512-dimensional input. You are just slicing it differently.
More heads means each head works in a lower-dimensional subspace. Fewer heads means each head has more dimensions to work with. The tradeoff: more heads give you more diverse attention patterns, but each head has less capacity to model complex relationships within its subspace.
What Do You Think?
If we doubled the number of heads from 8 to 16 but halved each head's dimension from 64 to 32, what happens to the total computation?
Research has shown that different heads naturally specialize during training, without being explicitly told to. In BERT-like models, researchers have found:
Positional heads: Attend primarily to adjacent tokens (previous or next word). These capture local context -- bigram-like patterns.
Syntactic heads: Track subject-verb dependencies, even across long distances. "The keys to the cabinet are" -- despite "cabinet" being closer, the head attends to "keys" for subject-verb agreement.
Separator heads: Focus on special tokens like [SEP] or punctuation. These act as "reset" signals that mark boundaries between segments.
Rare token heads: Activate strongly for infrequent vocabulary items, possibly routing them through specialized processing.
Coreference heads: Link pronouns to their antecedents ("she" to "Marie").
This specialization emerges purely from training -- no one programs the heads to do this. The model discovers that having diverse attention strategies produces better representations.
Try different sentences and switch between attention heads using the dropdown. Notice how each head produces a different attention pattern -- this is multi-head attention in action. Head 1 might focus on adjacent words while Head 3 might capture longer-range dependencies.
Experiments to try:
Type "The cat that the dog chased ran away" -- nested clauses stress-test heads
Try "She gave him the book and he read it" -- watch for coreference patterns
Switch between heads on the same sentence -- see the diversity of attention patterns
After concatenation, the output passes through one more linear projection W^O. This step is often overlooked but serves a critical purpose: it lets the model mix information across heads.
Without W^O, each head's contribution to the final output would be completely independent -- limited to its own d_k-dimensional subspace. The output projection allows cross-head interaction, enabling richer representations than any single head could produce.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin (2017)
Section 3.2.2 introduces multi-head attention. The paper's key insight is that projecting to multiple lower-dimensional subspaces is more effective than a single full-dimensional attention function.
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yinfei Yang, Sumit Sanghai, Santiago Ontanon (2023)
Introduces Grouped-Query Attention, the compromise between MHA and MQA that is now standard in modern LLMs like LLaMA 2/3 and Mistral.
Multiple heads let the model attend to different relationship types simultaneously. One head might capture syntactic dependencies (subject-verb), another semantic similarity, and another positional proximity
Heads operate in parallel on different subspaces. The model dimension is split evenly across heads (d_model = num_heads x d_k), so multi-head attention costs the same as single-head attention with full dimensionality
Concatenation and output projection combine all heads. After each head computes its attention independently, results are concatenated and projected through a learned matrix to produce the final output
GQA and MQA reduce memory during inference. By sharing Key/Value projections across groups of heads, Grouped-Query Attention and Multi-Query Attention dramatically reduce the KV cache size for serving large language models
Why does multi-head attention use multiple heads instead of one large attention computation?
Multi-head attention can capture what relates to what, but it has no idea about order. "Dog bites man" and "man bites dog" look identical. Next up: Positional Encoding -- giving the Transformer a sense of sequence.
Multiply by a final weight matrix W_O of shape [512 x 512]. This is the critical step that mixes information across heads. Without it, each head's contribution would remain isolated in its own 64-dimensional subspace. W_O lets the model combine insights from all heads into a unified representation.