The 2017 paper "Attention Is All You Need" replaced a decade of NLP research with one architecture. ChatGPT, Claude Sonnet 4.6, Gemini 2 Pro, every modern LLM — same shape, same math, just scaled differently. Learn it once, understand all of them.
Learning Objectives
After this lesson, you will be able to:
Follow data from raw text through the entire Transformer, step by step
Understand residual connections (information highways) and why they prevent vanishing gradients in deep stacks
Distinguish pre-norm from post-norm layer normalization and know why modern LLMs switched to pre-norm
See the feed-forward network as an expand-then-compress thinking step that holds two-thirds of the parameters per layer
Know the three flavors of Transformers: encoder-only (BERT, bidirectional), decoder-only (GPT, causal), and encoder-decoder (T5)
Understand the encoder-decoder cross-attention mechanism: how the decoder queries the encoder's representations
Reason about why decoder-only models won the architecture race -- and when encoder-only models still dominate
We have built up the pieces across four lessons: attention, self-attention, multi-head attention, and positional encoding. Now we assemble them into the complete architecture that changed the world.
Attention Is All You Need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin (2017)
The paper that introduced the Transformer. Section 3 describes the full architecture. Pay special attention to Figure 1 -- one of the most reproduced diagrams in ML history.
Before anything enters the Transformer, raw text must be converted to tokens. This is often overlooked but is a critical first step.
Loading visualization...
Try it! Type your name into the tokenizer above and see how it gets split. Is your name one token or several? Now try a technical word like "backpropagation" -- watch how BPE handles it.
Type a sentence above and watch BPE (Byte-Pair Encoding) merge characters step by step. Modern LLMs use BPE variants (SentencePiece, tiktoken) to split text into subword tokens. The vocabulary typically contains 32K-128K tokens. Common words become single tokens ("the", "and"), while rare words get split into subword pieces ("un" + "familiar" + "ity").
Each input token (a word or subword) is converted to a dense vector using a learned embedding table. A vocabulary of 50,000 tokens with d_model = 512 means a 50,000 x 512 embedding matrix -- about 25 million parameters just for the embedding layer. The token "cat" maps to token ID 8765, which indexes into row 8765 of the embedding matrix to retrieve a 512-dimensional vector.
The positional encoding vector is added to each token embedding. Now each vector carries both what the token is and where it sits in the sequence. Without this, "dog bites man" and "man bites dog" would be indistinguishable. Modern models use RoPE instead of sinusoidal encoding, but the principle is the same.
Every token attends to every other token (and itself). Each of h heads computes its own attention pattern in parallel. The outputs are concatenated and projected through W_O. Each token's representation is now enriched with context from the entire sequence. This is where "it" learns to look at "animal" and "tired" in our earlier example.
Try it: Walk Through the Full TransformerInteractive
Loading visualization...
Click through each component to see what happens at every step of the pipeline. Follow a token from raw input through embedding, attention, FFN, and out to prediction.
Every sub-layer (attention and FFN) has a residual connection -- the input is added directly to the output:
output=LayerNorm(x+Sublayer(x))
Without residual connections, training deep Transformers (32+ layers) would be nearly impossible. Gradients would vanish as they flow backward through dozens of layers. The residual connection gives the gradient a direct path back to earlier layers -- an "information highway" that keeps learning alive.
Layer norm normalizes each token's activation vector to have zero mean and unit variance, then applies a learned scale and shift:
LN(x)=γ⊙σ2+ϵx−μ+β
Why does this matter? Without normalization, activations in deep networks tend to drift -- growing or shrinking as they pass through layers. This makes training unstable and slow. Layer norm acts as a stabilizer, ensuring each layer receives inputs in a predictable range. Modern alternatives include RMSNorm (used in LLaMA), which skips the mean-centering step for efficiency.
The FFN expands from 512 to 2048 dimensions and then contracts back to 512. Why expand at all if we end up at the same size?
The FFN applies the same two-layer network to every token independently:
FFN(x)=GELU(xW1+b1)W2+b2
The expansion to 4x the model dimension creates a higher-dimensional "thinking space." In this expanded space, the network can represent more complex, nonlinear functions -- combining and transforming the contextual information gathered by attention. The contraction back to d_model ensures the output can be fed to the next layer.
The original Transformer has both an encoder and a decoder. But the field has diverged into three distinct architectures, each optimized for different tasks.
Every token can attend to every other token -- past and future. The word "it" can look both backward (at "animal") and forward (at "was tired") simultaneously. This makes the encoder ideal for tasks requiring understanding the full context.
Models: BERT, RoBERTa, DeBERTa, sentence-transformers
Best for: Classification, named entity recognition, semantic similarity, retrieval, embeddings
Tokens can only attend to earlier positions. When generating token 5, the model can see tokens 1-4 but not tokens 6 onward -- because those have not been generated yet. This is enforced by causal masking: future positions in the attention score matrix are set to negative infinity before softmax, producing zero attention weights.
Models: GPT series, LLaMA, Claude, Mistral, DeepSeek
Best for: Text generation, chatbots, code generation, reasoning -- any task that can be framed as "predict the next token"
The encoder processes the full input bidirectionally. The decoder generates the output autoregressively while cross-attending to the encoder's representations. The decoder has both self-attention (causal) and cross-attention (to encoder).
Cross-attention is the bridge between encoder and decoder. At each decoder layer, the decoder's queries (Q) come from the decoder's own representations, but the keys (K) and values (V) come from the encoder's final output. This lets every decoder token look at any position in the source sequence — the original attention mechanism from the Bahdanau (2014) paper applied inside a transformer block.
Here is the full picture of one Transformer layer, in order:
Multi-Head Self-Attention: Tokens exchange information across the sequence.
Residual Add + Layer Norm: Stabilize and preserve the original input signal.
Feed-Forward Network: Each token processes independently in a higher-dimensional space.
Residual Add + Layer Norm: Stabilize again before the next layer.
Stack N of these layers. Precede them with embeddings + positional encoding. Follow them with a final linear projection (to vocabulary size for language models) and softmax (to get next-token probabilities). That is the entire Transformer.
The original paper: 6 layers, 8 heads, d_model = 512, d_ff = 2048, ~65 million parameters. GPT-3: 96 layers, 96 heads, d_model = 12,288, d_ff = 49,152, 175 billion parameters. LLaMA 3 (405B): 126 layers, 128 heads, d_model = 16,384. Same architecture, scaled up thousands of times.
The Transformer's success comes from three properties that make it uniquely scalable:
Parallelism: Unlike RNNs, all tokens in a sequence are processed simultaneously. This maps perfectly to GPU architecture, which excels at massive parallel computation. Training a Transformer on 1000 tokens takes the same wall-clock time as 10 tokens (ignoring memory).
Uniformity: Every layer has the same architecture. Adding more layers is trivial -- just stack more identical blocks. This makes the architecture easy to scale from 6 layers to 126 layers.
Compositionality: Each layer builds on the previous layer's output. Early layers capture simple patterns; deeper layers compose these into complex representations. This hierarchical composition is what enables reasoning, analogy, and abstraction at scale.
Scaling Laws for Neural Language Models
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei (2020)
Demonstrates that Transformer language model performance follows smooth power laws in model size, dataset size, and compute. This paper showed that scaling is predictable, enabling deliberate decisions about how to allocate compute budgets.
Residual connections and layer normalization enable deep stacking. Residual connections let gradients flow directly through the network, while layer normalization stabilizes training, allowing transformers to scale to hundreds of layers
The feedforward network expands then contracts. Each transformer block contains an FFN that projects to a wider dimension (typically 4x), applies a nonlinearity (GELU), then projects back, acting as a per-token processing step
Encoder-only, decoder-only, and encoder-decoder serve different purposes. BERT (encoder-only) excels at understanding, GPT (decoder-only) excels at generation, and T5 (encoder-decoder) handles sequence-to-sequence tasks
Scaling laws predict performance from compute budget. Larger models trained on more data with more compute systematically improve, following power laws that let researchers predict performance before training
Tokenization is the critical first step. BPE and SentencePiece convert raw text into token sequences that the model processes; tokenizer quality directly impacts model capability, especially for code and non-English languages
What is the purpose of residual connections in the Transformer?
You now understand the complete Transformer architecture -- from raw tokens through tokenization, embedding, positional encoding, multi-head self-attention, feed-forward networks, residual connections, and layer normalization to the final prediction. This is the foundation behind GPT, BERT, T5, LLaMA, Claude, and every modern large language model. The next tracks will build on this foundation to explore how these models are trained, scaled, and deployed in the real world.
The attention output is added to the original input (residual connection) and then normalized (layer norm). The residual connection ensures that information can flow straight through without being mangled by attention -- a highway that prevents gradient death. Layer norm keeps activation magnitudes stable across layers.
Each token passes independently through a two-layer neural network: expand from d_model to 4*d_model (e.g., 512 to 2048), apply an activation function (ReLU, GELU, or SwiGLU), then contract back to d_model. This is the "thinking step" -- where each token processes the context it gathered from attention. Critically, the FFN applies to each token independently with shared weights.
Again: add the FFN output back to its input (residual connection) and normalize. This completes one Transformer block (also called a layer). The output has the same shape as the input: [seq_len x d_model].
Stack N identical blocks (the original paper used N = 6). Each block refines the representations further. Early layers tend to capture local syntax; middle layers capture semantics and entity relationships; later layers handle abstract reasoning and task-specific patterns. Modern LLMs use 32-96+ layers.
The final layer's output passes through a linear projection from d_model to vocabulary size (e.g., 512 to 50,000), producing logits for each vocabulary token. Softmax converts these to probabilities. The highest-probability token is the model's prediction for the next token. For generation, this token is appended to the input and the process repeats.