Track 05 · NLP & Transformers · 14 min
The architecture that ate language.
One paper in 2017 changed everything. The transformer is the engine behind every modern LLM, every chat AI, every code assistant. Here's how it actually works — animated through six interactive demos that make attention click.
“Attention is all you need.”
#The hook
Five years later, ChatGPT broke into a hundred million users in two months. Every part of the modern AI economy — Claude, Gemini, GPT-5, Llama, DeepSeek, and the agents using them — sits on top of the architecture that paper introduced. The transformer.
#Why this matters in 2026 — the receipts
Transformers by the numbers
The architecture that won
99%
Frontier LLMs that are transformers
OpenAI, Anthropic, Google, Meta
1T+
Parameters in 2026 frontier models
GPT-5, Claude Opus 4
2M+
Tokens of context (Gemini 3 Pro)
Google 2026
50+
Citations of 'Attention Is All You Need' in 2024
100K+ total — most-cited paper of decade
100K+
Citations of 'Attention Is All You Need'
The most-cited AI paper of the 2010s — and counting. Every modern LLM, every chat AI, every code assistant in 2026 traces directly to this single 2017 publication. Eight authors at Google. Six words in the title. The most lopsided idea-to-impact ratio in modern computer science.
Google Scholar, 2026
Vocabulary
Six transformer terms you'll meet daily
Concept
Token
A subword piece of text. The model's atom.
Like: Lego bricks for language.
e.g. 'preprocessing' → 'pre' + ' processing'
Concept
Embedding
Token → high-dimensional vector capturing meaning.
Like: Coordinates of a word in 'meaning space'.
e.g. 1024-D float vector per token
Concept
Attention head
One of many parallel 'looks' a layer takes at the sequence.
Like: Multiple readers focused on different aspects.
e.g. GPT-4 has ~96 heads per layer
Concept
Context window
How many tokens the model can attend to at once.
Like: How big a desk you can work on.
e.g. Gemini 3 Pro: 2M tokens
Concept
Autoregressive
Predict next token, append, repeat. The decoder pattern.
Like: Writing one word at a time without backspacing.
e.g. Every GPT-style LLM
Concept
RLHF
Reinforcement learning from human feedback — alignment after pretraining.
Like: A child learns from being told 'good' or 'bad'.
e.g. Turned bare GPT-3.5 into ChatGPT
#The four pieces of a transformer
The architecture
Tokenize → Embed → Attention → Predict
1. Tokenization
Words → numbersBreak text into 'tokens' (subword pieces) and assign each one a number.
- BPE (Byte-Pair Encoding) and SentencePiece are the field-standard tokenizers.
- GPT-4 uses ~100K tokens. 'preprocessing' becomes 'pre' + ' processing' + variants.
- Tokenization choices have downstream consequences — emoji, code, non-English all suffer if done wrong.
2. Embedding
Tokens → vectorsEach token becomes a learned high-dimensional vector. Similar meanings end up nearby.
- Position is also encoded — sinusoidal (original), learned, or rotary (RoPE) embeddings.
- These embeddings are *learned* during training, not hand-designed.
- The vector for 'king' minus 'man' plus 'woman' famously lands near 'queen'.
3. Attention
Tokens → contextualized tokensEvery token can 'look at' every other token to update its representation.
- Self-attention: each token computes Query, Key, Value vectors.
- Attention weights = softmax(Q · K^T / sqrt(d_k)). Pay more attention to relevant tokens.
- Multi-head: multiple parallel attention 'looks' for different relationship types.
4. Output (next-token prediction)
Vectors → next tokenAfter many transformer layers, project to vocab probabilities. Sample. Repeat.
- An LLM is just a function: input tokens → probability distribution over next token.
- Generation = sample one token, append to the input, predict next, repeat.
- Temperature, top-k, top-p control how 'creative' the sampling is.
#Step 1 — Tokenization, animated
#Step 2 — Embeddings, the meaning vectors
Position matters. Without explicit position encoding, attention is permutation-invariant — "the dog bit the man" looks identical to "the man bit the dog."
#Step 3 — Attention, finally
The full picture — every layer, every head, every connection:
#Step 4 — Generate
#What's been built on transformers
Transformers in production
What this architecture has actually shipped
Decoder transformer
GPT-4 / GPT-5
1B+
Weekly users
The product that put AI on the global agenda. Pure transformer, RLHF-tuned, scaled to ~1T parameters.
Decoder LLM
Constitutional AI
Claude Opus
200K
Context tokens
Anthropic's frontier model. Same transformer backbone, novel safety training. 200K-token context for full-codebase tasks.
Decoder LLM
Encoder transformer
BERT
1B+
Search queries / day
Powers Google Search ranking since 2019. Encoder-only, masked-language-model objective.
Encoder
Transformer for vision
Vision Transformer (ViT)
90%+
ImageNet top-1
Treats image patches as tokens. Now beats CNNs on classification, detection, segmentation.
ViT
Audio transformer
Whisper
99+
Languages supported
OpenAI's open-weight ASR system. Transformer-based, ~700K hours training audio, runs on a phone.
Audio transformer
Hybrid transformer
AlphaFold 2/3
200M+
Proteins predicted
Transformer-style attention for residue interactions, won the 2024 Nobel.
Bio transformer
#The 2026 frontier
#Where to go next
- NLP & Transformers track — 22 lessons: tokenization, attention, BERT, GPT, RLHF, MoE, Mamba.
- Deep Learning — prerequisites: tensors, autograd, training dynamics.
- Generative AI — diffusion + transformers for image and video.
- AI Agents — what to do with a transformer once you have one.
#Key takeaways
Key Takeaways
- A transformer is: tokenize → embed → many attention+FFN layers → project to vocab.
- Attention is the core innovation — each token can directly look at every other token in parallel.
- Multi-head attention runs many parallel attention 'looks' for different relationship types.
- An LLM is autoregressive: predict next token, sample, append, repeat.
- Transformers eat all sequential data: language, audio, video, vision (via patches), proteins.
- The 2026 frontier: MoE for sparsity, Mamba/SSM for long context, FlashAttention 3 for speed.
#References & further reading
- Vaswani et al. — Attention Is All You Need (NeurIPS 2017). The foundational paper.
- Karpathy — Let's Build GPT From Scratch (YouTube). Most beautiful explanation of transformers ever filmed.
- Lilian Weng — The Transformer Family (lilianweng.github.io). Comprehensive technical survey.
- Anthropic — Mathematical Framework for Transformer Circuits (transformer-circuits.pub). Mechanistic interpretability.
- Distill.pub — Visualizing the Geometry of Attention (interactive paper).