GPT predicts one word at a time. That's it. The "magic" is doing it across hundreds of billions of parameters, trillions of training tokens, and 96 layers of stacked transformer blocks. Here's how an architecture that simple — the same one in GPT-4o, Claude Sonnet 4.6, Llama 3.3, and DeepSeek-V3 — became smart enough to write production code.
Learning Objectives
After this lesson, you will be able to:
Understand causal masking: why the model can only look backward, never forward
Follow the GPT family tree from GPT-1 through GPT-4
See why "just predict the next word" turns out to be shockingly powerful at scale
Learn how temperature, top-k, and top-p let you control AI creativity vs. precision
Understand how instruction tuning turns a raw text predictor into a helpful chatbot
See why decoder-only models won the architecture race
The defining feature of a decoder model is causal masking (also called the causal attention mask). During self-attention, each token can only attend to tokens at its position and earlier -- never to future tokens.
For a sequence of length n, create an n x n matrix. The lower triangle (including the diagonal) is 0: these are allowed attention connections. The upper triangle is negative infinity: these are forbidden future positions. Token 3 can attend to tokens 1, 2, and 3 -- but not tokens 4, 5, or beyond.
After computing QK^T (the raw attention scores), add the mask matrix element-wise. Allowed positions keep their scores unchanged. Forbidden positions get pushed to negative infinity. This is computationally cheap -- just an element-wise addition.
When softmax encounters negative infinity, it outputs exactly zero. So future positions receive zero attention weight -- the model physically cannot extract information from them. The attention distribution is computed only over past and current positions.
During generation, the model processes the entire sequence so far, predicts a probability distribution over the vocabulary for the next position, samples a token, appends it to the sequence, and repeats. Each new token can attend to all previous tokens but nothing beyond.
What Do You Think?
During TRAINING, does a decoder model generate tokens one at a time (slowly) like during inference?
Try it! Open your phone keyboard and start typing "The president of the United" -- what does autocomplete suggest? Now imagine that same autocomplete but trained on every book, website, and conversation ever written. That is GPT. The gap between your phone's suggestions and GPT is just scale.
This is a crucial efficiency insight: during training, the causal mask lets the model process an entire sequence in one forward pass while still respecting the left-to-right constraint. Every position simultaneously predicts its next token. A 1000-token training example provides 999 prediction targets in a single pass. Only during inference (generation) does the model work one token at a time.
Start with the prompt: "Once upon". This is the seed text that the model will continue. The prompt could be a single word, a sentence, or thousands of tokens of conversation history -- the model treats it all as context for predicting what comes next.
The tokenizer (byte-level BPE) converts "Once upon" into token IDs: [7454, 2402]. Each token ID indexes into the embedding table to retrieve a dense vector. Positional encodings (RoPE in modern models) are applied so the model knows the order. The result: two embedding vectors ready for the Transformer.
The tokens pass through all Transformer layers with causal masking. Token "Once" can only attend to itself. Token "upon" can attend to "Once" and itself -- but never to future tokens that have not been generated yet. The upper triangle of the attention matrix is set to negative infinity, producing zero attention weights for future positions.
The training objective could not be simpler: given all previous tokens, predict the next one.
L=−t=1∑TlogP(xt∣x1,x2,…,xt−1)
Try it: Transformer Architecture FlowInteractive
Loading visualization...
Explore this: Step through the transformer layer by layer — watch token embeddings get enriched at each layer. Early layers capture syntax (word endings, punctuation patterns), middle layers capture semantics (subject-verb agreement, entity types), and late layers capture pragmatics (what the text means in context). This hierarchical enrichment is what gives transformers their power.
Try it: Watch GPT generate one token at a timeInteractive
Loading visualization...
Explore this: Watch the model generate text one token at a time, outputting a full probability distribution over ~50,000 vocabulary tokens at each step. Adjust the temperature: at 0.1 it picks the same token every time (deterministic but repetitive); at 2.0 it goes off-script (creative but potentially incoherent). The sweet spot for most tasks is 0.7–1.0.
⚡ Playground:Transformer → — step through the full transformer block layer by layer and watch information flow.
When the model produces a probability distribution over the vocabulary, how do we choose the next token? Different strategies produce dramatically different outputs.
Only consider the top k most likely tokens and redistribute probability among them. If k = 50, the model picks from the 50 most likely next tokens, ignoring all others. This prevents the model from ever selecting extremely unlikely tokens while maintaining diversity.
When the model is very confident about the next token (one token has 95% probability), which sampling strategy changes behavior the LEAST?
In practice, most production systems use a combination: temperature + top-p. Claude, GPT-4, and other chat models typically default to temperature around 0.7-1.0 with top-p around 0.9-0.95. Coding tasks benefit from lower temperature (0.0-0.3), while creative writing benefits from higher temperature (0.8-1.2).
Switch between strategies and observe which tokens each one considers (colored) vs. rejects (dimmed). Notice how greedy always picks one token, top-k keeps a fixed number, and top-p adapts to the model's confidence. The selected token pulses to show the final choice.
#Instruction Tuning: From Autocomplete to Assistant
A raw pretrained GPT model is an autocomplete engine -- it predicts what text comes next. Ask it a question and it might generate another question, or continue as if it were a web page. It does not "follow instructions" by default.
Train on trillions of tokens from the internet. The model learns language, facts, reasoning patterns, and code. But it also learns to mimic web text -- which includes spam, nonsense, and toxic content. The base model is powerful but uncontrolled. Ask "What is 2+2?" and it might respond "What is 3+3?" because that is what follows on a math worksheet.
Train on high-quality (instruction, response) pairs written by human annotators. "Explain quantum computing in simple terms" paired with a clear, helpful explanation. This teaches the model the format of helpful interaction. Thousands to millions of examples shift the model's behavior from "predict web text" to "follow instructions helpfully."
Reinforcement Learning from Human Feedback: humans rank multiple model responses from best to worst. A reward model learns these preferences. Then the language model is fine-tuned to maximize the reward model's score using PPO (proximal policy optimization). DPO (Direct Preference Optimization) is a simpler alternative that skips the reward model and trains directly on preference pairs.
Autoregressive generation is fundamentally sequential: GPT generates one token at a time, feeding each output back as input — this is why inference cannot be parallelized across tokens the way training is
Causal masking enables parallel training: by masking future positions, GPT trains on all sequence positions simultaneously, predicting the next token for every prefix in one forward pass
Temperature and top-p control the creativity/coherence tradeoff: temperature=0 gives deterministic greedy decoding; temperature>1 increases randomness; top-p=0.9 restricts sampling to the top 90% probability mass
Instruction tuning converts a text predictor into an assistant: SFT + RLHF teaches the model to follow instructions rather than just complete the next likely token — this is what separates GPT-base from ChatGPT
Chinchilla showed compute efficiency over raw scale: the optimal ratio is ~20 training tokens per parameter; "over-training" smaller models on more data gives better inference efficiency at the same quality level
What does the causal mask do in a decoder-only Transformer?
We have seen how decoders generate text token by token. But the quality of the output depends enormously on what you put IN. Next up: the art and science of prompt engineering -- how to talk to these models to get the best results.
The final layer outputs a vector of logits with one value per vocabulary token (e.g., 100,256 values for GPT-4's tokenizer). Higher logits mean higher predicted probability. The logit for "a" might be 8.2, for "time" it might be 3.1, for "the" it might be 2.5. These raw scores are not yet probabilities.
The logits are divided by temperature (controlling randomness), filtered by top-k (keep only the k most likely tokens) or top-p (keep tokens whose cumulative probability exceeds p), and then softmax converts them to probabilities. A token is randomly sampled from this distribution. With temperature 0.7 and top-p 0.9, "a" gets selected with high probability.
The sampled token "a" is appended to the sequence: "Once upon a". Its token ID is added to the input, and the KV-cache stores the key and value vectors from the previous forward pass so they do not need to be recomputed. The sequence grows by one token.
Go back to Step 3 with the extended sequence. Now "a" attends to "Once", "upon", and itself. The model predicts the next token -- "time" is likely. Append "time", repeat. "Once upon a time" becomes "Once upon a time," then "Once upon a time, there" and so on until the model outputs an end-of-sequence token, hits the max length, or a stop sequence is reached.
The aligned model follows instructions, refuses harmful requests, admits uncertainty, and formats responses helpfully. Same architecture, same parameters, same attention mechanism -- but dramatically different behavior. This is the difference between GPT-3 (base) and ChatGPT, between LLaMA (base) and LLaMA-Chat.