A trained LLM does not produce words — it produces a probability distribution over 100,000 tokens at every step. The decoding strategy is the rule that turns those probabilities into text, and it is the single biggest knob between "helpful chatbot" and "incoherent ramble." Temperature, top-p, beam search, speculative decoding — this is the layer where Claude, GPT-4o, and Llama 3.3 all hand control back to you.
Learning Objectives
After this lesson, you will be able to:
Pick the right decoding strategy — greedy, beam, top-k, top-p, temperature — for translation, creative writing, code, or structured output, and know why each one fails on the wrong task
Tune temperature, top-k, and top-p together without canceling each other out, and avoid the classic 'why is my model producing gibberish' bugs
Explain how speculative decoding gets a 2-3x inference speedup with zero quality loss, and why every major LLM provider uses Medusa, EAGLE, or vLLM-style draft models in production
Use constrained decoding (Outlines, Guidance, JSON mode) to force a model to produce valid JSON, regex-conformant strings, or schema-compliant output without retraining
Don't worry if all the knobs feel overwhelming — once you see them on a real distribution, the temperature and top-p choices become intuitive in about an hour.
A decoder LLM produces a vector of logits z ∈ ℝ^V at every step (one number per vocab token, often V ≈ 30k–100k). Softmax turns those logits into a probability distribution over the vocabulary. Decoding is the rule that turns this distribution into the actual next token.
Maintain b candidate sequences (the beam). At each step, expand each candidate by all V vocab tokens, score the b·V resulting sequences by joint log-probability, keep the top b.
score(x1:t)=i=1∑tlogP(xi∣x1:i−1)
Try it! Open a notebook and run model.generate(..., num_beams=5) on a translation task and the same with num_beams=1 (greedy). For news-headline translation you'll see beam beating greedy by 2-4 BLEU points.
Beam wins when
Translation, summarization, image captioning — tasks with a notion of "correct"
Constrained generation where you'd rather get a high-confidence safe output than a creative one
Beam fails when
Open-ended generation — output is generic, "boring," and statistical-mode-collapsed
Beam sizes >10 — outputs degrade as longer sequences get penalized into vacuous ones
Pure sampling: draw the next token from the distribution P(token | x at positions 1 to t). This explores the full vocabulary including the long tail of low-probability tokens — which is usually where the gibberish lives.
To make sampling usable, we shape the distribution before drawing.
Only sample from the k most-probable tokens (set the rest to zero, renormalize).
Vk=top-ktokens by P;P′(v)=∑v′∈VkP(v′)P(v)⋅1[v∈Vk]
Top-k weakness: at positions where the model is very confident (e.g., predicting "the" after "I went to"), top-k=50 still includes 49 unlikely tokens. At positions where the model is very uncertain, top-k=50 may exclude many plausible tokens. This is what nucleus sampling fixes.
In production you almost always combine them — apply temperature first, then top-p (or top-k).
What Do You Think?
You set temperature=0 and top-p=0.9 simultaneously in the OpenAI API. What controls the generation?
Temperature=0 effectively rounds logits to spike at the argmax (one massive logit, the rest at -∞ after division). Once that happens, top-p sees a distribution with one token at probability 1.0 — which trivially covers ≥ 0.9 of mass — so the nucleus contains exactly that one token. Temperature 0 wins; you get greedy decoding regardless of top-p.
Repetition penalty divides the logits of recently-emitted tokens by a constant > 1, making them less likely to be picked again. Helpful but blunt — can suppress legitimate repetition (lists, code).
No-repeat-ngram size simply forbids any n-gram from repeating. Effective for catching loops but breaks legitimate repetition (variable names in code, refrains in poetry).
Min-p sampling (Minh 2024): keep tokens whose probability is ≥ min_p × max_prob. Adaptive like top-p but threshold-based — recently gaining traction in open-source serving as a top-p replacement.
Top-k, top-p, and temperature were the workhorses of 2019-2023, but the open-source serving stack has moved on. The samplers below are now first-class options in llama.cpp, vLLM, SGLang, exllamav2, and most local-LLM front-ends. Each addresses a specific failure mode that the older trio handled poorly.
Min-p sampling (Minh 2024). Keep tokens with probability ≥ min_p × max_p — i.e., set the floor as a relative fraction of the most likely token's probability. The threshold is dynamic: when the model is confident, the nucleus shrinks; when it is uncertain, the nucleus widens. Min-p is now the default in llama.cpp and Mistral inference, and it has largely replaced top-p in open-source roleplay/creative-writing setups because it is more robust to the "long tail of garbage" problem that flat top-p suffers from at high temperature.
Vmin-p={x:P(x)≥min_p⋅ymaxP(y)}
Mirostat (Basu 2020, widely adopted 2023-2024). Adaptive sampler that targets a fixed output entropy (often called perplexity surprise τ) by tuning an internal k value on the fly. Useful for long generation where you want consistent variety — neither slowly collapsing into greedy nor drifting into chaos. Mirostat-v2 is the version in most inference servers; it is especially popular for story-writing and dialog generation where you want the model's surprise level to feel uniform across thousands of tokens.
DRY — Don't Repeat Yourself (pcalc 2024). Penalty applied to tokens that would extend a detected n-gram repetition. Unlike no-repeat-ngram (which forbids any repetition outright), DRY uses an exponential penalty that grows with the length of the matched repeated suffix, so legitimate short repetitions (variable names, refrains, "no, no, no") pass through while genuine loops are choked off. Now standard in oobabooga/SillyTavern stacks.
XTC — eXclude Top Choices (p-e-w 2024). Counterintuitive: with some probability, drop the highest-probability tokens before sampling, forcing the model to pick from less obvious continuations. The goal is to escape the strong attractor of greedy-ish outputs without flattening the distribution wholesale. It produces noticeably more creative text on roleplay/fiction tasks, though it is risky on factual workloads.
Eta sampling (Hewitt 2022). Drops tokens whose probability falls below a learned, distribution-aware threshold η = min(ε, √ε · exp(H)), where H is the current entropy. It interpolates between aggressive truncation when the model is confident and gentler truncation when it is uncertain — a more principled cousin of min-p.
Top-q sampling. Similar in spirit: keep tokens whose log-probability is within q of the maximum log-probability (i.e., a multiplicative quality floor in log space rather than absolute probability space). Less common than min-p in production but useful for quantized models whose softmax is slightly miscalibrated.
Locally typical sampling (Meister 2023). Selects tokens whose information content (−log P) is close to the expected information content of the distribution, rather than just the most likely tokens. The justification is information-theoretic: human-generated text tends to hover near typical surprise levels, not at the extreme low-surprise tail that greedy decoding produces.
Quick check
Which sampler keeps tokens with probability ≥ min_p × max_p?
#Speculative Decoding: 2-3x Speedup, Zero Quality Loss
The big idea: a fast, small draft model proposes the next k tokens. The big model runs ONE forward pass on all k proposed tokens in parallel, then accepts a prefix of the draft based on a probability ratio test. Most of the time, all k tokens are accepted — so the big model effectively decoded k tokens in one forward pass.
Real applications often need exactly-formatted output: JSON conforming to a schema, code matching a regex, SQL with valid column names. Three techniques:
JSON mode / structured outputs (OpenAI, Anthropic, Mistral) — the API accepts a JSON schema and the server-side decoding masks invalid tokens at every step.
Outlines / Guidance — Python libraries that wrap a local model with a Pydantic / regex / context-free-grammar constraint, masking the logits to set invalid tokens to -∞.
Logit bias / fine-tuning — explicit per-token bias (OpenAI's logit_bias) or fine-tuning on schema-conforming examples.
Constrained decoding is "free" — no retraining needed, just decode-time masking. The cost: walk an automaton for the constraint, which can be slow for complex schemas.
Tests · Verify that gibberish tokens are excluded by top-p but appear in pure sampling. Verify that lowering temperature concentrates output on the top tokens. Verify your min-p implementation produces stable output.
Sampling PlaygroundInteractive
Adjust temperature, top-k, and top-p and watch how the same model produces wildly different completions of the same prompt.
Greedy is deterministic, beam is mode-seeking, sampling is creative — pick by task. Greedy and beam for translation/summarization/code/structured output. Top-p sampling for chat, creative writing, and any open-ended task.
Top-p is the modern default for sampling — top-k can't adapt to peaked vs flat distributions. Top-p (nucleus) sampling adjusts the candidate set per-position to match the distribution's shape; that's why it became standard after Holtzman 2019.
Temperature is the personality dial; T=0 → greedy, T~0.3 → factual/code, T=0.7 → chat, T>1 → creative, T>1.5 → noise. Combine with top-p/top-k; temperature is applied first, then the truncation.
Speculative decoding gives 2-3x speedup with zero quality loss. Production LLMs use Medusa, EAGLE, or vLLM speculation — the technique is provably equivalent to sampling from the big model.
Constrained decoding (JSON mode, Outlines, Guidance) makes models reliably emit schema-conforming output without retraining. Mask invalid tokens at decode time — fast, free, and the modern way to plug LLMs into structured pipelines.
You're building a customer-support chatbot that generates open-ended replies. Which decoding strategy is the best default?
You now have the full decoding toolkit. Next up: prompt engineering — the art of designing the input that drives all of this decoding machinery toward what you actually want.