From tokenization and embeddings through GPT/BERT/T5, RLHF, FlashAttention, MoE, Mamba, ViT, mechanistic interpretability, tool-calling, structured generation, knowledge editing, synthetic data, modern safety/red-teaming, distributed training, pretraining data pipelines, and 2026 frontier evals. 35 lessons covering the foundations, the architectures, the alignment pipeline, the efficiency stack, and the deployment reality of 2026 LLMs.
BPE, WordPiece, SentencePiece — how raw text becomes tokens that models can process.
Word2Vec, GloVe, and embedding layers — how words become meaningful vectors that capture meaning.
"You shall know a word by the company it keeps." That 1957 quote by linguist J.
Why one attention head is not enough, and how multiple heads capture different relationships in parallel.
How transformers understand sequence order without recurrence — sinusoidal, learned, RoPE, ALiBi.
Putting it all together: encoder, decoder, layer norms, and residual connections.
Masked language modeling, bidirectional attention, and why encoders dominate understanding tasks.
Causal masking, autoregressive generation, and the GPT family evolution from GPT-1 to GPT-5.
T5, BART, Flan-T5 — the third architecture family. Span corruption, denoising, and when encoder-decoder beats decoder-only.
Kaplan, Chinchilla, emergent abilities — the laws that govern how loss falls with compute, data, and parameters.
Greedy, beam, top-k, top-p, temperature, speculative decoding — turning a token distribution into actual text.
Zero-shot, few-shot, chain-of-thought, prompt injection, and practical prompt design.
When to fine-tune vs prompt engineer. LoRA, QLoRA, and the SFT pipeline — customize any LLM on a single GPU.
The alignment pipeline — SFT, reward modeling, PPO, DPO, and Constitutional AI. How every major LLM learns to be helpful, harmless, and honest.
How reasoning models spend more compute on harder questions — extended thinking, process reward models, and the test-time scaling paradigm.
The modern efficiency stack — FlashAttention, GQA/MQA, KV cache, paged attention, sliding window, attention sinks.
Making LLMs cheap and fast at inference — GPTQ/AWQ/GGUF quantization, speculative decoding, vLLM PagedAttention serving.
Switch Transformer, Mixtral, GShard — total params scale much faster than active params per token.
Mamba, S4, RWKV, Hyena — linear-time sequence models that compete with attention on quality.
ViT, Swin, DEiT, MAE, DINOv2 — how transformers process images as patch sequences.
Visual encoder + projection + LLM — how Gemini, Claude 3, GPT-4V, LLaVA understand images alongside text.
GPT-4o/Claude 4/Llama-3/Gemini/DeepSeek/Qwen survey + MMLU/HumanEval/MT-Bench/Chatbot Arena. The capstone.
GRPO algorithm, DeepSeek R1 training recipe, process vs outcome rewards, and why reasoning emerged from pure RL.
Anthropic's CAI approach — critique-revision loops, RLAIF, and scaling alignment without human labelers.
LoRA/QLoRA internals, PEFT comparison, and merging fine-tuned models with SLERP, TIES, and DARE.
GPT-4 will not tell you what it was trained on.
A 70-billion-parameter model in fp32 needs 280 GB just for weights.
In February 2024, an Air Canada chatbot promised a bereaved customer a discount that did not exist.
By 2026 the old benchmark canon is rubble.
In May 2024 Anthropic put up a public demo called Golden Gate Claude.
GitHub Copilot crossed $400M ARR in 2024, powered for years by a Codex model that started as a research curiosity.
Underneath every agent demo, every "ChatGPT booked your flight" headline, and every Claude Code session, there is exactly one mechanism doing the real work: tool calling.
Every production LLM pipeline eventually hits the same wall: the model emits text, but the next stage needs JSON.
In 2023 an OpenAI customer support engineer noticed that GPT-4 was confidently telling users that the CEO of Twitter was Jack Dorsey.
Synthetic data closes the NLP & Transformers track — thirty-five lessons from tokenization to RLHF, FlashAttention, MoE, Mamba, ViT, mech-interp, tool-calling,…
14 interactive labs — hands-on exercises for this track
How does AI read? First it breaks text into tokens — type a sentence and see it split into pieces
Words live in a space where similar meanings are close together — explore king - man + woman = queen
Watch data flow through a transformer step by step — the architecture behind ChatGPT
See which words a transformer pays attention to — a vivid heatmap of what the model focuses on
Without this, 'dog bites man' and 'man bites dog' look identical to a transformer — see how position information is injected.
Step through every matrix operation in self-attention: Q·K^T, scale, softmax, weighted sum of V.
Eight attention heads side-by-side — each specializing in a different linguistic pattern. Hover to enlarge and watch the concatenated output emerge.
Type any sentence and watch it fragment into BPE tokens. Compare English-only, Multilingual, and Character-level vocabularies.
Watch data flow through a single transformer encoder block — multi-head attention, residuals, LayerNorm, and FFN. Toggle residuals off and see gradient flow collapse.
Drag a single slider and reshape the next-token probability distribution in real time — focused, balanced, creative, chaotic.
Watch beam search branch, score, and prune multiple candidate completions in parallel. Compare greedy vs k=3 vs k=5.
Toggle bidirectional (BERT) vs causal (GPT) attention and step through autoregressive generation position-by-position.
[batch=4, seq=6, dim=8] tensor. BN highlights columns; LN highlights rows. Toggle variable seq lengths — BN stats rot, LN stays identical.
Step through autoregressive generation. Without cache: quadratic recompute. With cache: one row appended per step.
900 questions across 35 modules — check how well you understood this track.