In 2018, Google's BERT shocked the NLP world by learning language through a fill-in-the-blank game. Train on hundreds of millions of sentences with random words masked, predict the missing words, and somehow you get a model that beats every previous NLP system on 11 benchmarks. BERT is still inside Google Search ranking your results today — and modern siblings like ModernBERT (2024) keep encoder-only models alive for embeddings, classification, and retrieval-heavy workloads where decoder-only GPTs are overkill.
Learning Objectives
After this lesson, you will be able to:
Understand masked language modeling (MLM): BERT learns by predicting 15% of randomly masked tokens using full bidirectional context
Know the 80/10/10 masking rule: 80% replaced by [MASK], 10% replaced by a random token, 10% left unchanged -- and why this mixture prevents the model from only learning to predict [MASK]
See what the [CLS] token does and how its final hidden state serves as a fixed-size sentence embedding for classification
Walk through BERT's two-stage recipe: pretrain on unlabeled text (expensive, done once), then fine-tune with a task head (cheap, done per task)
Understand Next Sentence Prediction (NSP) -- BERT's secondary pretraining objective -- and why later research (RoBERTa) found it unnecessary
Know when to use an encoder model (understanding, embeddings, classification) vs. a decoder model (generation, chat, reasoning)
See real use cases: sentiment analysis, named entity recognition, semantic search, and extractive question answering
Before BERT, the dominant paradigm was unidirectional language modeling -- predict the next token given all previous tokens. This worked well for generation but poorly for understanding. If a model cannot see future context, it cannot fully understand the current word. The word "bank" in "I sat on the river bank" needs the word "river" (which comes before) and the overall sentence context to disambiguate from a financial bank.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova (2018)
The paper that proved bidirectional pretraining produces superior representations for understanding tasks. Table 1 shows BERT beating every existing model on every benchmark.
BERT's pretraining objective is elegantly simple: randomly mask 15% of input tokens and train the model to predict them. MLM is the textbook example of self-supervised pretraining: the labels (the masked words) come for free from the unlabeled corpus itself, and the corrupted-input / reconstruct-target setup is a denoising form of contrastive learning. See self-supervised pretraining for the broader family (contrastive image models, SimCLR/MAE, audio masking) that MLM belongs to.
Try it: BERT Masked Language ModelingInteractive
Loading visualization...
Try it! Type a sentence you know well -- maybe a famous quote or a song lyric. Then watch BERT mask random words and try to predict them. Can you beat BERT at guessing the blanked-out words? You will be surprised how good it is.
Type a sentence, adjust the mask probability, and watch BERT mask tokens and predict them using bidirectional context. Hover over any token to see that it attends to every other token -- both left and right. This is the key difference from GPT-style models.
Given an input sentence, randomly select 15% of tokens. For "The cat sat on the fluffy mat", you might select "cat" and "fluffy". Of the selected tokens: 80% are replaced with a special [MASK] token, 10% are replaced with a random word, and 10% are left unchanged. This mixture prevents the model from learning that [MASK] is special.
The masked input ("The [MASK] sat on the [MASK] mat") passes through the full Transformer encoder. Every token attends to every other token -- including tokens on both sides of the masked positions. The [MASK] at position 2 can see "The", "sat", "on", "the", "mat" simultaneously. No causal masking, no left-to-right restriction.
At each masked position, the model outputs a probability distribution over the entire vocabulary. For position 2 (where "cat" was), the model should assign high probability to "cat" and related words ("dog", "kitten") and low probability to unrelated words ("computer", "democracy"). The loss is computed only on the masked positions.
4
Step 4: Compute the Loss
LMLM=−i∈masked∑logP(xi∣x\i)
What Do You Think?
BERT masks 15% of tokens during training. Why not mask 50% to give the model a harder challenge and learn faster?
Start with raw text from the training corpus: "The cat sat on the fluffy mat." This text is tokenized into subword tokens using WordPiece, producing tokens like ['[CLS]', 'The', 'cat', 'sat', 'on', 'the', 'flu', '##ffy', 'mat', '.', '[SEP]']. The special [CLS] token is prepended and [SEP] is appended.
Randomly select 15% of tokens to mask. Of those selected: 80% become [MASK], 10% become a random token, and 10% stay unchanged. The sentence might become: ['[CLS]', 'The', '[MASK]', 'sat', 'on', 'the', '[MASK]', '##ffy', 'mat', '.', '[SEP]']. The original tokens "cat" and "flu" are recorded as targets.
The masked input passes through all 12 (BERT-base) or 24 (BERT-large) Transformer encoder layers. Each layer applies multi-head self-attention and a feed-forward network. Critically, attention is bidirectional -- every token sees every other token, both left and right. The at position 3 can see "The" on its left and "sat on the ... mat" on its right simultaneously.
BERT prepends a special [CLS] (classification) token to every input. After passing through all Transformer layers, this token's final hidden state serves as a summary of the entire sequence -- a fixed-size vector representation of the input.
This is powerful because it reduces any variable-length text input to a single fixed-size vector that captures its meaning. You can then attach a simple classifier on top -- a single linear layer -- to adapt BERT to any downstream task.
Train on massive unlabeled text (English Wikipedia + BookCorpus, ~3.3 billion words) using MLM and a secondary objective called Next Sentence Prediction (NSP). This takes days on TPU clusters and costs hundreds of thousands of dollars. But it only needs to happen once. The result is a general-purpose language understanding model.
Take the pretrained BERT, add a task-specific output layer (often just a single linear layer), and train on a small labeled dataset for your specific task. Sentiment analysis? Add a 768-to-2 linear layer on top of [CLS]. Named entity recognition? Add a 768-to-N linear layer on top of each token. Fine-tuning is fast (minutes to hours on a single GPU) and requires as few as 1,000 labeled examples.
Pretraining teaches BERT the structure of language -- grammar, semantics, world knowledge, coreference. Fine-tuning teaches it to apply that knowledge to a specific task. It is the same principle as training a medical student: years of general education (pretraining), then a brief residency in a specialty (fine-tuning). The student does not learn biology from scratch for each specialty.
Identify and classify entities in text: person names, organizations, locations, dates. BERT's bidirectional attention is ideal because entity classification depends on both left and right context. "Washington" is a person, city, or state depending on surrounding words.
Classify text as positive, negative, or neutral. Fine-tuned BERT models achieve 95%+ accuracy on standard benchmarks with just the [CLS] token and a linear classifier. The bidirectional context helps resolve sarcasm and negation ("not bad" = positive).
Sentence-BERT (SBERT) adapts BERT to produce sentence embeddings that can be compared with cosine similarity. This powers semantic search, duplicate detection, and recommendation systems. Encode millions of documents offline, then match queries in milliseconds.
Given a passage and a question, find the answer span within the passage. BERT predicts start and end positions of the answer. SQuAD 2.0 benchmarks showed BERT surpassing human-level performance on extractive QA.
Try it: Tokenizer — see how BERT splits wordsInteractive
Loading visualization...
Explore this: Type "unbelievable" and watch it split into subword tokens. Try rare words, technical jargon, and emojis — rare words get split into many small pieces, common words stay whole. This is BPE (Byte Pair Encoding) — the tokenization algorithm that lets transformers handle any vocabulary without an out-of-vocabulary problem.
⚡ Playground:Tokenizer → — type any text and watch it split into the subword tokens BERT actually sees.
#ModernBERT (Dec 2024) and the encoder renaissance
For most of 2022-2024, encoders looked like a sunsetting category — every press release was about a new decoder-only LLM, and the original BERT codebase sat untouched since 2018. Then in December 2024, Answer.ai and LightOn released ModernBERT, a from-scratch redesign that pulled six years of transformer research into the encoder stack and showed encoders still have a lot of headroom.
ModernBERT keeps BERT's bidirectional MLM objective but rebuilds the architecture with: rotary position embeddings (RoPE) instead of learned positionals (which is why it handles long contexts gracefully), FlashAttention 2 for memory-efficient quadratic attention, GeGLU activations replacing GELU in the FFN, alternating local/global attention for sequence-length scaling, unpadding so batched inference doesn't waste compute on padding tokens, and a context window of 8192 tokens — 16× the original BERT's 512. The net effect is roughly 10× faster inference than the original BERT at higher quality, and the long context unlocks document-level classification and retrieval that BERT-base could never handle.
NeoBERT (Feb 2025, Boscovich et al.) is a parallel effort that pushes a similar set of modernizations — depth-to-width ratios borrowed from the Pythia scaling sweeps, sequence length 4096, DeepSeek-style data filtering — and matches or beats ModernBERT on GLUE and BEIR. The two papers together mark the encoder renaissance: it turns out the architecture had been stagnating, not topping out.
Why does this matter when we have GPT-4o and Claude? Because most production NLP work is still understanding, not generation — and encoders dominate that work on cost, latency, and quality. RAG retrieval is the obvious one: every serious embedding model in 2024-2026 (BGE-large, jina-embeddings-v3, intfloat/e5-mistral, Voyage-3) is built on an encoder backbone, because dense retrieval needs a bidirectional representation and an LLM-grade decoder would be wasteful at indexing scale. Reranking, classification, NER, content moderation, deduplication, and semantic search all run on encoders for the same reason — sub-10ms latency at thousands of QPS, on hardware that costs orders of magnitude less than serving a 70B decoder. The lesson of ModernBERT is not "encoders are back" so much as "encoders never left — they just needed an update."
Bidirectional attention is BERT's core advantage: unlike GPT, BERT can attend to both left and right context simultaneously, making it far better at understanding than at generating
MLM trains understanding implicitly: predicting 15% of masked tokens forces the model to learn syntax, semantics, and world knowledge from context alone — no labeled data required
The [CLS] token is a sentence embedding: its final hidden state aggregates full-sequence information; fine-tune a linear layer on top of [CLS] for classification tasks
Pretraining + fine-tuning transfers cheaply: a fine-tuned BERT-base classifies text in 2–5ms and costs ~0.0001 cents per call — 10,000x cheaper than a frontier LLM for tasks that don't require generation
Encoders are the right tool for RAG retrieval: sentence-transformers (encoder-based) produce dense embeddings; use them at indexing time and query time, then pass retrieved chunks to a decoder for generation
BERT showed us the power of encoders for understanding. But what about generation -- writing new text, holding conversations, reasoning step by step? For that, we need to look at the other side of the Transformer: the decoder. Next up: GPT and the autoregressive revolution.
Cross-entropy loss between the predicted token probabilities and the true tokens, computed only at masked positions. The model learns to use full bidirectional context to reconstruct the missing words. Over billions of training examples, this forces the model to develop rich, contextual representations of language.
At each masked position, the final hidden state is projected to the full vocabulary size (30,522 tokens) and softmax is applied. For the mask where "cat" was, the model should output a probability distribution peaking at "cat." For the mask where "flu" was, it should peak at "flu." The model uses full bidirectional context to reconstruct the missing words.
Cross-entropy loss is computed between the predicted probabilities and the true masked tokens. The loss is summed only over masked positions -- non-masked tokens do not contribute to the loss. The model learns to build deep contextual representations that capture grammar, semantics, and world knowledge over billions of training examples.
After pretraining, add a task-specific head on top of BERT. For classification: take the [CLS] token's final hidden state (768 dimensions) and pass it through a single linear layer mapping to the number of classes (e.g., 2 for sentiment). For NER: add a linear layer on top of every token. Fine-tune the entire model on a small labeled dataset.
The fine-tuned model produces task-specific outputs. For sentiment analysis: "POSITIVE" or "NEGATIVE." For NER: entity labels like "B-PER," "I-ORG," "O" for each token. For question answering: start and end positions of the answer span. The pretrained representations transfer knowledge from billions of unlabeled tokens to your specific task with as few as 1,000 labeled examples.