The original 2017 transformer paper was encoder-decoder. Then GPT-style decoder-only models won the LLM race so completely that most people forgot encoder-decoder existed. Wrong move. Google Translate, every modern summarization pipeline, and a quiet 2024 wave of "T5 is back" research all run on this architecture — and once you see span corruption pretraining, you will understand why decoder-only is not always the right answer.
Learning Objectives
After this lesson, you will be able to:
Tell encoder-decoder, encoder-only (BERT), and decoder-only (GPT) apart by which task each was built for, and pick the right family for a given problem
Walk through T5's text-to-text framing — every NLP task becomes 'input string → output string' — and write the task-prefix prompts T5 was trained on
Trace span corruption pretraining: mask contiguous chunks of text, train the model to fill in the masked spans, and see why it generalizes to translation, summarization, and Q&A
Decide when an encoder-decoder will beat a decoder-only LLM (translation, summarization, structured generation) and when decoder-only wins (open-ended generation, in-context learning, chat)
Don't worry if "encoder-decoder" sounds like another architecture to memorize — it is genuinely the third family alongside BERT and GPT, and once you see span corruption you'll understand why it exists.
Watch how each decoder query reaches back across every encoder position to pull in the source representation.
Loading visualization...
The original transformer beat all prior NMT systems on WMT'14 English-German. But the architecture was complicated, training was finicky, and BERT (encoder-only, 2018) and GPT (decoder-only, 2018) showed that for many tasks, you only needed half of it. So encoder-decoder went out of fashion for a few years — until T5.
T5's revolutionary idea was simple: stop having different output heads for different tasks. Every NLP task becomes (input_string, output_string). You distinguish tasks with a task prefix in the input.
translate English to German:This is good.summarize:<long article>cola sentence:The course are jumping well.stsb sentence1:...sentence2:...→Das ist gut.→<short summary>→not_acceptable→3.8
The C4 dataset (Colossal Clean Crawled Corpus, ~750GB of cleaned web text) and 1.5T training tokens gave T5 the breadth to handle hundreds of downstream tasks with one frozen pretrain.
T5 does not use masked language modeling like BERT, and not causal language modeling like GPT. It uses span corruption — a hybrid that suits the encoder-decoder shape.
The recipe:
Pick contiguous spans in the input (avg length 3 tokens, masking 15% of total tokens).
Replace each span with a unique sentinel token (<X>, <Y>, <Z>, ...).
The encoder sees the corrupted input.
The decoder is trained to emit the sentinels followed by their original spans.
Original:Corrupted (encoder input):Decoder target:Thank you for inviting me to your party last week.Thank you ⟨X⟩ me to your party ⟨Y⟩ week.⟨X⟩ for inviting ⟨Y⟩ last ⟨Z⟩
This objective is much more sample-efficient than BERT's MLM (which only predicts ~15% of positions and is per-token). It is also better aligned with downstream generation tasks than GPT's left-to-right CLM.
BART takes the same encoder-decoder shape but uses a different family of corruption schemes:
Token masking. Like BERT.
Token deletion. Remove tokens entirely (no placeholder), forcing the decoder to figure out which positions were corrupted.
Text infilling. Replace spans with a single mask, model has to figure out length too.
Sentence permutation. Shuffle sentences in a document.
Document rotation. Rotate the document so it starts at a random token.
The decoder is always trained to reconstruct the original document. Because the corruption can be aggressive, BART learns very robust representations of language structure.
BART uses around 400M parameters and dominated abstractive summarization benchmarks (CNN/DailyMail, XSum) in 2020-2021, beating GPT-2 (1.5B) by clear margins.
What Do You Think?
You need to translate confidential medical documents with strict input-output pairing — every translated sentence must correspond exactly to a source sentence. Which architecture?
Encoder-decoder. BERT cannot generate (no decoder). Decoder-only LLMs can translate via prompts but they hallucinate, drift in style, and are wasteful for a strict mapping task. Purpose-built MT models like NLLB-200 and M2M-100 are encoder-decoder for exactly this reason: the encoder builds a deep bidirectional representation of the source sentence, the decoder generates a target with full cross-attention back to it, and the architecture mirrors the task.
Autoregressive generation matches the typing pattern
Sentiment classification
Encoder-only or T5
Both work; encoder-only is cheaper if you only do this one task
The decoder-only camp won 2020-2025 because (1) LLM scaling laws favor a single objective, (2) in-context learning emerges in decoder-only models more cleanly, (3) one model that does everything via prompts is simpler to serve. But for tasks with strict input-output structure, encoder-decoder is still the right tool — and the 2024+ research wave (UL2, code-T5, retrieval-augmented encoder-decoder) is bringing it back.
Tests · Verify the model loads, the four tasks all run without error, and the outputs look reasonable (translations roughly match, summary captures the gist, cola classification is acceptable/not_acceptable).
Three transformer families exist. Encoder-only (BERT) for understanding, decoder-only (GPT) for generation, encoder-decoder (T5/BART) for input → output mapping. The original 2017 transformer was encoder-decoder.
T5's text-to-text framing was a conceptual breakthrough. Every NLP task becomes a string-in, string-out problem, distinguished by a task prefix. This blueprint inspired modern instruction tuning and chat-LLM prompting.
Span corruption pretraining beats MLM and CLM for encoder-decoder shape. By masking contiguous spans and training the decoder to fill them, T5 learns rich bidirectional input representations AND an autoregressive generation head in one objective.
Pick encoder-decoder for translation, summarization, structured generation. Tasks with strict input-output mapping. Pick decoder-only for chat, code, and any open-ended generation.
The 2024+ encoder-decoder renaissance is real. UL2, CodeT5, retrieval-aware models, and constrained-generation pipelines are bringing encoder-decoder back for production tasks where decoder-only LLMs are wasteful.
Which task is the canonical fit for an encoder-decoder model?
Encoder-decoder is the third family — the one Vaswani designed in 2017, displaced by GPT and BERT in 2018, then resurrected by T5 and BART. Next: how scaling laws turned bigger models into a science — and why Chinchilla rewrote the GPT-3 playbook.
Structured extraction (form → JSON)
Encoder-decoder or constrained decoder-only
Bidirectional input understanding helps; decoding under a schema