AI has had three "springs" and two brutal "winters." Each winter began the same way — researchers promised thinking machines, delivered demos, and watched funding evaporate when reality fell short. We are in the biggest spring yet: $200B+ invested in 2024 alone. Learning the cycle is how you spot the difference between the next breakthrough and the next bubble.
Learning Objectives
After this lesson, you will be able to:
Walk through the key breakthroughs in AI from 1943 to 2026, understanding why each mattered and what it made possible
Explain why AI 'died' twice (the two AI winters) and what warning signs to watch for in today's AI boom
Connect the three ingredients of every AI breakthrough: a new architecture, enough compute, and a large dataset — and explain why all three must arrive together
Look at today's biggest AI trends (bigger models, AI agents that take actions, AI that sees and hears) and form your own informed opinions about what comes next
History is not boring -- it is your cheat code. Every "new" breakthrough in AI is actually an old idea whose time finally came. Understanding this pattern means you will spot the next big thing before everyone else does.
1943 -- McCulloch-Pitts neuron: Warren McCulloch and Walter Pitts published "A Logical Calculus of the Ideas Immanent in Nervous Activity," describing a mathematical model of a biological neuron. This was the first formal model of neural computation -- the ancestor of every artificial neuron in every neural network today.
1950 -- Turing's question: Alan Turing published "Computing Machinery and Intelligence," asking "Can machines think?" He proposed the Turing Test and predicted that by 2000, a computer could fool 30% of human judges in a 5-minute conversation. (His prediction was roughly correct: chatbots began passing versions of the Turing Test around 2014-2023.)
1956 -- The Dartmouth Conference: John McCarthy, Marvin Minsky, Claude Shannon, and Nathaniel Rochester organized a summer workshop at Dartmouth College to study "artificial intelligence" -- the term McCarthy coined for this proposal. The attendees believed that "every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it." They estimated it would take one summer. It has taken 70 years and counting.
1957 -- The Perceptron: Frank Rosenblatt built the Mark I Perceptron, a physical machine that could learn to classify patterns. The New York Times reported it as "the embryo of an electronic computer that the Navy expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence." The overpromise began early.
What Do You Think?
The Perceptron was demonstrated in 1957. How long before neural networks could reliably recognize handwritten digits?
The answer is roughly 40 years. While the Perceptron showed the basic idea, it took until 1998 for Yann LeCun's LeNet to reliably recognize handwritten digits on MNIST -- and even that required convolutional architectures and backpropagation, neither of which existed in 1957. Progress in AI is far slower than early optimists predicted.
ELIZA (1966): Joseph Weizenbaum's chatbot that mimicked a therapist. People became emotionally attached to it, astonishing its creator.
SHRDLU (1970): Terry Winograd's system that understood natural language in a limited "blocks world."
Expert systems: Rule-based systems that encoded human expertise.
Early machine learning: decision trees, nearest neighbors, early reinforcement learning.
The fatal flaw: Researchers promised general intelligence. They delivered narrow demonstrations. Governments poured money in expecting thinking machines. They got programs that could play checkers and answer questions about blocks.
What was the primary cause of the first AI winter (1974-1980)?
What happened:
The Lighthill Report (1973, UK): Criticized AI research as failing to deliver on promises. UK government cut AI funding dramatically.
US defense funding (DARPA) reduced as Vietnam War costs mounted and AI systems failed to meet military specifications.
Perceptron limitations: Minsky and Papert's book "Perceptrons" (1969) proved that single-layer perceptrons could not solve non-linearly separable problems (e.g., XOR). Neural network research was abandoned for over a decade.
Expert systems boom: Companies spent billions on rule-based AI systems. Lisp machines (specialized hardware for AI) became a market. Digital Equipment Corporation's XCON saved $40M/year configuring computer orders.
Backpropagation rediscovered (1986): Rumelhart, Hinton, and Williams published "Learning representations by back-propagating errors," showing that multi-layer neural networks could learn through gradient descent. This was the single most important breakthrough in the history of deep learning -- but it would take 25 years of hardware progress before its true potential was realized.
While AI was out of fashion, researchers made foundational progress:
1997 -- Deep Blue beats Kasparov: IBM's chess computer defeated the world champion. Important symbolically, but it was brute-force search, not learning.
1998 -- LeNet / MNIST: Yann LeCun's convolutional neural network for handwriting recognition. The architecture that would later power image recognition -- but in 1998, compute was insufficient for larger problems.
2000s -- The statistical revolution: Researchers largely stopped trying to hand-write rules for intelligence and started letting algorithms find patterns in data instead. The winning methods of this decade have names you will meet properly in the Classical ML track. What matters here is the shift: from telling the machine what to think, to showing it examples.
2006 -- Deep belief networks: Geoffrey Hinton showed that deep networks could be pre-trained layer by layer, reigniting interest in deep learning. The "deep learning" brand began.
2011 -- Watson on Jeopardy!: IBM's Watson defeated human champions. Complex NLP + information retrieval. Impressive, but still not "understanding."
What Do You Think?
What technological development (outside of AI research itself) was the primary enabler of the deep learning revolution in 2012?
2012 -- AlexNet: Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton won the ImageNet competition by a massive margin using a deep CNN trained on GPUs. This was the moment deep learning proved its superiority. Error rate dropped from 26% to 15% in one year (and would reach 3.5% by 2015, surpassing human performance).
What Do You Think?
AlexNet reduced ImageNet error from 26% to 15% in a single year. What was the key ingredient that made this possible?
The answer is all three working together. Deep CNNs were known since the 1990s (LeNet), GPUs existed for gaming, and ImageNet provided 1.2M labeled images. AlexNet was the first to combine all three -- none alone was sufficient. This recipe (architecture + compute + data) has defined every AI breakthrough since.
2014 -- GANs: Ian Goodfellow introduced Generative Adversarial Networks at NIPS. A generator and discriminator locked in a game. The idea was described as "the coolest idea in deep learning in the last 20 years" by Yann LeCun.
2014 -- Seq2Seq + Attention: Bahdanau et al. introduced the attention mechanism for machine translation. The seed that would grow into the Transformer.
2015 -- ResNet: Kaiming He introduced residual connections, enabling training of networks with 100+ layers. Skip connections solved the vanishing gradient problem for very deep networks.
2016 -- AlphaGo: DeepMind's system defeated world Go champion Lee Sedol. Go was considered far harder than chess due to its enormous search space (10^170 possible games). Move 37 in Game 2 was described as a move no human would ever play -- and it was brilliant. This was the moment the world realized AI could find solutions beyond human imagination.
2017 -- "Attention Is All You Need": Vaswani et al. at Google introduced the Transformer architecture. The paper's unpretentious title belied its revolutionary impact. By eliminating recurrence entirely and relying solely on attention, Transformers enabled massive parallelization during training. This architecture would become the foundation of virtually every major AI model that followed.
2018 -- BERT: Google's bidirectional Transformer for language understanding. Pre-trained on massive text corpora, then fine-tuned for specific tasks. Crushed every NLP benchmark.
2018 -- GPT-1: OpenAI's first generative pre-trained Transformer. "Only" 117M parameters. The idea: unsupervised pre-training on text, then fine-tuning.
2019 -- GPT-2: 1.5B parameters. OpenAI initially withheld the full model, citing concerns about misuse. Generated text so coherent that it seemed eerily human.
2020 -- GPT-3: 175B parameters. The scaling laws were clear: bigger models with more data produced dramatically better results. Few-shot learning emerged -- the model could perform tasks from just a few examples in the prompt, without fine-tuning.
2020 -- AlphaFold 2: DeepMind solved protein structure prediction, a 50-year grand challenge in biology. This was AI making a scientific breakthrough, not just beating humans at games.
2021 -- DALL-E and Codex: AI that generates images from text and writes code from natural language. Multimodality and practical utility arrived simultaneously.
2022 -- ChatGPT: GPT-3.5 with RLHF (Reinforcement Learning from Human Feedback) in a chat interface. Reached 100M users in 2 months -- the fastest adoption of any technology in history. This was the moment AI went from a technical topic to a cultural phenomenon.
What Do You Think?
GPT-3 had 175 billion parameters and was trained on 570 GB of text. What unexpected capability emerged that nobody explicitly programmed?
2023 -- GPT-4 and the frontier model race: Claude (Anthropic), Gemini (Google), Llama (Meta), Mistral. Multiple frontier labs competing. Multimodal models (text + image + code). Reasoning capabilities approaching human level on many benchmarks.
2023 -- Open source catches up: Meta's Llama 2, Mistral 7B, and the open-source community proved that smaller, well-trained models could rival much larger proprietary ones. Democratization of AI capabilities accelerated.
2024 -- Reasoning models: OpenAI's o1 and o3 demonstrated chain-of-thought reasoning at inference time. The shift from pure pre-training scaling to inference-time compute scaling. "Think harder, not just bigger."
2024-2025 -- Agentic AI: AI systems that plan, use tools, and take multi-step actions autonomously. Coding agents (Claude Code, Cursor, Devin), research agents, and workflow automation agents. The shift from AI as "oracle" (ask a question, get an answer) to AI as "agent" (describe a goal, AI accomplishes it).
2025-2026 -- Where we are now:
Frontier models match or exceed human performance on most standardized benchmarks
AI agents can write code, browse the web, manage files, and execute multi-step tasks
Multimodal models process text, images, audio, video, and code natively
The industry is investing $100B+ per year in AI infrastructure
Regulatory frameworks (EU AI Act) are taking effect
The gap between "AI can do X in a demo" and "AI reliably does X in production" remains substantial
More capable agents: AI agents will handle increasingly complex, multi-step tasks with less human oversight. Software development, data analysis, and research tasks will be heavily AI-assisted.
Multimodal everything: Models that seamlessly process and generate across all modalities (text, image, audio, video, 3D, code) will be standard, not special.
Smaller, better models: Continued improvements in training efficiency, distillation, and architecture will make GPT-4-level capabilities available on smartphones.
AI regulation: More countries will implement AI-specific regulations. Compliance will become a standard part of AI engineering.
Long-horizon autonomy: AI agents that work on tasks for hours or days, not minutes. Software projects, research investigations, and business workflows completed with minimal human input.
Science acceleration: AI-driven breakthroughs in drug discovery, materials science, mathematics, and climate modeling. AlphaFold was just the beginning.
Personalized AI: Models fine-tuned to individual users, understanding their preferences, communication style, and context. Privacy challenges are enormous.
Optimists (e.g., some researchers at OpenAI, DeepMind): AGI by 2030
Moderate view: continued progress on narrow capabilities, but general intelligence requires breakthroughs we have not yet made
Skeptics (e.g., some cognitive scientists): current architectures are fundamentally insufficient for general intelligence; we need new paradigms
AI consciousness: Does a sufficiently complex AI system develop subjective experience? This is a philosophical question that computer science alone cannot answer. Most researchers consider it premature, but it may become relevant as systems become more capable.
Economic transformation: Will AI automate most knowledge work? Predictions range from "modest productivity gains" to "wholesale economic restructuring." History suggests the truth will be somewhere in between, and the transition will take longer than optimists predict.
pythonplayground.py · Pyodide
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
Tests · Score at least 30/50 on the quiz by adding keyword-matching rules to guessYear().
Try it: Explore the AI timelineInteractive
Scroll through 75 years of AI history. Click milestones for details. Toggle the compute scaling overlay to see the exponential growth of training compute.
Loading visualization...
#How to Spot the Next Breakthrough (Practical Patterns)
History rhymes. Here are five concrete patterns you can use today to evaluate whether a hot new AI development is real or hype.
Every major AI breakthrough was sitting in a research paper for 3-7 years before it became practically useful. Watch this list:
Idea
Published
Made Useful
Delay
Backpropagation (training deep networks)
1986
~2012
26 years
Convolutional Neural Networks (LeNet)
1998
2012 (AlexNet)
14 years
Attention mechanism
2014
2017 (Transformer)
3 years
RLHF (reinforcement learning from human feedback)
2017
2022 (ChatGPT)
5 years
Chain-of-thought reasoning
2022
2024 (o1 models)
2 years
Practical takeaway: If you want to spot what's coming in 2027, read the 2024 papers that nobody is talking about yet. The next big thing is probably already published — just waiting for compute, data, or a clever engineer to put it together.
Every breakthrough needs architecture + compute + data. If a "breakthrough" is missing one of these, it's likely premature. Examples:
Quantum machine learning (2020-2024): Architecture exists, but quantum hardware is too small and noisy to train anything useful. Missing: compute. Verdict: too early.
Brain-computer interfaces (Neuralink, etc.): Architecture is sophisticated, compute is sufficient, but there is no large dataset of brain signals from millions of people. Missing: data. Verdict: narrow demos for years.
Multimodal agents (browsing + vision + planning, 2024-2025): All three ingredients present. Verdict: real, expanding fast.
Watch the gap between "AI can do X in a demo" and "AI does X in production at scale." Self-driving cars are the classic example: demos in 2015 promised "robotaxis by 2020." We are now in 2026 and most are still geofenced to specific cities (Waymo in Phoenix/SF; nothing widespread). The gap doesn't always close on the timeline boosters predict.
Real AI breakthroughs almost always show up as line items on quarterly earnings calls within 18 months. GitHub Copilot did. ChatGPT did. AlphaFold did (DeepMind's revenue). If a "breakthrough" has been around for 2+ years and no public company is reporting revenue from it, that's a yellow flag.
Listen to what AI researchers say off record (on podcasts, in lab meetings, on Twitter) vs. what their marketing departments say. The gap is usually large. Yann LeCun's tweets about LLM limitations are more accurate about LLM limitations than Meta's press releases. This signal-to-noise ratio is one of your most valuable skills.
Given the historical pattern of AI summers followed by AI winters, is a third AI winter likely?
The key difference between now and previous AI summers is economic integration. During previous summers, AI was a research curiosity with limited commercial applications. Today, AI generates billions in revenue (GitHub Copilot, ChatGPT, recommendation systems, ad targeting, fraud detection). Companies have restructured around AI capabilities. This economic foundation makes a full winter (where funding dries up and research stagnates) unlikely.
However, a correction is plausible. If AGI claims do not materialize on the promised timeline, if the cost of AI infrastructure does not produce proportional returns, or if safety incidents erode public trust, we could see reduced investment and increased skepticism. The difference is that the floor is much higher than in previous winters -- AI is now too useful to abandon.
AI history follows a boom-bust pattern. Explosive breakthroughs lead to overpromising, then crushing disappointments and "AI winters"; understanding this pattern helps calibrate expectations for current hype
Each breakthrough built on forgotten foundations. Backpropagation (1986), CNNs (1998), transformers (2017), and GPT (2020) all drew on decades-old ideas that became practical only when compute and data caught up
Scaling laws reveal a predictable path forward. Larger models trained on more data with more compute systematically improve, following power laws that let researchers predict capabilities before training
The current era's distinguishing feature is generality. Previous AI systems excelled at narrow tasks; foundation models (GPT, Claude, Gemini) demonstrate broad capabilities across language, vision, code, and reasoning in a single system
What was the significance of the 1986 backpropagation paper by Rumelhart, Hinton, and Williams?
The historical names you just walked through are not trivia — each one becomes a full lesson elsewhere on this platform. Use these forward-links to jump straight to where you will actually build the ideas.
Next up: AI Solving Real Problems. You now know the 80-year arc — what AI is, where it came from, and the three-ingredient recipe (architecture + compute + data) behind every breakthrough. The next lesson gets concrete: how AI actually solves real problems today, across eight industries, with company names, dollar figures, and the three "waves" (Traditional → Generative → Agentic) you'll keep hearing about for the rest of your career.