Self-attention lets every word look at every other word simultaneously — a sentence of 100 words does 100×100 = 10,000 comparisons in a single layer.
Watch how 'sat' attends to 'cat' — the model learns this dependency without any hard-coded grammar rules.
This lab builds on concepts from:
Self-Attention Mechanism
Example sentence
Step 1 of 5
Input Embeddings
X ∈ ℝ^{N×d}Each token is represented as a dense vector (embedding). We have 5 tokens, each with a 3-dimensional embedding. These are the raw inputs to the self-attention layer.
Each token is mapped to a 3-dimensional embedding vector. These are the inputs X to the attention layer.
Each row = one token. Each column = one embedding dimension.