RugvAI Labs

Positional Encoding

Attention is order-agnostic by default — "dog bites man" and "man bites dog" look identical to the raw attention operation. Positional encoding is the fix, and it has quietly become the most-iterated piece of the transformer stack. Sinusoidal in 2017, RoPE in 2021 (used by Llama, Mistral, DeepSeek), ALiBi for long context, YaRN to stretch RoPE to 1M tokens — every context-length breakthrough in 2024–2026 came from this layer.

Learning Objectives

After this lesson, you will be able to:

Understand why attention has no idea about word order -- and why that is a huge problem
See how sine and cosine waves give each position a unique fingerprint
Understand why these encodings let the model figure out relative distances between words
Compare four approaches to position encoding: sinusoidal, learned, RoPE, and ALiBi
Understand why RoPE became the go-to method in modern LLMs like LLaMA and Claude
Grasp the length extrapolation problem: what happens when the input is longer than anything seen during training

Prerequisites

Before You Start

Review

Active Recall

Before we diagnose the position problem, recall: write the attention formula Attention(Q,K,V) = ? from the self-attention lesson, then state in one sentence what `QK^T` is geometrically. The reason: the whole 'attention is permutation-equivariant' argument falls out of that one matrix product — and noticing why is much easier if you've reconstructed the formula yourself first.

Write your answer in your own words — don't look back at the lesson. This is the most effective way to remember what you just learned.

Type your explanation above

#The Bag-of-Words Blind Spot

Self-attention computes QK^T, which only depends on the content of tokens, not their positions. Shuffling the input sequence would shuffle the output in exactly the same way -- the attention mechanism is permutation-equivariant. For language, where word order carries critical meaning, we need to inject position information explicitly.

Why

Why This Matters

How ChatGPT knows that "dog bites man" is not the same as "man bites dog" -- without positional encoding, a transformer would treat these two sentences as completely identical because they contain the exact same words; positional encoding stamps each word with its position number so the model knows word order, which is literally the difference between a news story and a headline
How Claude, GPT-4, and LLaMA can read 200K+ tokens of text -- RoPE (Rotary Position Embedding) is the specific technique that lets modern models process entire books and massive codebases; earlier approaches broke down after a few thousand words because the model had no way to represent positions it had not seen during training
One of the hottest research topics in AI right now -- making models work on longer inputs than they were trained on (called length extrapolation) is an active research frontier; every improvement directly translates to AI that can read longer documents, hold longer conversations, and handle bigger codebases without losing track of context
Build this --> Build a position encoder that generates sinusoidal positional encoding vectors for a sentence, then demonstrates the key property: show that the similarity between positions 5 and 8 (distance 3) is nearly identical to the similarity between positions 20 and 23 (also distance 3) -- proving the encoding captures relative position

What Do You Think?

We need a positional encoding scheme where position 5 and position 8 have a relationship that is the same as position 20 and position 23 (both 3 apart). Which mathematical function naturally encodes this kind of relative shift?

Try it! Shuffle the words in this sentence: "The cat sat on the mat." One possible shuffle: "mat the on sat cat The." Does the meaning change? Absolutely. Now imagine a model that cannot tell these apart. That is exactly the problem positional encoding solves.

#The Solution: Add Position to Content

The original Transformer adds a positional encoding vector to each token embedding before passing it to the attention layers. Each position gets a unique vector, and this vector is added (not concatenated) to the token's embedding.

\text{input}_i = \text{Embedding}(\text{token}_i) + \text{PE}(i)

But how do we create these positional encoding vectors? The original paper used an elegant mathematical construction based on sine and cosine waves.

Intuition

Positional encoding is like a GPS coordinate for each word.

Transformers process all words in parallel — they have no built-in sense of order. Without positional encoding, "dog bites man" and "man bites dog" would look identical.

Sinusoidal encoding gives each position a unique "fingerprint" using sine and cosine waves of different frequencies. Think of it like a clock with multiple hands:

The second hand (high frequency) changes rapidly — nearby positions look very different
The hour hand (low frequency) changes slowly — positions far apart can still share similar patterns
Together, they create a unique combination at every position

Why sine/cosine? Because the model can learn to compute relative positions using simple addition. Position 5's encoding minus position 3's encoding always equals the same "distance of 2" vector, regardless of absolute position.

#Sinusoidal Positional Encoding

The encoding uses sine and cosine functions of different frequencies across dimensions:

PE(pos, 2i) = \sin\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)

PE(pos, 2i{+}1) = \cos\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)

#The Clock Analogy

Think of a clock. The second hand spins fast (high frequency) -- it can distinguish nearby moments in time. The minute hand spins at medium speed. The hour hand spins slowly (low frequency) -- it tells you roughly what part of the day it is. Together, all three hands give you precise time. Sinusoidal positional encoding works the same way, but with hundreds of "hands" spinning at different rates.

#Dimension 0-1: High Frequency

The first pair of dimensions oscillates rapidly, with a wavelength of about 2*pi (roughly 6.28 positions). This captures fine-grained position differences: position 5 looks very different from position 6 in these dimensions. These dimensions distinguish adjacent tokens.

#Dimension d_model-2, d_model-1: Low Frequency

The last pair of dimensions oscillates extremely slowly, with a wavelength of about 10000 * 2*pi (roughly 63,000 positions). In these dimensions, position 5 and position 6 look nearly identical, but position 5 and position 30000 look very different. These dimensions distinguish distant tokens.

#The Complete Fingerprint

Each position gets a unique pattern across ALL dimensions -- a unique "fingerprint" that combines information at every frequency. Position 0 has its fingerprint, position 1 has a different one, position 1000 has yet another. No two positions share the same encoding vector.

#The Genius: Relative Positions as Linear Transformations

PE(pos + k) = M_k \cdot PE(pos), \quad M_k = \begin{pmatrix} \cos(wk) & \sin(wk) \\ -\sin(wk) & \cos(wk) \end{pmatrix}

#Modern Alternatives

The sinusoidal encoding was the original approach, but the field has moved on significantly. Here is the landscape:

#Learned Positional Embeddings

Instead of fixed sinusoids, learn a separate embedding vector for each position (just like token embeddings). Used in BERT and early GPT models.

Pros: Simple to implement, can capture arbitrary position patterns
Cons: Limited to the maximum sequence length seen during training. Position 513 has no embedding if you trained on max 512 tokens.

#RoPE (Rotary Position Embedding)

f(x_m, m) = R_{\theta,m} \cdot x_m, \quad \text{where } R_{\theta,m} = \begin{pmatrix} \cos(m\theta_1) & -\sin(m\theta_1) \\ \sin(m\theta_1) & \cos(m\theta_1) \\ & & \ddots \\ & & & \cos(m\theta_{d/2}) & -\sin(m\theta_{d/2}) \\ & & & \sin(m\theta_{d/2}) & \cos(m\theta_{d/2}) \end{pmatrix}

Instead of adding position to the input, RoPE encodes position by rotating the query and key vectors in 2D subspaces. The rotation angle depends on position, so the dot product between a query at position m and a key at position n naturally encodes their relative distance (m - n).

Worked example: RoPE for d_head = 4

The cleanest way to internalize RoPE is to grind through one small example by hand. Pick a head dimension of d = 4, the standard base of 10000, and two token positions m = 3 (the query) and n = 7 (the key). With d = 4 the head splits into two 2D pairs, so we only need two frequencies.

The frequency formula is θ_i = base^(-2i/d) for i = 0, 1, ..., d/2 - 1. Plugging in:

θ_0 = 10000^(-0/4) = 10000^0 = 1
θ_1 = 10000^(-2/4) = 10000^(-0.5) = 1/√10000 = 0.01

Position m = 3 rotates the first pair by m·θ_0 = 3·1 = 3 radians and the second pair by m·θ_1 = 3·0.01 = 0.03 radians. Position n = 7 rotates by 7 radians and 0.07 radians respectively. The rotation matrices stay tiny — each is a 2×2 with sines and cosines of those angles — and you can verify by hand that cos(3) ≈ -0.990 and cos(7) ≈ 0.754.

Now the magic. Take the dot product of the rotated query and key. Because rotations preserve dot products and compose by adding angles, the rotation matrix R(m)^T · R(n) collapses to R(n - m) for each 2D block. The query at m = 3 and key at n = 7 therefore produce a score whose angular argument is exactly (n - m)·θ_i = 4·θ_i for each frequency — the absolute positions vanish.

\langle R_{\theta,m} q,\ R_{\theta,n} k \rangle = q^{T} R_{\theta,m}^{T} R_{\theta,n} k = q^{T} R_{\theta,\,n-m} k

The same relative-position property is what makes RoPE play nicely with KV-caches: a key vector rotated by its original position stays valid forever; you only need to rotate the new query.

\text{score}(m, n) = \sum_{i=0}^{d/2-1} \big(q_{2i}\,k_{2i} + q_{2i+1}\,k_{2i+1}\big)\cos\big((n-m)\theta_i\big) + \big(q_{2i}\,k_{2i+1} - q_{2i+1}\,k_{2i}\big)\sin\big((n-m)\theta_i\big)

Extrapolation: NTK-aware and YaRN

RoPE generalizes to longer contexts than seen during training, but only if you scale the frequencies carefully. Two recipes dominate practice:

NTK-aware scaling (originally posted on Reddit by bloc97 in 2023 and adopted by Code Llama, Mistral, and Qwen) keeps the high-frequency dimensions roughly intact and stretches only the low-frequency ones. The idea: divide the base of the exponent by a scale factor s = target_length / training_length, raised to a power that depends on the dimension.

\theta_i^{\prime} = \theta_i \cdot s^{-\,2i/(d-2)}, \qquad s = \frac{L_{\text{target}}}{L_{\text{train}}}

A LLaMA 2 model trained at 4K extended to 32K simply uses s = 8, and most fine-grained tasks survive without further training. The trade-off is a small but measurable hit at the original (4K) context — the high-frequency dimensions are subtly perturbed.

YaRN (Peng et al., 2023) refines NTK-aware by treating each frequency band differently: high-frequency dims (which encode local order) are left untouched, low-frequency dims (which encode long-range structure) get full linear stretching, and a middle band is interpolated by a smooth ramp. YaRN also rescales attention logits by a temperature 1/sqrt(t) where t = 0.1 ln(s) + 1 to compensate for the longer sequences having more keys to attend over. The net effect: Mistral 7B trained at 8K extends cleanly to 128K with under a billion tokens of fine-tuning, and Qwen2 uses YaRN for its 128K and 1M variants.

When the target length blows past the trained length without any scaling, the dot product oscillates wildly at frequencies the model has never seen, the attention distribution becomes nearly uniform, and the model effectively produces noise — exactly the failure mode the next quiz probes.

Quick check

A model trained with vanilla RoPE on 4K context is asked to process a 32K prompt without NTK-aware or YaRN scaling. What is the most likely failure mode?

#ALiBi (Attention with Linear Biases)

The simplest approach: do not modify embeddings at all. Instead, add a position-dependent bias directly to the attention scores. Tokens farther apart get a penalty (negative bias), making them attend less to each other by default. Remarkably effective and trivially extrapolates to longer sequences.

\text{score}(i,j) = q_i^T k_j - m \cdot |i - j|

Watch Out

90-second drill: four position encodings, four use cases (interleaving check)

Fix: Without scrolling back, match each scheme to its defining property — out of order: (a) Sinusoidal, (b) Learned, (c) RoPE, (d) ALiBi against (1) hard-capped at training length, no extrapolation, (2) adds a linear distance penalty to attention scores directly, (3) rotates Q and K so Q·K depends only on (m − n), (4) adds a fixed sin/cos pattern to embeddings before layer 0. Now: which one is in Llama 3? Which one in BERT? Which one in the original 2017 paper? Which one extrapolates 'for free' with no scaling tricks? If you stumble on any, the modern-alternatives section above is your key — but commit to your answers first. (Order: a→4, b→1, c→3, d→2; Llama 3 uses RoPE, BERT uses learned, original transformer uses sinusoidal, ALiBi extrapolates for free.)

#Try It: Positional Encoding in Code

pythonplayground.py · Pyodide

import numpy as np

# Generate sinusoidal positional encodings
# and verify key properties

def positional_encoding(pos, d_model):
  pe = np.zeros(d_model)
  for i in range(0, d_model, 2):
      freq = 1.0 / (10000 ** (i / d_model))
      pe[i] = np.sin(pos * freq)
      if i + 1 &lt; d_model:
          pe[i + 1] = np.cos(pos * freq)
  return pe

d_model = 64

# Generate encodings for several positions
pe0 = positional_encoding(0, d_model)
pe1 = positional_encoding(1, d_model)
pe5 = positional_encoding(5, d_model)
pe8 = positional_encoding(8, d_model)
pe20 = positional_encoding(20, d_model)
pe23 = positional_encoding(23, d_model)

# Check: adjacent positions should have high similarity
print(f"Similarity pos 0 vs 1: {np.dot(pe0, pe1):.3f}")
print(f"Similarity pos 5 vs 8: {np.dot(pe5, pe8):.3f}")

# Check: relative position property
# Similarity between (5,8) should be close to (20,23) -- both 3 apart
print("\nRelative position test (distance = 3):")
print(f"Similarity pos 5 vs 8:   {np.dot(pe5, pe8):.3f}")
print(f"Similarity pos 20 vs 23: {np.dot(pe20, pe23):.3f}")
print("These should be very close!")

# Check: distant positions should have lower similarity
print(f"\nSimilarity pos 0 vs 100: {np.dot(pe0, positional_encoding(100, d_model)):.3f}")

RoFormer: Enhanced Transformer with Rotary Position Embedding

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu (2021)

Introduces RoPE, which encodes position by rotating Q and K vectors in 2D subspaces. The key insight is that rotation naturally makes attention scores depend on relative position (m-n) rather than absolute position.

Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin (2017)

Section 3.5 introduces sinusoidal positional encoding. The authors hypothesized that the sinusoidal form would allow the model to easily learn relative positions -- a hypothesis later validated by the success of RoPE.

Try it: Explore sinusoidal position fingerprintsInteractive

Loading visualization...

Explore this: Each row is a position, each column is a dimension of the encoding. Notice how lower-frequency waves (right side) change slowly between positions while high-frequency waves (left side) change rapidly. Any two positions always have a unique combination — this is how the model tells position 47 apart from position 48 in a 10,000-token sequence.

Try it: Word Embedding SpaceInteractive

Loading visualization...

Explore this: Find "king" and "queen" in the embedding space — subtract the "man" vector from "king" and add "woman" and you approximately land on "queen". This famous vector arithmetic shows that embeddings capture semantic relationships as geometric structure — the same structure that positional encodings must slot their position signals into without disrupting.

⚡ Playground: Positional Encoding → — see how sine and cosine waves at different frequencies give each position a unique fingerprint.

#See It All Together

Head to the Full Transformer lesson to see how positional encodings feed into the complete architecture. In the interactive Transformer visualization, positional encoding is the second step -- right after token embedding and before the first attention layer.

#Key Takeaways

Attention is permutation-invariant without positional information. Self-attention treats its input as an unordered set, so "the cat sat on the mat" and "mat the on sat cat the" produce identical outputs unless position is explicitly encoded
Sinusoidal encodings use different frequencies across dimensions. Low-frequency sinusoids encode coarse position, high-frequency ones encode fine position, and the model can learn relative positions through their interactions
RoPE has become the modern standard in LLMs. Rotary Position Embeddings encode position by rotating query and key vectors, naturally capturing relative position and offering better length extrapolation than absolute encodings
Length extrapolation remains an active research challenge. Training on sequences of length N but deploying on length 2N requires positional encodings that generalize beyond training; ALiBi and dynamic NTK-aware scaling address this

#Quick Check

Quick Check1 / 4

Why does the Transformer need positional encoding at all?

Key Terms8 terms

Property of self-attention: shuffling the input tokens shuffles the output in the same way. This is why Transformers have no built-in sense of order and require explicit positional information.

A vector added to (or combined with) each token embedding to tell the model where that token sits in the sequence. Can be fixed (sinusoidal) or learned.

Original Transformer PE from 'Attention Is All You Need'. Uses sin/cos at geometrically spaced frequencies so that each position gets a unique fingerprint and relative shifts become linear transformations.

A separate trainable embedding vector per position, used in BERT and early GPT. Simple but capped at the maximum training length — cannot extrapolate to longer sequences.

Modern positional scheme used in LLaMA, Mistral, Qwen, DeepSeek. Encodes position by rotating 2D subspaces of the query and key vectors, so Q . K naturally depends on relative distance (m - n).

Adds a linear penalty proportional to token distance directly to attention scores, with a different slope per head. Cheap and extrapolates remarkably well to longer contexts.

A model's ability to process sequences longer than any seen during training. RoPE with NTK-aware scaling or YaRN, and ALiBi, are current techniques for extending context windows.

During autoregressive generation, the keys and values from past tokens are cached so only the new token needs to be projected. RoPE is cache-friendly because rotations are applied once per position and stay valid.

Where This Matters

Anthropic

200K Context Windows at Anthropic

Claude's long context uses RoPE plus NTK-aware scaling / YaRN-like techniques to extend positional encodings far beyond training length, letting the model reason over entire books and large codebases.

↑

200K+ token context — whole books processed in one prompt

Meta / Mistral AI / Alibaba

LLaMA / Mistral / Qwen (Open-Source LLMs)

Every major open-source LLM family since 2023 has standardized on RoPE because it gives relative position for free, adds zero learnable parameters, and plays well with FlashAttention and KV caching.

↑

RoPE is the de-facto default positional encoding in 2024-2025

GitHub Copilot / Cursor

Long-Context Code Assistants

Code copilots load hundreds of thousands of tokens of repository context. Positional encodings that extrapolate (RoPE + scaling, ALiBi) are what makes whole-repo reasoning feasible without retraining per-context-size.

↑

Enables repo-wide refactors and cross-file reasoning

Interview Practice

We now have all the building blocks: attention, self-attention, multiple heads, and positional information. Time to assemble the full machine. Next up: the complete Transformer architecture.

Your Reflection

Continue Learning

Discussion