RugvAI Labs

Transfer Learning

Training BERT from scratch costs $7M. Fine-tuning it on your task costs $20. Transfer learning is the difference between writing a research paper and shipping a product. Take a model someone else trained on the entire internet, replace the last layer with your task head, train for a few hours on a thousand examples — and you've got state-of-the-art results for the price of a pizza.

Learning Objectives

After this lesson, you will be able to:

Understand why pretrained models already know useful things and how that knowledge transfers to new tasks
Distinguish feature extraction (frozen backbone) from fine-tuning (unfrozen weights) and explain when each is appropriate
Apply LoRA and explain why it fine-tunes large models with a fraction of the memory and compute of full fine-tuning
Pick the right transfer learning strategy given dataset size, domain similarity, and compute budget

Prerequisites

Before You Start

Review

Why

Why This Matters

How to train an AI with just 100 photos instead of millions -- take a pretrained model that already knows edges, textures, and shapes, and teach it your specific task; a model that would get 40% accuracy from scratch can hit 95% with transfer learning on just 100 examples
How people customize ChatGPT on a laptop using LoRA -- LoRA fine-tunes a massive model by training only 0.1% of its parameters; you can make a personal AI tutor or code assistant on a single laptop without expensive cloud GPUs
Why foundation models are the "app stores" of AI -- just like you build apps on iOS instead of writing your own operating system, developers build AI products on top of pretrained models; transfer learning is the skill of building on these platforms
Build this --> Build a custom image classifier: take a pretrained ResNet, freeze the early layers, replace the final layer with your own classes, fine-tune on just 50-100 photos (like "my dog" vs. "not my dog"), and compare accuracy with vs. without transfer learning to see the dramatic difference

Transfer learning is arguably the most practically important technique in this entire track. Almost nobody trains from scratch anymore. If you learn one thing from this lesson, let it be this: start with a pretrained model. Always. You will save days of work and get better results.

#End-to-End Transfer Learning Process

#Step 1: Pretrained Model (ImageNet)

Start with a model already trained on a massive dataset (e.g., ResNet-50 trained on 1.2 million ImageNet images across 1,000 classes). This model has learned a rich hierarchy of visual features: edges, textures, shapes, object parts, and full objects. These features represent months of GPU compute distilled into a set of weights.

#Step 2: Freeze Early Layers

The early layers (edge detectors, texture recognizers) are universal -- they transfer to almost any vision task. Freeze these layers by setting requires_grad=False. Their weights will not be updated during fine-tuning. This preserves the general visual knowledge and prevents overfitting when your dataset is small.

#Step 3: Replace Final Layer

The original output layer classifies ImageNet's 1,000 classes -- not useful for your task. Remove it and attach a new classification head matching your number of classes (e.g., 5 categories for medical imaging). This new layer starts with random weights and will learn from scratch during fine-tuning.

#Step 4: Fine-Tune on Your Data

Loading visualization...

Intuition

Build Your Intuition

Three fine-tuning strategies -- think of hiring an experienced chef for your restaurant.

Frozen Backbone (Feature Extraction): You hire a master chef (pretrained model) but do not let them change any recipes. You only train a new dessert menu (new output layer) using their existing cooking skills. Fast and cheap, but limited.

Full Fine-Tuning: You let the chef modify ALL recipes for your restaurant's local taste. Powerful, but risky -- they might forget their original skills (catastrophic forgetting), and you need lots of your own recipe data.

Adapter/LoRA Fine-Tuning: You keep all original recipes untouched but add small sticky notes (adapters) to each recipe with local modifications. The chef follows the original recipe plus the sticky notes. Efficient, preserves original knowledge, and only trains the sticky notes (2-5% of parameters).

#Stand on the Shoulders of Giants

Intuition

Build Your Intuition

Training a large neural network from scratch requires massive amounts of data and compute. GPT-4 was trained on trillions of tokens using thousands of GPUs for months. ImageNet-trained ResNets see millions of images. Few organizations have these resources.

But here is the key insight: the knowledge learned by these models is largely reusable. A vision model trained on ImageNet learns to detect edges, textures, shapes, and objects -- features that are useful for almost any image task. A language model trained on the internet learns grammar, facts, reasoning patterns, and writing styles -- useful for almost any text task.

Transfer learning means taking a model pretrained on a large dataset and adapting it to your specific, smaller task. Instead of learning from scratch (random weights), you start with a model that already "knows" about the world and teach it your specific problem. This typically requires 10-100x less data and compute than training from scratch.

This is the most practically important technique in modern deep learning. Almost no one trains from scratch anymore.

Try it! If you have Python and PyTorch installed, load a pretrained ResNet in 3 lines: import torchvision; model = torchvision.models.resnet18(pretrained=True); print(sum(p.numel() for p in model.parameters())). You just loaded 11 million parameters of learned visual knowledge. Now imagine training those from scratch -- transfer learning saved you weeks of GPU time.

#Why Transfer Works: The Feature Hierarchy

#In Vision

A CNN trained on ImageNet develops a hierarchy:

Early layers (layers 1-3): Edge detectors, color blobs, textures -- universal across all image domains
Middle layers (layers 4-7): Object parts, patterns -- somewhat task-specific but broadly useful
Late layers (layers 8+): Class-specific detectors -- most task-specific

When you transfer to a new task (e.g., classifying medical X-rays), the early layers' edge and texture detectors are immediately useful. The middle layers may need slight adjustment. Only the late layers need significant retraining.

#In Language

A language model like BERT or GPT learns:

Early layers: Token embeddings, basic syntax, POS-like features
Middle layers: Semantic relationships, coreference, basic reasoning
Late layers: Task-specific representations, complex reasoning

These features transfer to sentiment analysis, question answering, named entity recognition, summarization, and nearly any NLP task.

Try it: Freeze and unfreeze layers to see how transfer learning adapts a pretrained networkInteractive

Loading visualization...

#Strategy 1: Feature Extraction

Use the pretrained model as a fixed feature extractor. Freeze all pretrained weights and only train a new classifier head on top.

\text{features} = f_{\text{pretrained}}(x) \quad \text{(frozen)} \\[4pt] \hat{y} = g_{\text{new}}(\text{features}) \quad \text{(trainable)}

When to use: Small dataset (< 1000 examples), very different from pretraining domain, limited compute. Fast and unlikely to overfit.

#Strategy 2: Full Fine-Tuning

Unfreeze all layers and train the entire model on your new task, starting from pretrained weights.

\theta_{\text{fine-tuned}} = \theta_{\text{pretrained}} - \eta \sum_{t} \nabla L_{\text{new}}(\theta, x_t, y_t)

When to use: Moderate to large dataset (1000+ examples), task is related to pretraining domain, sufficient compute.

What Do You Think?

When fine-tuning a pretrained model, should you use the same learning rate as training from scratch?

#Discriminative Fine-Tuning

Different layers need different learning rates. Early layers contain general features that should change minimally; later layers contain task-specific features that need more adaptation:

\eta^{(l)} = \eta_{\text{base}} \cdot \alpha^{L-l}

#Strategy 3: Parameter-Efficient Fine-Tuning (PEFT)

Full fine-tuning a 7B-parameter LLM requires storing 7B gradients and optimizer states -- ~100GB of GPU memory. PEFT methods fine-tune only a small fraction of parameters, dramatically reducing memory and compute.

#LoRA (Low-Rank Adaptation)

W' = W + \Delta W = W + BA \\[4pt] \text{where } B \in \mathbb{R}^{d \times r}, \, A \in \mathbb{R}^{r \times d}, \, r \ll d

LoRA is the most popular PEFT method because:

Memory efficient: Only store and update ~0.1-1% of parameters
No inference overhead: Merged into base weights after training
Composable: Multiple LoRA adapters can be swapped for different tasks
Quality: Matches full fine-tuning on most benchmarks

#QLoRA: Quantized LoRA

Combine LoRA with 4-bit quantization of the base model. Fine-tune a 65B parameter model on a single 48GB GPU. This democratized LLM fine-tuning for individual researchers and small companies.

#LoRA: low-rank adaptation math

The qualitative pitch above -- "LoRA trains 0.1% of the parameters" -- is the marketing line. The actual math is short, elegant, and worth knowing if you ever need to debug a fine-tune, pick a rank, or estimate GPU memory. This section unpacks Hu et al. (2021) end to end.

#The premise: the update is low-rank

A pretrained model is a stack of weight matrices. For any single matrix W in R^{d_out x d_in} -- an attention projection, an MLP, an embedding -- fine-tuning replaces it with W + ΔW. Full fine-tuning learns every entry of ΔW directly, which means d_out * d_in new parameters per layer.

The Hu et al. (2021) hypothesis is that ΔW has low intrinsic rank: most of the meaningful change concentrates in a handful of directions, not the full d_out x d_in matrix. Pretraining has already built a dense, near-complete feature basis; fine-tuning mostly rotates and amplifies a few directions in that basis. If that hypothesis holds, we never have to materialize the full ΔW -- we only need a thin slice of it.

#The LoRA parameterization

Decompose ΔW into the product of two skinny matrices:

\Delta W = B A \quad\text{with}\quad A \in \mathbb{R}^{r \times d_{\text{in}}},\; B \in \mathbb{R}^{d_{\text{out}} \times r},\; r \ll \min(d_{\text{in}}, d_{\text{out}})

By construction rank(BA) <= r. Trainable parameter count drops from d_out * d_in to r * (d_out + d_in). That is the entire trick.

\frac{\text{params}_{\text{LoRA}}}{\text{params}_{\text{full}}} = \frac{r \,(d_{\text{out}} + d_{\text{in}})}{d_{\text{out}} \, d_{\text{in}}} = \frac{r}{d_{\text{out}}} + \frac{r}{d_{\text{in}}}

#Concrete count: LLaMA-7B attention projection

For W_Q in R^{4096 x 4096} (a single query projection in LLaMA-7B):

Full fine-tune: 4096 * 4096 = 16,777,216 params per matrix -- ~16.78M.
LoRA r=8: 8 * (4096 + 4096) = 65,536 params per matrix.
Reduction: 16,777,216 / 65,536 = 256x fewer trainable parameters, or equivalently 4r / (d_in + d_out) = 32 / 8192 ~ 0.4%.

Apply this across every attention projection in every layer and the cumulative savings become the headline "train 0.1% of params" number you see in blog posts.

#Initialization: identical to pretrained at t=0

LoRA's initialization is asymmetric on purpose:

A \sim \mathcal{N}(0, \sigma^{2}), \qquad B = 0 \;\;\Longrightarrow\;\; \Delta W \big|_{t=0} = BA = 0

This is the single most important practical detail. Many LoRA-like methods that initialize both matrices randomly suffer an initial loss spike because the model is no longer the pretrained model on step 0.

#Inference: keep separate or merge

At inference time you have two equivalent options:

W' = W + \frac{\alpha}{r} \, B A

Keep separate (forward = Wx + (alpha/r)(B(Ax))): two extra small matmuls per layer, but the same base model can serve many different LoRA adapters -- one for code, one for legal, one per customer. This is how LoRAX, S-LoRA, and vLLM's multi-LoRA server work.
Merge once (W_merged = W + (alpha/r) BA): identical FLOPs to the original pretrained model after the one-time merge. Pay zero inference cost, lose hot-swap.

#Memory: optimizer state is the real story

The "0.1% of params" headline undersells LoRA's actual win. The dominant memory cost during training is not parameters -- it is optimizer state. AdamW maintains two fp32 moment buffers per trainable parameter (m and v), which is 2 x 4 = 8 bytes per param, on top of the 4-byte param and 4-byte gradient. For a 7B model in mixed precision:

Component	Full fine-tune (7B)	LoRA r=8
Frozen base weights (bf16)	14 GB	14 GB
Trainable params (fp32 master)	28 GB	~16 MB
Gradients (fp32)	28 GB	~16 MB
AdamW `m, v` state (fp32, 2x)	56 GB	~32 MB
Activations (per batch)	~10-30 GB	~10-30 GB
Total optimizer overhead	~112 GB	~0.06 GB

The base weights are frozen, so they need no optimizer state, no gradient buffer, no fp32 master copy. That is where the 100x training-memory reduction comes from -- not the headline parameter count.

#QLoRA: 4-bit base + bf16 adapters

QLoRA (Dettmers et al. 2023) pushes this further by storing the frozen base in NF4 (a 4-bit data type optimized for normally-distributed weights) and double-quantizing the quantization constants themselves. The LoRA adapters stay in bf16 because they actually receive gradients. The combination:

7B base: 14 GB bf16 -> ~3.5 GB NF4
65B base: 130 GB bf16 -> ~33 GB NF4 -> fits on a single 48GB A6000 with room for adapters and activations
Accuracy cost: roughly 1% degradation versus full bf16 fine-tuning on GLUE/MMLU

QLoRA is the standard recipe today for fine-tuning anything above 30B on consumer or single-GPU server hardware.

#DoRA: separate magnitude from direction

DoRA (Liu et al. 2024) noticed that a learned weight update has two parts: it changes the magnitude of each row and it rotates each row's direction. The decomposition is:

W = m * (V / ||V||_c) -- a per-column magnitude scalar m times a unit-norm direction.

DoRA freezes the magnitudes (or learns them separately) and applies LoRA only to the direction matrix V. Empirically this matches full fine-tune on tasks where vanilla LoRA leaves a couple of points on the table. Same parameter budget, slightly better quality, marginally more compute per step.

#Why r=8 works in practice

Hu et al. (2021) swept r in {1, 2, 4, 8, 16, 32, 64} on GLUE (text classification) and E2E NLG (text generation) and found a striking result: r=8 already captures 98%+ of full fine-tune performance, and going higher gives diminishing returns. The intuition lines up with the IntuitionBox above: pretrained representations are already near-complete; fine-tuning amplifies and rotates a small number of existing directions rather than building new ones. Common picks today:

r=4 or r=8: classification, simple instruction tuning
r=16: general chat fine-tuning
r=32 to r=64: domain shifts (code, math, scientific reasoning) where the task is meaningfully different from pretraining
alpha = 2*r: a common heuristic so that alpha/r = 2

Loading visualization...

What Do You Think?

Fine-tune Llama-3-8B with LoRA r=16 on every attention projection. The model has 32 transformer layers, and each layer has 4 attention matrices (Q, K, V, O) of shape 4096x4096. How many trainable LoRA parameters total?

Watch Out

LoRA makes inference faster

This is false. A common misconception: "LoRA is parameter-efficient, so it must run faster too." LoRA does not change inference FLOPs at all. If you keep adapters separate, you actually do slightly more work per forward pass (two extra small matmuls per LoRA-targeted layer). If you merge W_merged = W + (alpha/r) BA, the merged model has identical shape, identical FLOPs, and identical latency to the original pretrained model -- no win, no loss.

What LoRA actually saves:

Training memory: ~100x less optimizer state, gradient buffers, fp32 master weights.
Training compute: smaller gradient computations, smaller AdamW updates.
Disk: a 7B LoRA checkpoint is ~10-50 MB versus ~14 GB for the full base.
Serving cost when multi-tenanting: one base in GPU memory, many tiny adapters hot-swapped at request time.

What LoRA does not save: per-token inference latency, per-token inference FLOPs, KV cache size, activation memory. If your bottleneck is "how fast can I generate one token from one model," LoRA does nothing for you -- you need quantization, speculative decoding, or a smaller model.

Watch Out

Common Mistake

Fine-tuning too aggressively: Using a large learning rate or training for too many epochs can cause catastrophic forgetting -- the model loses its pretrained knowledge. The fine-tuned model performs well on the new task but forgets general capabilities. Use small learning rates (1e-5 to 5e-5 for language models) and early stopping.

Not freezing batch norm during fine-tuning: Batch normalization layers have running statistics from pretraining. Fine-tuning with small batches can corrupt these statistics. Best practice: freeze batch norm layers or use a large enough batch size.

Choosing the wrong strategy: Feature extraction when you have enough data for fine-tuning (leaving performance on the table). Full fine-tuning when your dataset is tiny (overfitting). LoRA when the task is very different from pretraining (may need more adaptation).

Watch Out

Common Mistake

Confusing feature extraction with zero-shot inference: Feature extraction means running data through a frozen pretrained model to get embedding vectors, then training a new classifier on those vectors. Zero-shot inference means using the model's output directly without any training. They are different approaches with different accuracy profiles. Feature extraction almost always outperforms zero-shot when even 50-100 labeled examples are available.

Applying LoRA to the wrong module types: LoRA is most effective when applied to attention projection matrices (q_proj, k_proj, v_proj, o_proj) in transformer models. Applying it only to the final linear layer or to embedding tables often underperforms full fine-tuning significantly. Most LoRA tutorials target all attention projections by default -- this is the right starting point. When in doubt, target all linear layers and reduce rank if memory is constrained.

#Decision Framework

Your Situation	Strategy	Why
Tiny dataset (< 100 examples)	Feature extraction + linear probe	Cannot afford to overfit
Small dataset (100-1000)	Feature extraction or LoRA	Low risk of overfitting
Medium dataset (1K-100K)	LoRA or full fine-tuning	Enough data to adapt
Large dataset (100K+), similar domain	Full fine-tuning	Can fully adapt without overfitting
Large dataset, very different domain	Full fine-tuning with lower lr for early layers	Need to significantly change features
Limited GPU memory	QLoRA	Fine-tune large models on small GPUs
Multiple tasks on one base model	LoRA per task	Swap adapters cheaply

#Step 1: Choose Your Pretrained Model

Select a model pretrained on a task similar to yours. For vision: ResNet, EfficientNet, or ViT pretrained on ImageNet. For text: BERT, RoBERTa, or LLaMA. For code: CodeLlama or StarCoder. The closer the pretraining data is to your task, the better transfer works.

#Step 2: Assess Your Dataset

How large is your labeled dataset? How similar is it to the pretraining data? Small + similar = feature extraction. Large + different = full fine-tuning. Everything in between = LoRA.

#Step 3: Prepare the Model

Replace the output head to match your task (number of classes, regression vs. classification). Decide which layers to freeze. Set layer-specific learning rates if using discriminative fine-tuning.

#Step 4: Train with Care

Use a small learning rate (10-100x smaller than training from scratch). Monitor validation loss for early stopping. Use learning rate warmup for the first 5-10% of training. Track both task performance and general capability (if applicable).

Step 5: Evaluate Thoroughly

#Try It Yourself

pythonplayground.py · Pyodide

# Simulate transfer learning vs training from scratch
# See how pretrained features accelerate learning
import numpy as np

def simulate_training(start_loss, learning_rate, noise, epochs):
  loss = start_loss
  losses = [loss]

for e in range(1, epochs + 1):
      gradient = -0.3 * loss + (np.random.rand() - 0.5) * noise
      loss = max(0.05, loss + learning_rate * gradient)
      losses.append(loss)
  return losses

epochs = 50

# From scratch: high starting loss, slow convergence
scratch = simulate_training(4.5, 0.05, 1.0, epochs)

# Feature extraction: pretrained features, only train head
feature_extract = simulate_training(2.0, 0.1, 0.3, epochs)

# Full fine-tuning: pretrained start, careful learning
fine_tune = simulate_training(1.5, 0.03, 0.4, epochs)

# LoRA: efficient adaptation
lora = simulate_training(1.8, 0.05, 0.35, epochs)

print("=== Transfer Learning Comparison ===\n")
print("Epoch | Scratch | Feature Ext | Fine-Tune | LoRA")
print("------|---------|-------------|-----------|-----")

for e in range(0, epochs + 1, 5):
  print(f"{e:5d} | {scratch[e]:7.3f} | {feature_extract[e]:11.3f} | "
        f"{fine_tune[e]:9.3f} | {lora[e]:5.3f}")

print("\nKey observations:")
print("1. From scratch starts with highest loss and converges slowest")
print("2. Pretrained methods start with lower loss (pretrained knowledge!)")
print("3. Fine-tuning often achieves the lowest final loss")
print("4. Feature extraction is fastest but may plateau higher")
print("\nCHALLENGE: What happens if you increase the noise for")
print("fine-tuning (simulating a very different target domain)?")

Tests · Verify pretrained methods achieve lower loss than from-scratch at epoch 10. Verify fine-tuning achieves lowest final loss.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova (2018)

Established the pretrain-then-fine-tune paradigm for NLP. BERT's bidirectional pretraining produced representations that achieved state-of-the-art results on 11 NLP tasks with minimal task-specific modification. Over 100,000 citations.

LoRA: Low-Rank Adaptation of Large Language Models

Edward Hu et al. (2021)

Introduced low-rank adaptation for efficient fine-tuning. LoRA reduces trainable parameters by 10,000x while matching full fine-tuning quality. Became the standard approach for adapting large language models.

#Key Takeaways

Pretrained models contain reusable knowledge. Features learned on large datasets (ImageNet, Common Crawl) transfer to new tasks because early layers learn universal patterns (edges, syntax) that apply broadly
Fine-tuning adapts a pretrained model to your specific task. Freeze early layers (universal features), unfreeze later layers (task-specific features), and train on your smaller dataset with a low learning rate
LoRA and adapters enable parameter-efficient fine-tuning. Instead of updating all weights, these methods learn small low-rank updates, reducing memory and compute by 10-100x while maintaining most of the performance
Dataset size and domain similarity determine the strategy. Small dataset + similar domain: feature extraction (freeze everything); large dataset + different domain: full fine-tuning with low learning rate; limited compute: LoRA

#Quick Check

Quick Check1 / 4

Why does transfer learning work -- what makes pretrained features reusable?

Transfer learning is the most practical technique in modern deep learning. Next: Practical Deep Learning -- a decision framework for when to use deep learning vs. classical ML, and how to avoid the common trap of reaching for a neural network when a simpler model would do.

Your Reflection

Continue Learning

Discussion

Transfer Learning

#End-to-End Transfer Learning Process

#Step 1: Pretrained Model (ImageNet)

#Step 2: Freeze Early Layers

#Step 3: Replace Final Layer

#Step 4: Fine-Tune on Your Data

#Stand on the Shoulders of Giants

#Why Transfer Works: The Feature Hierarchy

#In Vision

#In Language

#Strategy 1: Feature Extraction

#Strategy 2: Full Fine-Tuning

#Discriminative Fine-Tuning

#Strategy 3: Parameter-Efficient Fine-Tuning (PEFT)

#LoRA (Low-Rank Adaptation)

#QLoRA: Quantized LoRA

#LoRA: low-rank adaptation math

#The premise: the update is low-rank

#The LoRA parameterization

#Concrete count: LLaMA-7B attention projection

#Initialization: identical to pretrained at t=0

#Inference: keep separate or merge

#Memory: optimizer state is the real story

#QLoRA: 4-bit base + bf16 adapters

#DoRA: separate magnitude from direction

#Why r=8 works in practice

#Decision Framework

#Step 1: Choose Your Pretrained Model

#Step 2: Assess Your Dataset

#Step 3: Prepare the Model

#Step 4: Train with Care

Step 5: Evaluate Thoroughly

#Try It Yourself

Deep Dive

#Key Takeaways

#Quick Check

Related Lessons

#Step 5: Adapted Model