RugvAI Labs

Evaluating Generative Models: FID, CLIP & Beyond

Midjourney v7 vs v6, FLUX vs SDXL, Sora vs Veo 3 — how do labs prove their new model is actually better? FID for image quality, CLIP Score for prompt fidelity, GenEval and HEIM for compositional reasoning, LMSYS arena for human preference. Without these metrics, every "state-of-the-art" claim is just vibes. With them, you can audit any model launch in 15 minutes.

Learning Objectives

After this lesson, you will be able to:

Understand what FID measures (how similar generated images are to real ones) and why lower is better
Compare the main evaluation metrics: FID for quality, CLIP Score for prompt-following, and human evaluation
Spot failure modes like mode collapse and know which metrics catch them
Pick the right metric for your specific generative task

Prerequisites

Before You Start

Review

#The Fundamental Challenge

Evaluation might sound like the boring part, but it is actually one of the hardest open problems in generative AI. How do you objectively measure whether an image is "good"? This lesson gives you the tools that the entire field uses to compare models, and understanding them will help you critically evaluate any AI art tool's claims.

Intuition

Comparing generative models across three key metrics

Consider three models evaluated on the same benchmark:

Model	FID (lower = better)	Recall (higher = better)	CLIP Score (higher = better)
Model A	8	0.42	0.32
Model B	25	0.61	0.28
Model C	45	0.55	0.35

The ranking changes depending on what you optimize for. If you care about overall quality (how statistically similar the outputs are to real images), Model A wins by a large margin — FID of 8 is near state-of-the-art. But Model A has the worst recall (0.42) — it covers less of the data manifold, a classic mode-collapse signature. If you care about text-image alignment (how faithfully the model follows prompts), Model C surprisingly wins with CLIP Score 0.35, despite having the worst FID. Model B sits in the middle on quality but covers the most modes.

This illustrates the core tradeoff: a model can generate highly realistic images that do not match the prompt (high quality, low alignment), or it can follow prompts faithfully but produce less photorealistic results (high alignment, lower quality). The right metric to optimize depends entirely on your use case.

#Frechet Inception Distance (FID)

Try it! Generate 10 images with the same prompt using any AI art tool. Rate each one 1-10 for quality. Now ask a friend to rate them independently. Compare your scores. The disagreement you see is exactly why human evaluation is noisy and why automated metrics like FID are so valuable -- they give consistent, reproducible scores every time.

FID is the most widely used metric for evaluating generative image models. It compares the distribution of real images to the distribution of generated images.

#How FID Works

Pass all real images through a pretrained Inception-v3 network and extract features from the penultimate layer (a 2048-dimensional vector per image)
Compute the mean and covariance of these feature vectors for the real set
Do the same for a set of generated images
Measure how different these two multivariate Gaussian distributions are

\text{FID} = \|\mu_r - \mu_g\|^2 + \text{Tr}\left(\Sigma_r + \Sigma_g - 2(\Sigma_r \Sigma_g)^{1/2}\right)

Where:

mu_r, mu_g are the mean feature vectors for real and generated images
Sigma_r, Sigma_g are the covariance matrices of those features
Tr is the matrix trace (sum of diagonal elements)
Lower FID = better (0 means the distributions are identical)

#FID Reference Points

FID Score	Interpretation
0-5	State-of-the-art (nearly indistinguishable from real)
5-20	Very good quality, minor statistical differences
20-50	Noticeable quality gaps, useful for research
50-100	Clearly distinguishable from real images
100+	Poor quality or fundamentally different distribution

What Do You Think?

Model A has FID=10, Model B has FID=50. Which produces more realistic images?

The answer is Model A. Lower FID means the distribution of generated images more closely matches the distribution of real images. An FID of 10 indicates the model's outputs are statistically very similar to real images in terms of both quality and diversity.

#Precision & Recall: Decomposing Quality from Mode Coverage

FID conflates two very different failure modes into one scalar. A model that produces 100% high-quality images but only ever generates golden retrievers (when the real data has 120 dog breeds) can post a low FID by being statistically close on the dimensions the Inception features capture — while completely missing 119 of the 120 modes. Kynkäänniemi et al. 2019, "Improved Precision and Recall Metric for Generative Models" introduced separate precision/recall metrics that pull these failure modes apart and have become the modern complement to FID.

The intuition is geometric. Build a k-NN manifold estimate of the real-data feature distribution: for each real point r_i, draw a hypersphere out to its k-th nearest neighbor. The union of these spheres is the empirical "real manifold." Do the same for generated samples to get the "generated manifold."

\text{precision} = \frac{1}{|G|}\sum_{g \in G}\, \mathbb{1}\big[g \in \mathcal{M}_R\big] \qquad \text{recall} = \frac{1}{|R|}\sum_{r \in R}\, \mathbb{1}\big[r \in \mathcal{M}_G\big]

Why this matters for modern evaluation: FID can be deceptively low while a model has catastrophic mode collapse. Imagine a GAN that always produces the same 10 sharp, realistic faces. The mean and covariance in Inception feature space may still be close to the real distribution if those 10 faces happen to sit near the centroid — FID looks fine, the model is broken. Precision will be ~1.0 (every sample is realistic), recall will be ~0.02 (you cover ~2% of the actual face manifold). The diagnostic is unambiguous in PR space and invisible in FID alone. This is why Kynkäänniemi-style precision/recall — and its 2020+ refinements (Naeem et al.'s density/coverage, Sajjadi et al.'s PR curves) — appear next to FID in every serious paper since 2020.

Quick check

A GAN posts precision ≈ 0.92 and recall ≈ 0.18. What is it doing?

#The 2025-2026 Metrics That Actually Matter

FID is a starting point, not a finish line. The frontier-lab playbook for text-to-image and video has moved on to compositional, preference-trained, and Elo-based evaluations that better correlate with what users actually want. Pick a basket of these — no single metric is sufficient on its own.

#GenEval (Ghosh et al. 2023): compositional T2I evaluation

GenEval evaluates a text-to-image model on prompts that test specific compositional skills: object presence, counting (e.g., "three cats"), color (e.g., "a red apple and a blue mug"), position (e.g., "a dog left of a cat"), and attribute binding (e.g., "a red cube on a green sphere"). For each prompt, an off-the-shelf object detector verifies the property automatically. GenEval became the de-facto compositional benchmark for SD3 / FLUX / DALL-E 3 head-to-heads because it surfaces failures FID is blind to — a model can have a great FID and still systematically fail "three apples on a blue plate."

#HPSv2: Human Preference Score v2

HPSv2 is a CLIP-style model fine-tuned on the HPDv2 dataset of 798K human preference pairs spanning DALL-E 2, SD 1.5, SDXL, Midjourney v5, and others. Given a prompt and a generated image, HPSv2 outputs a scalar that correlates strongly with which image humans pick in pairwise tests. It became the standard during the diffusion-DPO / direct preference optimization era — "improve HPSv2" is now a default training signal, not just an eval.

#PickScore (Kirstain et al. 2023)

A sibling preference-trained scorer trained on the Pick-a-Pic dataset of 1M+ real user preference pairs collected from a public T2I playground. Closely related to HPSv2 and often reported alongside it; PickScore is the metric Stable Diffusion fine-tuning papers (DPO-SDXL, Diffusion-DPO) commonly use as their preference reward.

#ImageReward (Xu et al. 2023)

A BLIP-based reward model trained on 137K expert ratings across text-image alignment, fidelity, and harmlessness axes. Widely used as an RLHF reward signal for diffusion fine-tuning; included in the standard eval basket together with HPSv2 and PickScore so a model isn't gamed against just one preference distribution.

#VBench (Huang et al. 2023, ongoing)

The video equivalent of GenEval. VBench scores video generators across 16 axes including subject consistency, background consistency, temporal flickering, motion smoothness, dynamic degree, object class accuracy, multiple-object handling, color, spatial relationship, scene, and overall video-text alignment. It's how Sora, Veo, Kling, Pika, and HunyuanVideo are compared in published leaderboards.

#LMSYS Image Arena: Elo at scale

Modeled after the famous LMSYS Chatbot Arena, the Image Arena shows users two anonymized images for the same prompt and asks them to pick a winner. Thousands of votes per day produce an Elo ranking that, like the LLM arena, has become the single most cited community benchmark because it is hard to game, blind, and rooted in real human preferences. Its limitation: prompts skew toward what arena users like, and the model identity is hidden but the style often isn't.

#FID: still useful, but Goodhart-prone

FID is fine as a distribution-similarity check and is what you'd report for unconditional generation (e.g., CIFAR-10, FFHQ benchmarks). For modern T2I/T2V however it's increasingly Goodharted: teams optimize directly against FID-on-COCO and end up with models that look statistically right but are obviously worse to humans. Keep it on your dashboard, don't make it your North Star.

#CLIPScore: careful of reward hacking

CLIPScore was the original "does the image match the prompt?" metric. The problem: CLIP itself has biases and is trivially gamed by adding text labels into the image or matching CLIP's lexical preferences rather than the prompt's semantics. Modern pipelines either retire it in favor of HPSv2 / PickScore / ImageReward, or report it alongside them so any one reward model can't be overfitted in isolation.

Quick check

You release a new T2I model in late 2026 and want a contamination-resistant headline number. Which evaluation should you weight most heavily?

#CLIP Score: Text-Image Alignment

For text-to-image models like DALL-E, Stable Diffusion, and Midjourney, quality and diversity are not enough. The generated image must also match the text prompt. CLIP Score measures this alignment.

#How CLIP Score Works

Encode the text prompt with CLIP's text encoder to get a text embedding
Encode the generated image with CLIP's image encoder to get an image embedding
Compute cosine similarity between the two embeddings

A higher CLIP Score means the image better matches the text description. This is the same contrastive learning objective CLIP was trained on -- matching images with their captions.

#When to Use Each Metric

Metric	Measures	Best For
FID	Distribution similarity to real images	Unconditional benchmarks (CIFAR, FFHQ); sanity check
GenEval	Compositional skills (count, color, position, binding)	Headline T2I capability number
HPSv2 / PickScore / ImageReward	Learned human preference	RLHF reward, fine-tuning signal, eval basket
VBench	Multi-axis video quality	Video model leaderboards
LMSYS Image Arena Elo	Blind pairwise human preference	Hardest-to-game North Star
CLIP Score	Text-image alignment (legacy)	Quick sanity check; report alongside HPSv2
Human Eval	Everything (subjective)	Final validation, ablation studies

Intuition

Choosing the right evaluation metric — think of judging a cooking competition.

FID (Frechet Inception Distance): Compare the overall "feel" of the competition menu to a Michelin-star restaurant menu. Are the distributions of quality similar? Low FID = your generated food is statistically indistinguishable from real gourmet food. Good aggregate check; chefs eventually learn to cook for the FID judge specifically.

GenEval: A checklist judge — "you said three apples, did you make three apples?" Catches the cases where the food looks great but doesn't match the recipe.

HPSv2 / PickScore / ImageReward: A panel of chefs that has tasted millions of dishes and learned what diners actually like. Cheaper than real diners, but if you teach to this test the panel starts agreeing with itself in suspiciously narrow ways.

LMSYS Image Arena: A live blind taste test in the dining room every night. Hardest to game — diners are fresh, votes are anonymous, and no one is allowed to read the recipe.

LPIPS (Learned Perceptual Image Patch Similarity): How similar does one specific dish LOOK to the reference dish? Useful for paired tasks like image editing or super-resolution where you have a ground truth.

Rule of thumb: Treat FID as a smoke test, GenEval / HPSv2 / PickScore / ImageReward as your fine-tuning compass, and Arena Elo as the final exam.

Try it: Adjust thresholds and see metric trade-offsInteractive

Loading visualization...

#Human Evaluation: The Gold Standard

Automated metrics are proxies. The ultimate test is whether humans find the outputs compelling. Common human evaluation protocols include:

Two-alternative forced choice (2AFC): Show a person two images (one from Model A, one from Model B) and ask which is better
Mean Opinion Score (MOS): Rate images on a 1-5 scale for quality, realism, and prompt adherence
Elo ratings: Run a tournament where images from different models compete head-to-head, building a ranking (this is how Chatbot Arena works for LLMs)

#The Tradeoff

Human evaluation is the gold standard but it is expensive (hiring annotators), slow (days to weeks for large studies), and noisy (people disagree). In practice, teams use automated metrics for rapid iteration during training and reserve human evaluation for milestone comparisons.

#Detecting Mode Collapse

Mode collapse occurs when a generative model produces only a limited variety of outputs. Detection strategies include:

High precision but low recall (Kynkäänniemi 2019): Samples that fall sit are inside the real-data manifold (high precision = realistic), but only a small fraction of the manifold is covered (low recall = limited diversity). This is the most direct modern signal for mode collapse and has largely replaced "high IS, high FID" diagnostics.
Low variance in feature space: Extract DINOv2 or Inception features from many generated samples and measure their spread. Low variance means the model is stuck in a few modes.
Nearest-neighbor analysis: For each generated image, find its nearest neighbor in the generated set. If many images are near-duplicates, mode collapse is occurring.
GenEval per-category dropouts: A model that generates beautiful single objects but fails every "two-object" or counting prompt is exhibiting compositional mode collapse — common in older or over-distilled models.

#Try It Yourself

pythonplayground.py · Pyodide

import numpy as np

# Compute a simplified diversity score for a set of generated outputs
# In practice this would use Inception features; here we use raw "feature vectors"

def diversity_score(feature_vectors):
  n = len(feature_vectors)
  if n &lt; 2:
      return 0.0

vecs = np.array(feature_vectors)
  total_distance = 0.0
  pairs = 0

for i in range(n):
      for j in range(i + 1, n):
          cos_sim = np.dot(vecs[i], vecs[j]) / (
              np.linalg.norm(vecs[i]) * np.linalg.norm(vecs[j]) + 1e-8
          )
          total_distance += 1 - cos_sim  # cosine distance
          pairs += 1

return total_distance / pairs

# Diverse set: 5 different "images" with varied features
diverse_set = [
  [1.0, 0.2, 0.0, 0.5],
  [0.1, 0.9, 0.3, 0.1],
  [0.3, 0.1, 0.8, 0.2],
  [0.0, 0.5, 0.2, 0.9],
  [0.7, 0.0, 0.6, 0.1],
]

# Mode-collapsed set: 5 nearly identical "images"
collapsed_set = [
  [0.9, 0.1, 0.2, 0.3],
  [0.91, 0.09, 0.21, 0.29],
  [0.89, 0.11, 0.19, 0.31],
  [0.90, 0.10, 0.20, 0.30],
  [0.92, 0.08, 0.22, 0.28],
]

print(f"Diverse set score: {diversity_score(diverse_set):.4f}")
print(f"Collapsed set score: {diversity_score(collapsed_set):.4f}")
print("\nHigher = more diverse. Mode collapse = low score.")

# CHALLENGE: Create a set of 5 vectors with maximum diversity.
# What is the theoretical maximum diversity score?

Tests · Verify diverse set has higher score than collapsed set. Verify collapsed set score is close to 0.

#CodePlayground: Computing FID Conceptually

pythonplayground.py · Pyodide

import numpy as np

# CONCEPTUAL FID COMPUTATION
# In practice, features come from Inception-v3's penultimate layer (2048-dim).
# Here we simulate the entire pipeline with small feature vectors.

def compute_fid(real_features, generated_features):
  """
  Compute Frechet Inception Distance between two sets of features.
  
  FID = ||mu_r - mu_g||^2 + Tr(Sigma_r + Sigma_g - 2*sqrt(Sigma_r @ Sigma_g))
  
  For 1D features, this simplifies to:
  FID = (mu_r - mu_g)^2 + sigma_r^2 + sigma_g^2 - 2*sigma_r*sigma_g
      = (mu_r - mu_g)^2 + (sigma_r - sigma_g)^2
  """
  real = np.array(real_features)
  gen = np.array(generated_features)
  
  # Step 1: Compute statistics of real features
  mu_r = np.mean(real, axis=0)
  sigma_r = np.std(real, axis=0)
  
  # Step 2: Compute statistics of generated features
  mu_g = np.mean(gen, axis=0)
  sigma_g = np.std(gen, axis=0)
  
  # Step 3: FID = difference in means + difference in spread
  # (Simplified for diagonal covariance -- real FID uses full covariance matrix)
  mean_diff = np.sum((mu_r - mu_g) ** 2)
  std_diff = np.sum((sigma_r - sigma_g) ** 2)
  fid = mean_diff + std_diff
  
  return fid, mu_r, sigma_r, mu_g, sigma_g

# Simulate "Inception features" for different quality generators
rng = np.random.default_rng(42)

# Real image features: centered at [5, 3, 7], moderate spread
real_features = rng.normal([5.0, 3.0, 7.0], [1.0, 0.8, 1.2], size=(100, 3))

# GOOD generator: similar distribution to real
good_gen = rng.normal([5.1, 2.9, 7.1], [1.1, 0.9, 1.1], size=(100, 3))

# MEDIOCRE generator: shifted mean, different spread
mediocre_gen = rng.normal([4.0, 3.5, 6.0], [1.5, 1.2, 1.8], size=(100, 3))

# BAD generator (mode collapse): tight cluster far from real
bad_gen = rng.normal([3.0, 5.0, 4.0], [0.2, 0.2, 0.2], size=(100, 3))

# NOISE generator: random features
noise_gen = rng.normal([0.0, 0.0, 0.0], [3.0, 3.0, 3.0], size=(100, 3))

print("=== FID Score Comparison ===")
print("Lower FID = closer to real data distribution\n")

generators = {
  "Good Generator": good_gen,
  "Mediocre Generator": mediocre_gen,
  "Bad (Mode Collapse)": bad_gen,
  "Random Noise": noise_gen,
}

for name, gen in generators.items():
  fid, mu_r, sig_r, mu_g, sig_g = compute_fid(real_features, gen)
  print(f"{name:25s} FID = {fid:7.2f}")
  print(f"  Mean shift:    real={mu_r.round(2)} gen={mu_g.round(2)}")
  print(f"  Spread diff:   real={sig_r.round(2)} gen={sig_g.round(2)}")
  print()

# Demonstrate: FID is sensitive to both quality AND diversity
print("=== Why FID Catches Mode Collapse ===")
print("The 'Bad' generator has LOW spread (mode collapse).")
print("Even if individual samples look 'OK', the distribution is wrong.")
print("FID penalizes both mean shift AND spread mismatch.\n")

# Demonstrate: sample size matters
print("=== Sample Size Sensitivity ===")
for n in [10, 50, 100, 500]:
  subset_real = real_features[:n]
  subset_gen = good_gen[:n]
  fid_val, _, _, _, _ = compute_fid(subset_real, subset_gen)
  print(f"  n={n:4d} samples: FID = {fid_val:.2f}")
print("FID stabilizes with more samples. Use n >= 10,000 in practice.")
print()
print("CHALLENGE: Create a generator that achieves FID &lt; 0.5.")
print("(Hint: match both the mean AND standard deviation of real features.)")

Tests · Verify Good Generator has the lowest FID. Verify mode-collapsed generator has higher FID than mediocre. Verify FID decreases as the generated distribution approaches the real distribution.

#Key Takeaways

FID compares distributions, not individual images -- it measures how statistically similar a set of generated images is to a set of real images using Inception network features, with lower scores indicating closer match
Precision/recall replace the old Inception Score -- precision (quality) and recall (mode coverage) decompose what FID measures jointly, catching mode collapse FID can hide. Classic Inception Score is deprecated; use FID + DINOv2-FID + precision/recall instead.
CLIP Score bridges text and image evaluation -- essential for text-to-image models, it measures how well generated images align with their text prompts using CLIP's shared embedding space
No single metric is sufficient -- FID for overall quality, CLIP Score for alignment, and human evaluation for final validation form a comprehensive evaluation stack

#Quick Check

Quick Check1 / 4

What does a lower FID score indicate?

Watch Out

Common Mistake

Comparing FID scores across different papers requires caution. FID is sensitive to the number of generated samples (use at least 50,000), the reference dataset (FID on CIFAR-10 vs ImageNet are not comparable), the Inception model version, and even image preprocessing (resize method, center crop vs full image). Always check that you are comparing FID scores computed under identical conditions. A "better" FID in one paper may simply use different evaluation settings.

A second common mistake is treating CLIP Score as a comprehensive quality metric for text-to-image models. CLIP Score only measures whether the generated image and the text prompt end up near each other in CLIP's embedding space -- it does NOT measure photorealism, fine-grained spatial accuracy, or whether the model correctly rendered object relationships ("a red cube on top of a blue sphere" vs "a blue cube on top of a red sphere" may score similarly). Always pair CLIP Score with FID for quality and ideally with human preference evaluation for alignment nuance.

A third mistake: running FID with fewer than 10,000 samples and reporting the result as definitive. With small sample sizes, the Gaussian approximation that FID uses for the distribution statistics is unreliable, and the score is highly variable (can swing by 10-30 points between runs). Use the clean-fid library (Parmar et al., 2021) which standardizes preprocessing, and always report the sample count alongside the FID value.

Now you can measure how good generative models are. Next: The Ethics of Generative AI -- deepfakes, copyright, consent, bias, and environmental cost; the challenges every AI practitioner must understand before deploying generative systems.

Your Reflection

Continue Learning

Discussion