Generative Evaluation in Depth

#FID in Depth: The Fréchet Distance

The basics lesson gave FID as "gap between centers plus gap between spreads". This section shows why that is the right form. The formula is the squared Fréchet distance, also called the 2-Wasserstein distance, between two Gaussians: the least average squared distance you must move a sample from one Gaussian to turn the cloud into the other. In one dimension, and in any dimension where the two covariance matrices commute, you can derive it with tools from earlier lessons. For general covariances the same formula holds (Dowson and Landau, 1982) and is the one Heusel et al. (2017) use for FID; we state that case and check it numerically below.

#KID, DINOv2-FID and FVD: Other Distance Scores

FID is not the only distance between two sets of features. The recipe is always the same: choose a feature network, describe each set by its features, and compare the two sets. What changes is the network or the comparison.

• KID (Kernel Inception Distance) keeps the Inception features but compares the two sets with a kernel (a smooth similarity function) instead of fitting two bell curves. Its estimate is unbiased, so it is not pushed up by a small sample the way FID is, which makes it a common choice when you can only afford a few thousand images (Bińkowski et al., 2018).
• DINOv2-FID keeps the FID formula but takes features from DINOv2. In their tests, Stein et al. (2023) reported that these features agree better with human judgement of image quality than Inception features do.
• FVD (Fréchet Video Distance) keeps the FID formula but takes features from a video network (I3D), so it also sees motion and consistency over time.

None of them replaces the others. Always report which feature network and how many samples you used.

#Precision and Recall in Depth: The Nearest-Neighbour Test

The basics lesson gave the picture. Here is how it is computed. Build a k-nearest-neighbour (k-NN) estimate of the real-data feature cloud: for each real point r_i, draw a hypersphere (a ball in many dimensions) out to its k-th nearest neighbour. The union of these balls is the empirical "real manifold". Do the same for generated samples to get the "generated manifold".

Reading the formula: precision is the fraction of generated samples g that fall inside the real manifold M_R. Recall is the fraction of real samples r that fall inside the generated manifold M_G. Sajjadi et al. (2018) proposed precision-recall curves first, Kynkäänniemi et al. (2019) gave the k-NN version above, and Naeem et al. (2020) added density and coverage. They appear next to FID in many papers.

• Do: Move the "Clumps the generator never makes" slider to 2, then press Reset and squeeze the spread to 0.4. Switch between "Show precision" and "Show recall" after each change.
• Notice: With all three clumps and matching spread, precision and recall are both 97% at k = 3. With two clumps dropped, precision stays at 95% while recall falls to 31%. Squeezing the spread to 0.4 alone gives precision 100% and recall 48%. Raising k from 1 to 10 lifts both scores toward 100% with no change to the points.
• Why: Realistic fakes sit inside real balls, so dropping variety does not touch precision, but every real point whose neighbourhood the generator never visits goes uncovered, so recall falls. Bigger k means bigger balls, which is why a paper must state k. A generator that copied a few real points scores perfect precision and a poor recall; one that copied every real point would score high on both, and neither number can tell copying from learning. Drawn stand-in, not a trained model: the points are two dimensional and seeded, where real papers use features with hundreds of dimensions.

#The 2025-2026 Metrics That Actually Matter

FID is a starting point, not a finish line. Labs that build text-to-image and video models have moved on to composition tests, preference-trained scores and Elo rankings, which track what users actually want better. Use several of them. No single score is enough on its own.

#GenEval (Ghosh et al. 2023): compositional T2I evaluation

#HPSv2: Human Preference Score v2

HPSv2 is a CLIP-style model fine-tuned on the HPDv2 dataset of 798K human preference pairs spanning DALL-E 2, SD 1.5, SDXL, Midjourney v5, and others. Given a prompt and a generated image, HPSv2 outputs a scalar that correlates strongly with which image humans pick in pairwise tests. It became the standard during the diffusion-DPO / direct preference optimization era — "improve HPSv2" is now a default training signal, not just an eval.

#PickScore (Kirstain et al. 2023)

A sibling preference-trained scorer trained on the Pick-a-Pic dataset of real user preference pairs (hundreds of thousands to about a million, depending on the version) collected from a public T2I playground. Closely related to HPSv2 and often reported alongside it; PickScore is the metric Stable Diffusion fine-tuning papers (DPO-SDXL, Diffusion-DPO) commonly use as their preference reward.

#ImageReward (Xu et al. 2023)

A BLIP-based reward model trained on roughly 137K expert comparisons across text-image alignment, fidelity, and harmlessness axes. Widely used as an RLHF reward signal for diffusion fine-tuning; included in the standard eval basket together with HPSv2 and PickScore so a model isn't gamed against just one preference distribution.

#VBench (Huang et al. 2023, ongoing)

The video equivalent of GenEval. VBench scores video generators across 16 axes including subject consistency, background consistency, temporal flickering, motion smoothness, dynamic degree, object class accuracy, multiple-object handling, color, spatial relationship, scene, and overall video-text alignment. It's how Sora, Veo, Kling, Pika, and HunyuanVideo are compared in published leaderboards.

#LMSYS Image Arena: Elo at scale

#FID: still useful, but Goodhart-prone

#CLIPScore: careful of reward hacking

CLIPScore was the original "does the image match the prompt?" metric. The problem: CLIP itself has biases, and it is easy to game by writing text labels into the image or by matching CLIP's word preferences instead of the prompt's meaning. Modern pipelines either retire it in favor of HPSv2 / PickScore / ImageReward, or report it alongside them so any one reward model can't be overfitted in isolation.

#Two older numbers: Inception Score and bits per dimension

Two older numbers still appear in papers. The Inception Score (IS) runs a classifier over the generated images and rewards images that are each clearly one class while the classes are all used. It is deprecated for rigorous comparison (it never looks at real images), but it is a clean application of mutual information from the Information Theory & Entropy lesson. Bits per dimension (bpd) reports the likelihood of a model that has one, such as a flow or an autoregressive model, in a unit that does not depend on image size.

#When to Use Each Metric

#Key Takeaways

• The Fréchet distance between two Gaussians is a gap in means plus a gap in spreads. FID is that distance on Inception features
• KID, DINOv2-FID and FVD change the feature network or the comparison. State which one you used, and how many samples
• Precision and recall come from nearest-neighbour balls. A bigger k makes every ball larger and raises both numbers, so papers state k
• Composition tests, preference scores and blind Elo catch what FID cannot. Every fixed score gets gamed once it becomes a target
• The classic Inception Score is deprecated for careful comparison; report FID plus precision and recall

#Quick Check