Scoring Retrieval

In the last lesson you built a helper that finds notes and answers from them. It worked on three questions. This lesson asks the harder question: how well does it work on all questions? You will write an answer key, compute three simple scores by hand, and then let a short program compute them for you. These scores are the most useful habit in RAG, because they tell you whether the find step is healthy before you look at any answer.

Learning Objectives

After this lesson, you will be able to:

Write an answer key: a list of questions with the notes that should come back for each
Compute hit rate, recall and reciprocal rank by hand on a small list of ranks
Read a scoreboard for three searches and say which one to trust, and why a small score table can still mislead
Decide whether a wrong answer came from the find step or from the model, by looking at what was retrieved

Words you need8 terms

A list of test questions, each paired with the notes that should be found for it. You write it once and reuse it every time you change the search.

A note's place in the list a search returns. Rank 1 is the top of the list.

Did the right note appear in the top k results? It is a yes or no for each question.

The share of questions where the right note made the top k. If 6 of 8 questions hit, the hit rate is 0.75.

Of all the notes that should have come back, the share that did come back in the top k.

Of the k notes the search returned, the share that were right.

1 divided by the rank of the first right note. Rank 1 gives 1, rank 2 gives 0.5, rank 10 gives 0.1.

Mean reciprocal rank: the average of the reciprocal rank over all your questions.

#Look at Retrieval Before You Look at the Answer

Suppose your helper gives a wrong answer. Two very different things could have gone wrong. The find step may have returned the wrong notes, so the model never saw the facts. Or the find step may have returned the right note, and the model misread it. These need different fixes, and you cannot tell them apart by reading the answer alone.

There is a simple habit that settles it: look at the notes that came back. If the right note is not in the prompt, the model had no chance, and you should fix the search. If the right note is in the prompt and the answer is still wrong, fix the prompt or the model.

One question tells you little. A single wrong answer could be bad luck. So we ask many questions, check each one, and turn the results into a few numbers. That needs an answer key.

In one sentence: check what the search returned before you judge the answer, and do it over many questions, not one.

#An Answer Key: Questions and the Notes That Should Come Back

An answer key is a table. The left column holds a question a real user might ask. The right column holds the note numbers that answer it. You decide these by reading the notes yourself, before you run any search. It is slow and a little dull, and it is the most valuable hour in a RAG project.

Here is the key for the ten library notes from the last lesson.

Question	Right note
What time do you open?	0
Can I extend my loan?	1
How much is a late fine?	2
How do I get a library card?	3
Can I reserve a study room?	4
What does printing cost?	5
Is there anything for kids on weekends?	6
Where do I find the wifi password?	7
What do a late book and a lost card each cost?	2 and 8

Notice how the questions are worded. "Extend my loan" is not the note's wording, which says "renewed". "Kids on weekends" is not the note's wording either. Real users do this all the time. An answer key made only of questions that copy the note's words would make every search look good.

The last question needs two notes. Keep that in mind: it matters for recall.

In one sentence: an answer key pairs real-sounding questions with the notes that should come back, and you write it before you run the search.

#Hit Rate: Did the Right Note Make the List?

The simplest score is yes or no for each question. Take the top k results. If the right note is in them, that question is a hit. Count the hits and divide by the number of questions.

Here is a small made-up example to practise on. Five questions were asked. The right note landed at these ranks: 1, 3, 1, 5, 2.

Hit at 1 counts the questions where the right note was rank 1. Ranks 1 and 1: that is 2 of 5, so 0.40.
Hit at 3 counts rank 3 or better. Ranks 1, 3, 1 and 2 qualify; rank 5 does not. That is 4 of 5, so 0.80.

What Do You Think?

Same five ranks: 1, 3, 1, 5, 2. A developer says 'our search is great, hit at 3 is 0.80'. What does that still not tell you?

Which k should you use? The number of notes you really paste into the prompt. If you paste three, hit at 3 is the honest question: did the model get to see the right note?

In one sentence: hit rate is the share of questions where the right note appears in the top k.

#Recall: What If Two Notes Are Needed?

Some questions need more than one note. "What do a late book and a lost card each cost?" needs note 2 and note 8. If a search returns note 2 but not note 8, the model can answer half the question.

Recall at k counts that. For each question, divide the number of right notes found in the top k by the number of right notes that exist. Then average over questions.

For the two-note question, suppose the top three are notes 2, 5 and 9.

Right notes that exist: 2 and 8, so 2.
Right notes found in the top 3: only note 2, so 1.
Recall at 3 for this question is 1 / 2 = 0.50.

When every question has just one right note, recall at k and hit at k are the same number. They only split apart on questions like this one.

There is a partner score called precision at k: of the k notes returned, how many were right? In the example, one of three returned notes was right, so precision is 1 / 3, about 0.33. Pasting more notes raises recall, because there are more chances to include the right one, but it usually lowers precision, because the extra notes are often wrong. That tension shows up again below.

In one sentence: recall at k is the share of the right notes that came back, and it is the score that matters when a question needs several notes.

#Reciprocal Rank: How High Did It Land?

Hit and recall ignore the order inside the top k. Rank 1 and rank 3 both count as a hit at 3. But order matters, since models tend to lean on what comes first, and since you might paste only one note.

Reciprocal rank rewards early placement. It is 1 divided by the rank of the first right note.

Rank 1 gives 1 / 1 = 1.00.
Rank 2 gives 1 / 2 = 0.50.
Rank 3 gives 1 / 3 = 0.33.
Rank 5 gives 1 / 5 = 0.20.

Back to the five practice ranks: 1, 3, 1, 5, 2. Their reciprocal ranks are 1, 0.33, 1, 0.20 and 0.50. They add up to 3.03. Divide by 5 questions and the MRR is about 0.61.

The drop from rank 1 to rank 2 is big (1.00 to 0.50), and the drop from rank 9 to rank 10 is tiny (0.11 to 0.10). That matches how people use results: the first few places matter most.

In one sentence: reciprocal rank is 1 divided by the rank of the first right note, and MRR averages it over your questions.

#Run the Scoreboard

Now let a program do the arithmetic. The cell below holds the ten notes, the answer key, and three searches from the last lesson: overlap (count shared words), weighted (drop filler words, rare words count more) and meaning (the hand-made four-number embedding). It scores each search with hit at 1, hit at 3, recall at 3 and MRR.

One detail protects you from luck. When several notes tie, the program counts the right note as the last of the tied notes, so a lucky list order cannot help a search. This is how you should treat ties in your own tests.

pythonplayground.py · Pyodide

# Scoring retrieval by hand: three searchers, nine questions, one answer key.
import math

docs = [
  "The library opens at 9 am on weekdays and closes at 6 pm.",
  "Books can be borrowed for 21 days and renewed once online.",
  "A late book costs 10 cents per day, up to a limit of 5 dollars.",
  "Library cards are free for residents and need a photo ID.",
  "The study room can be booked for two hours at a time.",
  "Printing costs 5 cents a page and scanning is free.",
  "Children's story time is on Saturday mornings at 10.",
  "Wifi is free. Ask the desk for the password.",
  "Lost cards can be replaced at the desk for a 2 dollar fee.",
  "The library is closed on public holidays.",
]

# The answer key: for each question, which notes really answer it.
exam = [
  ("What time do you open?", {0}),
  ("Can I extend my loan?", {1}),
  ("How much is a late fine?", {2}),
  ("How do I get a library card?", {3}),
  ("Can I reserve a study room?", {4}),
  ("What does printing cost?", {5}),
  ("Is there anything for kids on weekends?", {6}),
  ("Where do I find the wifi password?", {7}),
  ("What do a late book and a lost card each cost?", {2, 8}),   # needs TWO notes
]

STOP = {"the", "a", "an", "is", "are", "at", "on", "of", "for", "to", "and", "be", "can", "do", "does", "i",
      "my", "what", "when", "how", "much", "long", "in", "it", "up", "per", "there", "where", "you", "get"}

def words(text):
  return text.lower().replace(".", "").replace(",", "").replace("?", "").split()

def stem(w):
  for tail in ("ed", "es", "s"):
      if w.endswith(tail) and len(w) > len(tail) + 2:
          return w[: -len(tail)]
  return w

def keep(text):
  return [stem(w) for w in words(text) if w not in STOP]

def overlap_score(q, d):
  return len(set(words(q)) & set(words(d)))

df = {}
for d in docs:
  for t in set(keep(d)):
      df[t] = df.get(t, 0) + 1

def rarity(t):
  return math.log((len(docs) + 1) / (df.get(t, 0) + 1)) + 1

def weighted_score(q, d):
  dk = set(keep(d))
  return sum(rarity(t) for t in set(keep(q)) if t in dk)

MEANING = {}
for w in ["open", "clos", "hour", "day", "weekday", "holiday", "saturday", "morning", "time"]:
  MEANING[w] = (1, 0, 0, 0)
for w in ["fee", "cost", "cent", "dollar", "pay", "price", "charge", "free"]:
  MEANING[w] = (0, 1, 0, 0)
for w in ["book", "borrow", "keep", "hold", "title", "renew", "lend", "card"]:
  MEANING[w] = (0, 0, 1, 0)
for w in ["wifi", "internet", "password", "online", "printing", "scanning", "connect"]:
  MEANING[w] = (0, 0, 0, 1)
MEANING["late"] = (1, 1, 0, 0)

def embed(text):
  total = [0.0] * 4
  for w in keep(text):
      for i, x in enumerate(MEANING.get(w, (0, 0, 0, 0))):
          total[i] += x
  return total

def cosine(a, b):
  dot = sum(x * y for x, y in zip(a, b))
  na, nb = math.sqrt(sum(x * x for x in a)), math.sqrt(sum(x * x for x in b))
  return 0.0 if na == 0 or nb == 0 else dot / (na * nb)

def meaning_score(q, d):
  return cosine(embed(q), embed(d))

def gold_rank(q, gold, score_fn):
  """Rank of the best answer note. Notes that tie with it count as ahead of it, so luck cannot help."""
  scores = [score_fn(q, d) for d in docs]
  best = max(scores[i] for i in gold)
  return sum(1 for s in scores if s >= best)

def top_k(q, score_fn, k):
  scored = sorted(((score_fn(q, d), i) for i, d in enumerate(docs)), key=lambda s: (-s[0], s[1]))
  return {i for s, i in scored[:k] if s > 0}

def evaluate(score_fn, k=3):
  hit1 = hitk = recall = mrr = 0.0
  for q, gold in exam:
      first = gold_rank(q, gold, score_fn)
      hit1 += first == 1
      hitk += first <= k
      recall += len(gold & top_k(q, score_fn, k)) / len(gold)
      mrr += 1 / first
  n = len(exam)
  return hit1 / n, hitk / n, recall / n, mrr / n

print(f"{'searcher':10} {'hit@1':>6} {'hit@3':>6} {'recall@3':>9} {'MRR':>6}")
for name, fn in [("overlap", overlap_score), ("weighted", weighted_score), ("meaning", meaning_score)]:
  h1, h3, r3, m = evaluate(fn)
  print(f"{name:10} {h1:6.2f} {h3:6.2f} {r3:9.2f} {m:6.2f}")

print()
for q, gold in exam:
  ranks = [gold_rank(q, gold, fn) for fn in (overlap_score, weighted_score, meaning_score)]
  print(f"{q:46} rank of the answer note: overlap {ranks[0]}, weighted {ranks[1]}, meaning {ranks[2]}")

# CHALLENGE: add these six words to MEANING above and run again. Compare the meaning row.
#   extend, loan -> (0, 0, 1, 0)    fine -> (0, 1, 0, 0)    reserve, weekend, kid -> (1, 0, 0, 0)
# Is a better score proof of a better search, or of a search tuned to this answer key?

Tests · Run it as it is. The scoreboard should read: overlap 0.56, 0.89, 0.83, 0.70; weighted 0.78, 0.78, 0.78, 0.80; meaning 0.33, 0.67, 0.67, 0.51. After adding the six words from the solution box, meaning becomes 0.44, 0.89, 0.89, 0.66, while overlap and weighted do not change.

Do: Run the cell and read the table. Find the question where overlap has rank 10 and weighted has rank 1.

Notice: Weighted wins on hit at 1 (0.78) and MRR (0.80). Overlap wins on hit at 3 (0.89). Meaning is last. The question "What time do you open?" sends overlap to rank 10, since it shares no word with note 0.

Why: Each search has a different weak spot. Overlap cannot see "open" and "opens" as the same word, weighted cannot see "extend my loan" means "renew", and the hand-made meaning table simply lacks most of the words in the key. Drawn stand-in: three toy searches over ten notes, not a real benchmark.

The weighted row is easy to check by hand. It put the right note at rank 1 for seven questions, and at rank 10 (no better than a tie with everything) for two. Hit at 1 is 7 / 9, which is 0.78. MRR is (7 x 1 + 2 x 0.1) / 9 = 7.2 / 9 = 0.80. The program and your hand agree.

The next picture shows the same three scores at larger scale: twenty chunks, six of them right, ranked by three pretend retrievers. Drag the slider for k and watch the bars.

Try it: Precision, recall and MRR at kInteractive

Loading visualization...

Do: Pick Tuned retriever and set K to 5. Then pick Noisy retriever, keep K at 5, and compare. Drag K up to 20 and watch recall climb to 1 while precision settles at 0.30.

Notice: At K = 5 the tuned retriever has three right chunks among its five, and its first right chunk is at rank 1, so MRR is 1. The noisy one has one right chunk among five, at rank 3, so MRR is one third. MRR does not move when you drag K.

Why: Precision and recall depend on how many results you take, but MRR only looks at where the first right one sits. Drawn stand-in: the three rankings are fixed examples, not the output of a real search.

In one sentence: a small program computes the same hit rate, recall and MRR you did by hand, so you can re-run the whole key after every change.

#Reading the Numbers Without Fooling Yourself

The scoreboard is useful and also easy to misuse. Four cautions.

Nine questions is very few. One question moves hit at 1 by 0.11. A real answer key has dozens or hundreds of questions, taken from real users where you can. Treat small differences in a small table as noise.

Do not tune to the key you score on. Add the six words from the solution box to the meaning table and the meaning row jumps from 0.67 to 0.89 on hit at 3. That feels like progress. But you chose those words by reading the failed questions, so the table now knows this exam. A fair test uses questions the search was never tuned on. Keep a second set of questions locked away, and only score on it at the end.

Do not trust a name. Meaning search is usually better than word counting in real systems. In this toy table it came last. The scoreboard did not care about reputation, only about the key. Measure your own data.

The key can be wrong. If you marked the wrong right note, the search is punished for being correct. Re-read the questions that every search fails, since they are often key mistakes.

In one sentence: score on enough questions, keep a locked-away set, and distrust any improvement you made by reading the exam.

#Which Half Broke?

Now put the score to work. When the whole helper gives a wrong answer, check two things, in this order.

What you see	Where the problem is	What to try
The right note is not in the top k	Find step	Better wording match, a meaning search, more notes in k, cleaner notes
The right note is in the top k but ranked low	Find step (ordering)	Better scoring, or a second pass that re-orders the short list
The right note is in the prompt, the answer is still wrong	Answer step	Clearer prompt rule, a stronger model, fewer distracting notes
The answer is right but cites the wrong note	Answer step	Ask for the quote, and check the quote against the note

This table is why we scored retrieval first. A search with hit at 3 of 0.67 means a third of the questions never show the model the right note, and no change to the model will fix those. A later lesson, RAG Evaluation, adds scores for the answer step. For now, remember the order: retrieval first, answer second.

In one sentence: if the right note is not in the prompt, fix the search; if it is, fix the prompt or the model.

#Key Takeaways

Check what the search returned before you judge the answer, and do it over many questions.
An answer key pairs real-sounding questions with the notes that should come back. Write it before you run the search.
Hit at k asks whether the right note is in the top k. Recall at k asks how many of the right notes came back. Precision at k asks how many returned notes were right.
Reciprocal rank is 1 divided by the rank of the first right note. MRR is its average.
A score table over nine questions is noisy, and improving a search by reading the exam tunes it to the exam.
If the right note is not in the prompt, fix the find step. If it is, look at the prompt and the model.

#Quick Check

Quick Check1 / 4

Five questions put the right note at ranks 2, 1, 4, 1 and 6. What is hit at 3?

Where to next. Go to Why RAG? for the bigger picture of when RAG beats a longer prompt or a retrained model. When you are ready for the full set of answer-step scores, read RAG Evaluation. To see how the ranking itself is improved, read Retrieval Algorithms.

Scoring Retrieval

Learning Objectives

After this lesson, you will be able to:

Write an answer key: a list of questions with the notes that should come back for each
Compute hit rate, recall and reciprocal rank by hand on a small list of ranks
Read a scoreboard for three searches and say which one to trust, and why a small score table can still mislead
Decide whether a wrong answer came from the find step or from the model, by looking at what was retrieved