RAG by Hand

#A Model That Has Not Read Your Notes

Imagine a small town library. Its front desk has a folder of ten short notes: opening hours, late fees, how to get a card. You ask a chatbot, "When does the Maple Street Library open?" The chatbot has never seen that folder. The folder was written for this one library, and no model was trained on it.

What can the chatbot do? It can guess a typical answer, such as "most libraries open at 9". That sounds sure and may be wrong. Or it can say it does not know. Neither helps you. The answer sits in note 0 of the folder, and the model is not allowed to look.

Now picture a different way. A helper reads your question, flips through the folder, pulls out the note about opening hours, and puts it on the desk next to the model with the words, "Answer from this note." The model now has the facts in front of it. It does not need to remember them, and it does not need to guess.

That is the whole idea of this lesson. The helper is not magic. In this lesson you will build one in about forty lines of plain Python, and you will watch it succeed and fail on real questions.

All the notes in this lesson are made up for teaching. The Maple Street Library is invented.

In one sentence: a model can only use what it learned or what is in its prompt, so we look up the right note and put it in the prompt.

#Three Steps: Find, Paste, Answer

The pattern has three steps, and each one is small.

1. Find. Score every note against the question and keep the best few. This step is called retrieval.
2. Paste. Put those notes into the prompt, above the question, with numbers so the answer can point back to them.
3. Answer. Let the model write a reply that uses only those notes.

Most of the work, and most of the ways to fail, are in step 1. If the wrong note is found, the model gets the wrong facts and will answer from them politely. So we spend most of this lesson on finding.

In one sentence: RAG is find the best notes, paste them into the prompt, and answer from them.

#Finding by Counting Shared Words

Here is the simplest possible way to score a note: count how many words it shares with the question. No maths, no model. Write down the words of the question, write down the words of the note, and count the ones that appear in both.

Try it by hand. The question is "What is the fee for a late book?" Its words are: what, is, the, fee, for, a, late, book. Look at two notes.

• Note 8: Lost cards can be replaced at the desk for a 2 dollar fee. It shares the, for, a, fee. That is 4.
• Note 2: A late book costs 10 cents per day, up to a limit of 5 dollars. It shares a, late, book. That is 3.

The code below runs the counting rule on the ten notes. Run it as it is first. Later cells in this lesson reuse the same notes.

Do: Run the cell and read the two lines printed under each question.

Notice: For the late-fee question the first method puts note 8 on top with 4 and note 2 second with 3, exactly as in the hand count. For the opening question, notes 0 and 9 both score 2.

Why: The first method counts every shared word the same, even the, for and a. When two notes tie, the code lists the lower note number first, which is luck and not skill.

#Rarer Words Count More

• late is in 1 note, so its weight is about 2.70.
• fee is in 1 note, so its weight is about 2.70.
• book is in 3 notes, so its weight is about 2.01.
• library is in 3 notes, so its weight is about 2.01.

In one sentence: dropping filler words and giving rare words more weight turns word counting from a coin flip into a usable search.

#When the Words Do Not Match

The weighted method gave every note a score of 0. It found nothing. Counting words cannot see that hold a title and borrow a book mean the same thing.

The hand-made table is a stand-in. A real embedding model learns its numbers from a huge amount of text, so it needs no table. Even so, the step after the table is identical: turn both texts into lists, score by cosine, keep the best.

The map below is a drawn version of the same idea. Ten sentences sit on a plane, grouped by topic, and a search for a new question is just "find the nearest dots".

Do: Click any sentence dot and read its three nearest neighbours. Then drag that dot toward a different group and watch the list change.

Notice: Sentences about the same topic sit close together, and the nearest-neighbour list changes as the dot moves.

Why: Retrieval by meaning is ranking by distance. Drawn stand-in: the positions are hand-placed and a typed query is placed by keyword matching, not by a trained model.

In one sentence: an embedding turns text into a short list of numbers so that similar meanings can be found even when no words match.

#Paste the Notes Into a Prompt

Finding the notes is half the job. Now we use them. The second cell below builds the prompt that we would send to a model. It does three things: it numbers the retrieved notes, it tells the model to answer only from them, and it keeps the question last.

A good prompt of this kind has four parts:

• A rule. "Answer using only the numbered notes. Cite the note number. If the notes do not say, reply: I could not find that."
• The notes, each with a number such as [1].
• The question.
• A way out. The rule says what to do when the notes are silent, so the model is not forced to guess.

We cannot call a real model from this page. So the cell ends with a stand-in answerer. It is not a language model. It copies the best note and adds the marker [1]. That is enough to show the shape of a grounded answer: the reply says where it came from, and a reader can check it.

Do: Run the cell. Then add the question "How much does a late book cost?" to the list at the bottom and run it again.

Notice: The hold-a-title question now finds note 1, which the word methods could not. The internet question also pulls in a weak second note, note 5, at 0.55. The coffee question finds nothing.

Why: Meaning scores can bridge different wording. The cut-off of 0.5 keeps weak matches out, but it also lets a weak match through when a question brushes a theme. Drawn stand-in: the meaning table was written by hand for these notes.

Look at the prompt for the coffee question. Because no note was close enough, the prompt says so, and the answer is "I could not find that in the notes." This is the most important habit in RAG. A system that always answers will sometimes answer from nothing, and that is where invented facts come from.

In one sentence: the prompt carries numbered notes plus a rule to answer only from them, and a way out for when they do not help.

#What Goes Wrong

You have now seen a full RAG helper. Here are the ways it breaks, so you know what to watch for.

• The right note exists but is not found. Different wording, filler words or ties hide it. This is a retrieval failure, and no model can fix it afterwards because the model never sees the note.
• A wrong note is found and trusted. The model answers politely from the wrong facts. The citation helps here, because a reader can click [1] and see the note does not fit.
• The right note is found and the model ignores it. This is a generation failure. The rule in the prompt and a citation make it easier to catch.
• The notes are old. A note that is out of date gives an out-of-date answer. RAG only helps if the collection is kept current.
• A note holds an instruction. Text inside a note is read by the model like the rest of the prompt. A note that says "ignore your rules" is an attack. A later lesson, RAG Security and Production, covers that.

#The Same Idea With Real Tools

Real systems follow the same three steps, with better parts: a trained embedding model in place of the hand-made table, a database that stores the lists, and a real language model for step 3. The cell below is a read-only copy for Google Colab. It needs a package that downloads a model, so it cannot run in this page.

# Copy to Colab (needs sentence-transformers and a model download; it cannot run in the browser)
# pip install sentence-transformers
from sentence_transformers import SentenceTransformer, util

notes = [
    "The library opens at 9 am on weekdays and closes at 6 pm.",
    "Books can be borrowed for 21 days and renewed once online.",
    "A late book costs 10 cents per day, up to a limit of 5 dollars.",
]
model = SentenceTransformer("all-MiniLM-L6-v2")      # a small, widely used embedding model
note_vectors = model.encode(notes)                    # one list of numbers per note
question = "How long can I hold a title?"
question_vector = model.encode(question)
scores = util.cos_sim(question_vector, note_vectors)[0]   # cosine similarity, as in this lesson
best = int(scores.argmax())
print(f"best note: {best} ({float(scores[best]):.2f})", notes[best])
# Next: paste notes[best] into a prompt like the one above and send it to the model you use.

Notice that the structure matches what you built: embed the notes, embed the question, score by cosine, keep the best, paste. Only the quality of the embedding has changed. The model name above is an example; check the package documentation for the current list.

In one sentence: real RAG is the same find, paste, answer loop with a trained embedding model and a vector database in place of the hand-made parts.

#Key Takeaways

• A model only knows what it learned and what is in its prompt, so RAG looks up the right notes and puts them in the prompt.
• RAG is three steps: find the best notes, paste them in, answer from them.
• Counting shared words is a weak search: filler words mislead and ties are luck.
• Dropping filler and weighting rare words fixes much of it, but it still misses a question worded differently from the note.
• An embedding is a list of numbers for meaning. Cosine similarity scores how well two lists point the same way.
• The prompt should number the notes, say to answer only from them, and give a way out when they are silent.
• Most failures are in finding. Look at what was retrieved before you blame the model.

#Quick Check