How a Chatbot Writes

A chatbot is a program you chat with in plain words. It does not plan a whole answer and then type it out. It writes one small piece of text, then another, then another. Each piece is a guess about what should come next. This lesson shows how one guess is made, how a random pick turns it into a word, and why that explains both the variety and the mistakes.

Recap of the last lesson. A generator can make something new by picking at random from many possible results, not by copying one. That idea comes back here, because a chatbot picks its next piece of text the same way. See Sampling and Latents if the idea is new.

Learning Objectives

After this lesson, you will be able to:

Say what a token is and why a chatbot reads and writes in tokens, not letters or whole sentences
Explain the one job a chatbot does at every step, and the loop that repeats it until the reply is done
Choose a temperature, top-k or top-p setting for a safe answer or for a more surprising one
Explain why the same question can get different answers, and why a confident answer can still be wrong

Words you need8 terms

A small piece of text that a chatbot reads and writes, such as a short word, part of a word, or a mark like a full stop.

All the text the chatbot has so far: your question plus the pieces it has already written.

A number the model gives to each possible next token. A higher score means the token fits better after the text so far.

Picking one token at random, where tokens with higher scores are more likely to be picked.

Always taking the single highest-scoring token, with no dice at all.

A dial for how adventurous the dice are. Low is safer, high is wilder.

A filter that keeps only the k most likely tokens before the dice are rolled.

A filter that keeps the fewest top tokens whose chances add up to p, then drops the rest.

#Tokens: Cutting Text Into Pieces

Picture a sentence written on a strip of paper. Now cut the strip into small tiles with scissors. Some tiles hold a whole short word. Some hold half of a long word. Some hold only a comma. Those tiles are tokens: the small pieces of text that a chatbot reads and writes.

Why not use letters? Letters are too small, so a reply would take far too many steps. Why not whole words? There are too many words, and new ones appear all the time. Pieces in between work well. Common words stay whole. Rare words are built from several tiles.

Here are three small examples. The splits are invented to show the idea. A real chatbot uses its own list of pieces, so its splits will differ.

Invented examples of cutting text into tokens

Text	Pieces (invented split)	Number of pieces
The cat sat.	The \| cat \| sat \| .	4
sunflower	sun \| flower	2
unbelievable	un \| believ \| able	3

Each piece in the chatbot's list also has a number. The chatbot never sees letters. It sees a row of numbers, and it writes a row of numbers back. The numbers are turned into text at the very end.

In one sentence: a token is a small tile of text, and a chatbot reads and writes tile by tile.

#The One Job: Guess the Next Piece

Picture a friend who has read a huge pile of text. You say "Peanut butter and" and stop. Your friend says "jelly" at once. Nobody taught the friend a rule about sandwiches. They have seen those words together many times.

A chatbot has one job like that. Given the context so far, it guesses the next token. It does this for every step. The context is all the text it has: your question plus anything it has already written.

The guess is not one word. The model looks at every token in its list and gives each one a score. A higher score means the token fits better after this text. After "Peanut butter and", the score for "jelly" is high. The score for "volcano" is very low.

The model does not look anything up while it scores. The scores come from patterns it picked up from the text it learned from. The details of that learning belong to later lessons.

In one sentence: at every step the model scores every possible next token, and its whole job is to make those scores good.

#Scores, Then a Dice Roll

Picture a spinner like the one in a board game. It has coloured slices. A big slice is easy to land on and a thin slice is hard. Now picture each possible next token as one slice. The slice size comes from its score.

Scores are turned into chances that add up to 100 out of 100. Here is an invented example after the text "The cat sat on the":

mat gets 70 out of 100
sofa gets 20 out of 100
roof gets 8 out of 100
moon gets 2 out of 100

Now the model spins. This is sampling: picking one token at random, with the likely ones picked more often. Try to guess what happens before you read on.

What Do You Think?

The chances are mat 70, sofa 20, roof 8 and moon 2, out of 100. You spin 10 times. How often do you expect to see mat?

This is the key point. The model does not always pick the top token. It picks by chance. Most of the time you see "mat". Now and then you see "sofa" or "roof". Rarely you see "moon", and then the sentence goes somewhere odd.

There is one other way to pick. A greedy pick always takes the highest score and never rolls. Greedy gives the same text every time. It can also get stuck repeating itself, as you will see below.

In one sentence: scores become chances, and a random pick chooses one token, so the likely tokens win often but not always.

#Dials on the Dice: Temperature, Top-k and Top-p

Picture a radio with a dial under the dice. You turn it before each spin. The dial does not change what the model knows. It changes how much the chances are pushed apart or pulled together before the spin. That dial is called temperature.

Test your guess first. Think about what the lowest setting should do.

What Do You Think?

You turn the temperature dial to its lowest setting, then pick a token 20 times after the same text. What do you expect to see?

Low temperature makes the top token even more likely. The dice become safe and predictable. High temperature flattens the chances, so lower tokens get a bigger slice. The dice become wilder. Use low for facts and steady wording. Use higher for stories and new ideas.

Two more filters make the dice safer. Both work by removing unlikely tokens before the spin:

Top-k keeps only the k most likely tokens. If k is 5, the five best stay and all the others are dropped.
Top-p keeps the fewest top tokens whose chances add up to p. If p is 90 out of 100, it keeps the best few until they cover 90 out of 100, then drops the rest.

In plain words, both say: only consider the likely words. After the filter, the leftover chances are shared out again and the dice roll.

Now try it. The scores in this stand-in are hand-set for ten candidate tokens after a short prompt. Three buttons choose the prompt: Geography, Code and Creative. The Temperature slider runs from 0.1 to 3.0. Two check boxes switch on the top-k and top-p filters, each with its own slider. The Sample button picks one token. The boxes named Entropy and Perplexity measure how spread out the chances are, and you can ignore them.

Loading visualization...

Do: Pick Geography. Slide Temperature to 0.1, press Sample five times, then repeat at 3.0. Then switch on top-k and press Sample.

Notice: At 0.1 the same token wins nearly every time. At 3.0 the bars level out and many tokens appear. Top-k greys out the rest.

Why: Temperature reshapes the chances and the filters remove tokens. Drawn stand-in, not a trained model: the ten scores per prompt are made up.

In one sentence: temperature sets how adventurous the dice are, and top-k and top-p cut away the unlikely tokens first.

Now put the dials to work on a whole sentence. This playground has a small hand-written table of next-word chances for three starting phrases. Turn the dials, press Generate next word, and watch the sentence grow.

Loading visualization...

Do: Pick Sky. Set Temperature to 0.2 and press Generate next word until the sentence ends. Press Reset sentence, set Temperature to 1.5, and repeat. Then set top-k to 2.

Notice: At 0.2 the top word "blue" goes from 45% to 98%, so the sentence is nearly the same every time. At 1.5 its share falls to 35% and rare words such as "falling" can appear. Top-k of 2 greys out the other words and shares their chance between the two left.

Why: Dividing the scores by a small temperature stretches the gaps between them, and a large one shrinks the gaps. The entropy number says how spread out the chances are. Toy distribution, drawn stand-in, not a trained model: the scores are hand-written.

#Repeat Until Done

So far the chatbot has picked one token. A reply needs many. The trick is a loop. The token that was picked is added to the end of the text. Then the whole job starts again with the longer text. This continues until the model picks a special "stop" token, or until it runs out of room.

Because each new token joins the text, it becomes part of the context for the next step. The model is reading its own words as it writes. This is also why an early odd pick can steer the whole reply. Everything after it is built on top.

Now watch the loop in a toy. This viz has a tiny vocabulary of twenty words and a hand-written table of chances. The three Starting prompt buttons are The cat, A big and The dog. The Temperature slider runs from 0.1 to 2.0. The Greedy Decoding switch always takes the top word. Buttons add one word, add five words, or reset to the start. A bar chart shows the chance of each of the twenty words as the next one. Ignore the small formula above the chart.

Be aware of one difference from a real chatbot. This toy looks only at the last word. A real chatbot reads all of the text so far.

Loading visualization...

Do: Pick The cat and press Generate Next Token eight times. Press Reset, switch on Greedy Decoding, and generate eight more.

Notice: With sampling, the words wander and differ from run to run. With Greedy, "sat on the cat" repeats in a loop.

Why: Each pick adds a word that changes the next chances. Drawn stand-in, not a trained model: the table is hand-written and uses only the last word.

In one sentence: the chatbot adds each picked token to the text and repeats the same guess until it decides to stop.

#Why the Same Question Gives Different Answers

Picture asking a friend to describe a sunset. They would use different words each time. The ideas are the same, but the choices of wording differ. A chatbot does the same, and the reason is the dice.

At each step the pick is random. Say two answers differ at step 3 only because one spin landed on "sofa" and another on "mat". From step 4 on, the texts are different, so the chances for the rest are different too. One small difference early on grows into two different replies.

You can see how this depends on the dials:

At a very low temperature, or with a greedy pick, the same question usually gives the same answer.
At a higher temperature, the same question gives more varied answers.
A different wording of your question gives a different context, so the scores change too.

None of this means the chatbot is being careless or changing its mind. Varied output is what sampling is for. It lets the chatbot write stories, names and ideas that were not written before.

In one sentence: the dice make each reply different, and a small early difference grows as the loop repeats.

#Why It Can Sound Sure and Still Be Wrong

Picture a friend who is good at finishing sentences in a confident voice. They finish "The first person to walk on the" with "moon" at once. Now ask them to finish a sentence about a town they have never heard of. They would still finish it smoothly. Smooth does not mean true.

A score tells how well a token fits after the text so far. It does not tell whether the sentence will be true. A made-up name and a real name can both fit the pattern of a sentence. When the model has little to go on, it still must pick something, and the pick is still fluent. The loop has no step that checks the claim against the world.

That is why a wrong answer can sound as sure as a right one. The confident tone comes from the style of the text, not from checking. A chatbot that is connected to a search tool or a document can be given facts to work from. The writing loop itself stays the same.

In one sentence: a high score means a token fits the pattern, not that it is true, so a confident reply still needs checking.

#Key Takeaways

Text is cut into tokens, small pieces that a chatbot reads and writes as numbers
The one job at each step is to give every possible next token a score
Scores become chances, and sampling picks one token at random, so likely tokens win often but not always
Temperature sets how adventurous the dice are, and top-k and top-p cut away unlikely tokens first
The picked token joins the text and the loop repeats until a stop token
The dice explain why answers differ. The scoring explains why a reply can sound sure and still be wrong

#Quick Check

Quick Check1 / 4

What is a token?

Where to next. Go to Noise to Picture to see a very different way to make things: pictures that start as noise and are cleaned up step by step. Then read Why AI Makes Things Up for the safe habits that follow from this lesson. When you want the full detail of tokens and picking rules, continue in Tokenization and Decoding Strategies.

How a Chatbot Writes

Learning Objectives

After this lesson, you will be able to:

Say what a token is and why a chatbot reads and writes in tokens, not letters or whole sentences
Explain the one job a chatbot does at every step, and the loop that repeats it until the reply is done
Choose a temperature, top-k or top-p setting for a safe answer or for a more surprising one
Explain why the same question can get different answers, and why a confident answer can still be wrong