Why AI Makes Things Up (and What to Do)

#The Fluent Student Who Bluffs

Picture a student who has read a great many books. They speak smoothly. The teacher asks about something the student never studied. They do not say "I do not know." They give a smooth answer in the voice of a good answer. It has the right shape. It might even be right. Nobody can tell from the voice alone.

Here is a tiny example. You ask a tool for a book about sleep. A smooth reply names a title, an author and a year. All three look right. Whether that book exists is a separate question. Writing the reply never checked.

Here is the path a good answer should take.

The step in the middle, your check, is not inside the model. It is yours.

#Why Text Gets Made Up

Nothing in that loop looks anything up. The chances come from patterns in the text it learned from. A strong pattern gives a high chance. A weak one gives a low chance. Neither tells you the claim is true.

Try the slider below. It is a drawn stand-in, so the numbers are hand-set.

Do: Pick Geography. Keep temperature at 1.0 and press Sample several times. Then set temperature to 3.0 and sample again.

Notice: Paris wins most often, but Lyon and Rome, both wrong, appear now and then. At 3.0 they appear more often.

Why: The pick follows the chances, not the facts. Drawn stand-in, not a trained model: the scores are hand-set.

Look at the bars at temperature 1.0. Paris has about 53 chances out of 100. Lyon and Rome have about 4 each. Sample often enough and a wrong city comes up. Raise the temperature to 3.0 and Paris falls to about 20, while Lyon and Rome rise to about 9 and 8.

Two lessons come from this. First, a pick can be wrong even when the tool "knew" the right one, because the choice is made by chance. Second, a high chance is not a truth meter. It says how likely the text is. It does not say whether the claim is correct.

Once a token is written, later tokens build on it. A wrong early pick can pull the rest of the answer along with it. And every token is written in the same smooth voice, whether its chance was 53 or 4 out of 100.

Invented sources work the same way. A source has a clear shape: author names, a title, a journal, a year, page numbers. The model knows that shape very well. It can fill each slot with pieces that fit. The result looks like a source. No library was visited.

Answers tend to go wrong most in these places:

• exact numbers, dates and quotes
• names of people, books and papers
• topics that few texts discuss
• questions that assume something false

Now try it yourself. Decide before you open the answer.

#Why Pictures Get Extra Fingers

Picture drawing a hand from memory, fast. You remember "a palm, then a row of fingers." You may not remember how many. Your drawing has the right look and the wrong count.

The same cause explains other slips:

• Letters on a sign that look like words but spell nothing.
• A clock with the wrong numbers, or a shadow pointing the wrong way.
• A building that looks real but could not stand up.
• An animal with one leg too many.

This matters beyond odd hands. A picture can look like a real photo of an event that never happened. How to handle that comes in the safe habits below.

#Where Bias Comes From

Picture a picture tool that learned the word "breakfast" from 100 photos. Suppose 90 show toast and 10 show rice dishes. Ask for a breakfast many times. About 9 pictures in 10 will show toast. Nobody told the tool that toast is the right breakfast. It copied the lean in its examples.

1. The examples lean. If most examples look one way, most outputs do too.
2. The examples have gaps. What is missing from the examples is rare or missing in the output.
3. People made the examples. Text and photos carry the habits of the people who wrote and took them.

For breakfast, a lean is harmless. For pictures and words about people, it can be unfair. A tool might assume a certain kind of person for a certain kind of job, or write more warmly about some groups than others. The tool has no opinion. It is repeating the pattern it was shown.

Builders try to reduce bias. They choose data with care and test the tool. That helps, and it does not remove the lean. So you keep a few habits:

• Ask for several different outputs, not one.
• Say clearly what variety you want.
• Read with this question in mind: who is missing here?

#Memorising Versus Creating

Go back to the student. Imagine two of them. One memorised a paragraph word for word. The other learned the idea and writes it in new words. Both can answer. Only one is making something new.

Notice the table has two rows and both have a risk. Memorised output is not safer for facts. Created output is not safer for ownership. Do not assume the text or picture you got is new.

This also explains why private details are a worry. If a private detail was in the examples, it could come back. And what you type into a service may be kept by that service, depending on its settings. When you are unsure, leave it out.

#Safe Habits

Imagine a study helper replies to you. Decide what to do first.

These five habits cover most of what you need.

1. Check names, numbers and sources. These are the parts most often made up. A quick number check helps. A tool says a trip of 150 kilometres at 60 kilometres an hour takes 3 hours. Divide 150 by 60 and you get 2.5. The claim is wrong.
2. Ask for sources, then open them. A source you did not open is not a source. Also read the page. Does it say what the tool claims?
3. Use the tool for drafts, not final truth. It is good for outlines, ideas and first versions. You make the final call on facts.
4. Keep private data out. Do not type passwords, health details, addresses, or other people's secrets.
5. Label AI-made media. When you share a picture, audio or video that AI made, say so. Others can then judge it correctly. A record of where media came from is called provenance.

Here is a short self-check to run before you use any AI answer.

• Did I check every name, number and date that matters?
• Did I open each source, and does it say what the tool says?
• Is this a draft, or am I treating it as final truth?
• Did I leave out private data?
• Did I look at more than one output for any lean?
• If I share media, did I say it is AI-made?

#Looking Things Up First

Picture an open-book exam. A closed-book student answers from memory and may bluff. An open-book student looks at the notes first and answers from them.

Below is a small stand-in. It has six short notes about three topics. It matches the words in your question to the notes, picks the best two, and pastes them into the answer.

Do: Press Search on the first example, then step to the end. Next type "What is the tallest mountain on Earth?" and search.

Notice: Two notes come back even for the unrelated question. The scores stay low, and the answer still repeats the notes.

Why: Search always returns its best two. A low score is a warning, but nothing stops an answer built on weak notes. Drawn stand-in, not a trained model.

Retrieval helps in real ways. The answer can point to a note you can open. The model has facts in front of it instead of guessing. Still, errors remain:

• Search can fetch the wrong note, as the stand-in just showed.
• The right note can be found and still be misread.
• The note itself can be wrong or out of date.

Checking steps reduce errors further. You can ask the tool to quote the line it relied on. A second pass can compare the answer with the note. A person can confirm the facts that matter. Each step cuts errors. None makes the number zero. So the habits in the last section stay in place.

#Key Takeaways

• A generative model produces what is likely to look right. It does not check what is true
• A confident voice tells you nothing about whether an answer is correct
• Text goes wrong when a pick follows the chances and lands on a wrong piece. Pictures go wrong when each patch looks right but the whole is never checked
• Bias is the lean in the examples showing up in the output. Look at several outputs and ask who is missing
• A model may copy an example it saw often, so do not assume an output is new
• Check names, numbers and sources, open them, use the tool for drafts, keep private data out, and label AI-made media
• Retrieval and checking steps reduce errors and never remove them

#Quick Check