Reinforcement Learning: Start Here

#Learning by Trying

Think of a small child learning to ride a bicycle. Nobody hands the child a sheet that says "lean 4 degrees left at this speed". The child pushes the pedals, wobbles, nearly falls, leans the other way, and stays up a little longer. After many tries the child rides. The child was never told the right move at each moment. The child only felt what happened after each try, and slowly kept the moves that worked.

Compare this with the kind of learning you may know already. A spam filter is shown 10,000 emails, and each one already has the right label written on it. That is learning from an answer sheet. In reinforcement learning there is no answer sheet. There is only a score that arrives after the program has done something, and the program has to work out for itself which of its choices earned it.

In one sentence: reinforcement learning is a program learning what to do by trying things and seeing the score each try earns.

#The Loop: Act, See, Adjust

Every reinforcement learning setup has the same two characters, and it helps to give them names.

The two take turns, over and over, in a loop with three beats.

1. See. The agent looks at the situation. The word for the situation is state, which just means "how things stand right now".
2. Act. The agent picks an action, which is one of the moves it is allowed to make.
3. Get a reward. The environment answers with a reward, a number that says how good that move turned out, and a new situation. Then the loop starts again.

Here is the whole thing for the bicycle child. State: leaning a little to the left at slow speed. Action: lean the body to the right. Reward: stayed up for 2 more seconds, which is good. New state: nearly upright. Then see, act and get a reward again.

Read the picture as a circle. The agent sends an action into the world. The world sends back a new situation and a score. The agent uses both to choose its next action. No part of this loop holds a list of right answers.

Do: Press Run. Then change random.Random(3) to random.Random(4) and run again. Try a few other numbers.

Notice: With the number 3 the robot bumps the left wall twice, wanders between cells 0 and 2, and finishes with total reward 0. Other numbers give different walks. Most short walks end with reward 0, because the prize is four cells away.

Why: The reward is 1 only when the robot lands on cell 4, so a random walker rarely gets any score at all. A learner needs a way to turn rare scores into better choices, and that is what the rest of this track teaches.

In one sentence: an agent sees a state, picks an action, and the environment answers with a reward and a new state, again and again.

#A Reward Is a Score, Not a Teacher

That makes the reward both powerful and easy to get wrong. Picture a maze game where you give 1 point for each step the agent takes toward the exit. The agent might learn to step back and forth next to the exit forever, collecting points without leaving. The program did exactly what the score asked for, and not what you wanted. A later lesson in this track, on reward design, looks at this trap in detail.

Two more facts about rewards help from the start.

Second, the agent cares about the total over time, not just the next score. Picture a student who skips a game tonight (a small loss now) to study for an exam (a bigger gain next week). A good agent can accept a small cost now for a bigger reward later. How to count a far-off reward against a near one is a topic for a later lesson, so just remember that the aim is a good total.

In one sentence: a reward is a score for what happened, often arriving late, and the agent chases it exactly as written.

#A First Game: Three Slot Machines

The simplest reinforcement learning game has no changing situation at all. Picture three slot machines in a row, which we will call A, B and C. Each time you pull a machine's lever, it either pays 1 point (a win) or pays nothing (a loss). Each machine has its own hidden chance of paying, and you are not told the chances. Your job is to collect as many wins as you can from a fixed number of pulls.

Let us play six pulls by hand. The results below are invented, written down in the order they happened.

Now count, machine by machine. Machine A was pulled 2 times and won 1 time, so its average is 1 out of 2, which is 0.5. Machine B was pulled 2 times and won 1 time, so its average is also 0.5. Machine C was pulled 2 times and won 2 times, so its average is 2 out of 2, which is 1.0.

Which machine looks best? Machine C, with the highest average so far. Notice the words "so far". Two pulls is a very short history. Machine C may have just had a lucky start, and A or B may be better in the long run. An average from 2 pulls is a weak guess, and an average from 200 pulls is a much stronger one.

In one sentence: a bandit game is the reward loop with the changing world removed, and the best machine so far is the one with the highest average, which is only a guess until there are many pulls.

#Try New or Repeat What Works

Here is the real puzzle. After the six pulls above, you believe machine C is best. Do you keep pulling C, or do you test A and B some more in case you misjudged them?

You meet this every day. Your favourite tea shop makes a good cup, so you could go there every day, which is exploitation. A new shop opens, and trying it is exploration: it might be worse, it might be your new favourite. A sensible person mostly goes to the favourite and sometimes tries somewhere new.

Try the live version below. It uses the same idea with more machines and shows how the choice of strategy changes how many wins you collect. Start with the epsilon-greedy setting.

Do: Reset the demo, pick Epsilon-Greedy, set Epsilon to a small value such as 0.05, and step forward 30 times. Then reset, set Epsilon to 1.0, and step 30 times again.

Notice: With a small Epsilon, one or two machines soak up most pulls and the total climbs steadily. With Epsilon at 1.0 the pulls spread almost evenly across all machines and the total grows more slowly.

Why: Epsilon is the chance of a random pull. At 1.0 every pull is random, which means no learning is used at all. A small Epsilon spends most pulls on the best guess, at the risk of settling on the wrong machine early.

In one sentence: exploring means trying to learn, exploiting means using what you know, and a good learner does mostly the second with a little of the first.

#Run It: Three Strategies Compared

Now compare three ways to play, in code. The cell below has three hidden machines that win 30 times in 100, 50 times in 100 and 70 times in 100. The player is not told this. Three strategies each play 60 pulls with the same seed.

The first strategy, "always machine 0", never looks at results. The second, "pick at random", spreads pulls evenly. The third tries each machine 3 times, and after that mostly pulls the machine with the best average so far, trying a random machine 1 time in 10.

Before you run it, make a prediction. Which strategy do you think wins most, and which wins least?

Do: Press Run. Then change seed=7 to seed=8 and run again. Try three or four seeds.

Notice: With seed 7, "always machine 0" wins 22 of 60, "pick at random" wins 32 of 60, and "try each, then mostly the best" wins 41 of 60, with 51 of its pulls on machine 2. Other seeds move the numbers a lot, and the learner does not always win: with seed 5 the random picker scores 33 and the learner 32.

Why: Always pulling machine 0 locks in the weakest machine. Random spreads the pulls evenly, so it earns about the average of the three chances. The third strategy learns from its results and moves most pulls to the machine that pays most.

This is reinforcement learning in its smallest form. The program was never told that machine 2 is best. It found out by trying, and it changed what it did because of the scores it saw.

In one sentence: a strategy that learns from results beats one that ignores them, but one short run is only a hint.

#What Comes Next

You have now seen the three ideas that the rest of this track builds on. There is a loop of state, action and reward. There is a score that arrives after the fact, often late. And there is a tug between exploring and exploiting.

The bandit game has no changing situation. Real problems do: a chess board changes after every move, and a robot moves through a house. In the next lesson we add a changing world, a small grid, and a table that remembers how good each move is in each cell. That table is the first real reinforcement learning method, and you will fill it in by hand.

#Where to Go Next

• Next: Your First Q-Table. It adds a small grid world and a table of scores, and you fill the table in one step at a time by hand.
• Then: RL Foundations gives the formal words for what you just did, such as state, policy and value.
• For bandits in more depth: Exploration vs Exploitation covers smarter strategies than epsilon-greedy.
• Programs that act: Agent Fundamentals shows how the same act-and-observe loop appears in language-model agents.

This track has five levels. This is the start, Level 0. After it come the foundations, the core algorithms, the advanced methods, and the expert topics, ending with RL for language models.

In one sentence: read the grid lesson next, then follow the track in order.

#Key Takeaways

• Reinforcement learning teaches a program by tries and scores, with no sheet of right answers
• The loop has three beats: the agent sees a state, picks an action, and the environment answers with a reward and a new state
• A reward is a score, not an instruction, and it is often delayed, so the agent must work out which earlier choices earned it
• A bandit game is the loop with the changing world removed, and the best machine so far is only a guess until there are many pulls
• Exploring learns more and exploiting uses what you know, and a good learner does mostly the second with a little of the first

#Quick Check