Sampling and Latent Space

#A Bag of Examples

Picture a cloth bag full of marbles. Some are red, some are blue, a few are green. You cannot see inside. If someone asks, "What is in the bag?", you could answer with a list: 60 red, 30 blue, 10 green.

Now swap the marbles for something richer. Picture a huge pile of photos of cats. Some cats are orange, some are black. Some sleep, some jump. A few photos are blurry. The pile is also a bag, and it also has a distribution. Most photos have two ears and whiskers. Few have a cat wearing a hat.

A picture maker or a chatbot is trained on a very large pile like this. The pile might hold photos, drawings, or pages of writing. Training means the program studies the pile until it knows what the pile is like.

In one sentence: a distribution is a description of how often each kind of example shows up in a bag.

#Reaching Into the Bag

Take a bag with 90 red marbles and 10 blue ones. You reach in without looking, pull out one marble, write down its colour, and drop it back. You do this ten times.

This is the key link. A single pull is random. A long run of pulls follows the shape of the bag. Common things come out often. Rare things come out rarely, but they do come out.

In one sentence: sampling means reaching into the bag and pulling one example by chance, so common kinds come out often and rare kinds come out rarely.

#From the Bag to Something New

Here is a puzzle. A bag of marbles can only hand you marbles that are already in it. A pile of cat photos is the same. If a program only ever reached into the pile, it would give back photos that already exist. That is a copy machine, not a picture maker.

Think of a cook who has tasted a thousand soups. The cook does not own the soups. The cook knows what soups are like: salty, warm, a little sour. Ask for a new soup, and the cook makes one that nobody has tasted before, yet it still tastes like soup.

Here is the whole flow in one picture.

Where does chance enter? In the reach. If the model always built the same example, it would be a vending machine with one button. Chance gives it a different starting point each time, so each result comes out a little different. The model keeps the habits of cats, but the details change: colour, pose, the tilt of the head.

In one sentence: a model learns the shape of the bag, and chance in the reach makes each new example differ from the ones it studied.

#Latent Space: A Map of Things

The model needs a way to say "this particular cat". Describing a photo by listing every dot of colour would take hundreds of thousands of numbers. There is a shorter way.

Picture a map of a town. You do not describe a house by listing every brick. You give its address: two numbers, such as street and number. Two numbers are enough to point at one house among many.

On a good map, nearby places are alike. Two houses on the same street look more similar than a house here and one across the country. The same holds for a latent map. Points that sit close together stand for pictures that look alike. A cluster of points might all be cats with long fur. Another cluster might be dogs.

Below is a drawn map with 80 dots in five coloured groups. Each dot is one pretend example.

Do: Click one dot, then a second dot in the same colour. Repeat with two dots of different colours. Switch on Cluster Boundaries.

Notice: Two dots close together show nearly the same row of bars below the map. Dots from far-apart groups show different rows.

Why: Nearby points get similar outputs. Drawn stand-in, not a trained model: the bars come from a fixed formula, and the dots are placed by hand.

The bars under the map stand for what the decoder would produce at each dot. They are eight numbers, not a real picture. That is enough to show the idea: close on the map means close in result.

In one sentence: latent space is a map where each point is a short list of numbers standing for one possible picture, and nearby points stand for similar pictures.

#Walking Between Two Points

Imagine two houses on a map, one mile apart. You leave the first and walk toward the second in a straight line. At each step you look at what is around you.

That is the strength of a good map. The walk does not jump from one picture to another. It slides. Every step is a little more like the end and a little less like the start. This is how some tools morph one face into another, or blend two designs.

Try it with four cartoon faces placed at the corners of a square.

Do: Pick a pair of faces, then move the t slider. Switch between Linear (lerp) and Spherical (slerp). Click the square to drop the point off the line.

Notice: Mouth and hair change smoothly. Lerp draws a straight path, slerp a curve around the centre. Off the line, the face blends all four corners.

Why: Each point is decoded by how close it sits to the four corner faces. Drawn stand-in, not a trained model.

Here is what the controls really do. The four corner faces are the anchors: a smiling woman, a smiling man, a frowning woman and a frowning man. The picture labels one direction "gender" and the other "smile". That tidy labelling is a feature of this drawing. On a real trained map the directions are rarely so neat, and you often cannot name them.

The face is not made by a trained model. It is a cartoon, redrawn from a weighted mix of the four corner faces, with more weight on the nearer corners. Below the square is a panel called Latent arithmetic. It shows that the step from a frowning woman to a frowning man, added to a smiling woman, lands near the smiling man. The step carries the same change wherever you apply it.

In one sentence: a walk between two points on a tidy map gives a smooth slide of in-between pictures.

#Postcard Directions: Why a Few Numbers Can Hold a Picture

Something should bother you here. A photo has hundreds of thousands of numbers. A latent code may have ten or so. How can ten numbers stand for a whole picture?

Picture sending a friend a postcard that says: "Beach. Sunset. Two people. Orange sky." That is only a handful of words. Your friend is a skilled painter who has seen thousands of beaches. They read your card and paint a full scene.

The card is short because your friend already knows how beaches look. The card does not carry the sand, the waves or the light. It only carries the choices that matter for this one picture. The painter fills in the rest from what they know.

A latent code is the postcard. The decoder is the painter. During training the decoder learns what pictures in the bag usually look like. After that, a short code is enough to pick one picture out of everything it can paint.

Here is a small count to hold on to. A tiny grey picture, 28 squares wide and 28 tall, holds 784 numbers. A code of 10 numbers is about 78 times shorter. The code can be that short only because the decoder already carries the knowledge of what such pictures look like. Put the decoder aside, and ten numbers alone say nothing.

This also explains why the map has structure. To fit a picture into so few numbers, the model must keep only what matters: the shape, the pose, the mood. Details that every picture shares need no space. So the few numbers tend to track real differences between pictures, and that is why close codes give close pictures.

In one sentence: a short code works like a postcard, because the decoder already knows how to paint the rest.

#The Honest Limits: Holes in the Map

So far the map sounds perfect. It is not. Think of a town map with a lake in the middle. Streets run all around the lake, but nobody lives in the water. If you pick a point in the lake, there is no house there. The map still has a place for it, but nothing real lives at that place.

A latent map can have such holes. If a model is trained only to rebuild its pictures exactly, it may place each picture at its own private spot and leave empty space in between. Walk across an empty gap, and the decoder is asked to paint a place it has never seen. The result may be a smear or a nonsense image.

Two problems follow. Reaching into the map at random may land in a hole. And a walk between two points may cross one.

Three more limits are worth knowing:

• The map is only as wide as the bag. If the pile had few photos of some kind of cat, that part of the map is thin, and results there are poor.
• The map copies the bag's habits, good and bad. If the pile leaned one way, the map leans the same way. Why AI Makes Things Up looks at this.
• A walk is a guess about the middle. The in-between picture is the model's best idea of what lies between. It need not match anything real.

In one sentence: a map can have holes where nothing real lives, so a tidy map is trained to avoid them, but even a tidy one is not perfect.

#Key Takeaways

• A distribution is a description of how often each kind of example appears in a bag.
• Sampling is reaching in and pulling one example by chance. Many pulls follow the shape of the bag.
• A model learns the shape of the bag, not the bag itself. Chance in the reach makes new examples differ from old ones.
• Latent space is a map. Each point is a short list of numbers for one possible picture. Nearby points are alike.
• A walk between two points gives a slide of in-between pictures, if the map is tidy.
• A short code works like a postcard, because the decoder already knows how to paint.
• Maps have holes unless trained to be tidy, and even tidy maps are only as good as the bag they came from.

#Quick Check