Text Conditioning: How a Prompt Steers the Picture

A picture maker cannot read. It works on numbers. So before your sentence can steer anything, it must be turned into numbers that keep its meaning. This lesson follows that journey. A text encoder turns the words into a list of number lists. Then, at every cleanup step, each part of the picture looks at that list and takes what it needs. That second step is called cross-attention.

Recap of the earlier lessons. In From Noise to Picture you saw that a model cleans a picture of static a little at a time, and that a prompt steers each cleanup. In Diffusion Models you met the guidance scale s. This lesson opens the box labelled "the prompt steers" and shows what is inside.

Learning Objectives

After this lesson, you will be able to:

Say why a prompt must become numbers, and what an embedding sequence is
Explain in plain words how CLIP learns to match pictures with captions, and name its limits, including the token limit
Say why a larger language encoder tends to follow long prompts better, as a general trend and not a claim about one product
Explain cross-attention with the library picture, then compute one round of it by hand with queries, keys and values
Say where cross-attention sits inside the denoiser and how it is used at every step
Use the guidance formula with a negative prompt, and name the limits of prompt weighting
Describe counting, attribute binding and text rendering as known weaknesses, with the likely reasons

Prerequisites

Before You Start

Review

Words you need8 terms

A small piece of text that a model reads as one unit. It is often a whole word or a part of a long word.

A list of numbers that stands for a piece of meaning. Pieces with similar meaning get lists that are close to each other.

A network that reads the prompt once and turns it into one embedding for every token.

A pair of networks, one for pictures and one for text, trained so that a picture and its caption land close together.

A step where one list (the picture parts) looks at a different list (the prompt tokens) and takes a weighted mix of what it finds.

Three roles. The query is a question, the key is a label that says what an item is about, and the value is what the item hands over.

A dial for how strongly the cleanup is pushed toward the prompt. At s = 1 there is no extra push.

Words describing what you do not want. The guidance step is pushed away from them.

#Why a Model Needs Numbers, Not Words

Imagine a chef who only understands weights and temperatures. You cannot hand over a recipe card that says "make it cozy". You must say "200 degrees for 30 minutes". Someone has to translate. A picture maker is like that chef. Its inner machinery adds, multiplies and compares numbers. It has no idea what the letters c, a, t mean.

So the first job is translation. The translator is called a text encoder. It reads your sentence and writes out numbers. The key requirement is that the numbers keep the meaning. The words "puppy" and "dog" should give numbers that are close to each other. The words "puppy" and "tractor" should give numbers that are far apart.

Here is a tiny picture of that idea with just two numbers per word. Say the first number is "how animal-like" and the second is "how machine-like". Then "dog" might be (0.9, 0.1), "puppy" might be (0.8, 0.1) and "tractor" might be (0.1, 0.9). The distance from dog to puppy is small. The distance from dog to tractor is large. Real encoders use hundreds of numbers, not two, and the numbers are learned, not chosen by hand. The idea is the same.

Why not just give the picture maker a fixed menu, such as "dog", "cat" and "tractor"? Because people write endless new sentences. A menu cannot hold them. A learned encoder can place any sentence on the map, even one it has never seen, because it learned how words combine. That is why you can type "a tractor wearing a party hat" and get a response.

Where does this step sit? Look at the whole pipeline from the Latent Diffusion lesson. The prompt goes into the text encoder once, at the start. The result is then used at every cleanup step. This lesson covers the text half of that pipeline. The picture half is in the other lesson.

Read the diagram in two rows. The top row happens once and turns words into the embedding sequence. The bottom row repeats at every cleanup step, and each time the denoiser reads the same sequence again.

In one sentence: A picture maker needs the prompt as numbers that keep its meaning, and a text encoder makes them once at the start.

#CLIP: Matching Pictures with Captions

The most famous text encoder for picture makers is the one inside CLIP. To see why it works, look at how it was trained. The training did not ask anyone to label things by hand. It used pictures from the web that already had captions.

Here is the game. Take a big batch of pictures, each with its own caption. Show the picture network all the pictures. Show the text network all the captions. Each network turns what it sees into one list of numbers. Now ask: which caption belongs to which picture? The goal is that a picture and its own caption get lists that point the same way. A picture and a wrong caption should point different ways.

A tiny example makes it concrete. Take a batch of 3 pictures: a dog, a car and a beach. Take the 3 captions: "a dog on grass", "a red car" and "waves on sand". There are 9 possible pairings. Only 3 are correct, the diagonal. Training raises the match score for those 3 and lowers it for the other 6. After millions of batches, the networks put a dog photo and the words "a dog on grass" close together. This shared place is called a shared space: pictures and text both live on the same map of meaning.

CLIP was introduced in a 2021 paper by Radford and colleagues. Its text half is a transformer that reads the caption and outputs vectors. The picture maker borrows only that text half. The trick is that this text half learned meaning that lines up with how things look, because it was trained next to pictures.

What Do You Think?

A prompt is far longer than the number of tokens the text encoder can read. What do you expect a common text-to-image tool to do with the extra words?

The CLIP text encoder used in many open image models has a limit of 77 tokens. Two of those slots are used for a start marker and an end marker. That leaves about 75 slots for your words. A token is often a whole word, but a rare word may use several tokens. So 75 tokens can be fewer than 75 words. In common implementations, words past the limit are cut off.

CLIP has other limits worth knowing:

It learned from web captions. Web captions are short, loose and often wrong, so the encoder is better at the gist of a scene than at long, precise instructions.
It matches whole sentences to whole pictures. That suits "what is this about", not "which object is left of which".
It sees tokens, not letters. Spelling is hidden inside the token pieces.

In one sentence: CLIP trains a picture network and a text network so a picture and its caption land close together, and its text half gives the picture maker meaning that lines up with how things look.

#Language Encoders and Long Prompts

CLIP is not the only choice. A second family is the pure language encoder. The best-known example is T5, from work by Raffel and colleagues. A T5-style encoder was trained only on text, with no pictures. Its job during training was to read text and fill in or continue missing parts. So it learned a lot about grammar, word order and who did what to whom.

Picture two readers. One has seen many captioned photos and knows how things look. The other has read a huge amount of text and knows how sentences work. A short prompt like "a red car" suits the first. A long prompt like "a small red car parked behind a tall blue truck" needs the second, because sentence structure carries the meaning.

Many newer picture makers therefore use a language encoder, often next to a CLIP-style one. Two things about these encoders are worth knowing in general terms:

They usually allow longer prompts. Their limit is set by how the system is built, and it is often larger than the CLIP limit.
Bigger ones tend to follow long prompts better. One 2022 paper on an image model called Imagen reported that making the frozen text encoder larger helped more than making the picture network larger. As a general trend, an encoder with more capacity captures more of how a sentence is built.

This is a general trend, not a promise about any one product. A particular system may use several encoders, different sizes and different limits, and those details change often. When you read about a specific model, check its own documentation.

A small test shows why this matters. "A dog chasing a cat" and "a cat chasing a dog" use the same words. An encoder that only noted which words appear would treat them alike. A stronger encoder keeps the order, so the two give different embeddings.

In one sentence: Language encoders learn sentence structure from text alone, and as a general trend larger ones carry more of a long prompt, but this does not make any specific product immune to errors.

#The Embedding Sequence

Now look at what the text encoder actually hands over. It does not give one list of numbers for the whole prompt. It gives one list for every token. Together these form the embedding sequence.

Take a concrete size. With the common CLIP text encoder, the sequence has 77 slots and each slot holds 768 numbers. In array terms, the shape is 77 by 768. Slots past the end of your prompt are filled with padding. For the prompt "a red cat", the first few slots hold the start marker, "a", "red", "cat", the end marker, and the rest is padding.

Why a sequence and not a single summary? Think of the difference between a book's title and its pages. A title is short but loses detail. Pages keep the detail. A sequence lets each part of the picture ask for the word it needs. A patch that shows fur can read "cat". A patch that shows paint can read "red". A single summary vector would blend all of that together first.

Here is the shape of the data, step by step, for one prompt.

Stage	What it holds	Example shape
Prompt	Text	"a red cat"
Tokens	Whole numbers, one per token	77 ids, padded
Embedding sequence	A list of numbers per token	77 by 768
Used by the denoiser	Read through cross-attention	at every step

Every number in that table is an example from one common setup. Other setups use other sizes. What stays the same is the pattern: one list of numbers per token, computed once, read many times.

In one sentence: The text encoder returns one list of numbers per token, called the embedding sequence, so each part of the picture can look up the word it needs.

#Cross-Attention: The Library Picture

Now the main idea. The denoiser has a noisy picture and a sequence of word vectors. How does a part of the picture use the words? It uses cross-attention.

Start with a picture, not a formula. Imagine a library. The picture is divided into patches, like tiles on a floor. Each patch is a visitor with a question. The prompt words are the books on the shelf. Each book has a label on its spine and some contents inside.

A visitor walks in with a question, such as "what do I look like here?" The visitor compares the question with every spine label. Some labels match well, others barely. The visitor then takes a mix of the contents of all the books, taking more from the books whose labels matched better. Different questions lead to different mixes.

That is the whole mechanism. In short: the image patches ask, the words answer.

Notice three things in the library picture.

Every visitor asks its own question. Different patches read different words, so one prompt can steer many parts differently.
The visitor takes a mix, not one book. A patch can lean on "cat" and still take a bit of "hat".
The books do not change. The words are the same for every visitor and at every step. Only the questions change, because the picture changes.

Now the contrast with self-attention, which you may know from the transformer lessons. In self-attention the visitors ask each other. The picture patches look at other picture patches. In cross-attention the visitors look at a different group, the words. That is the meaning of "cross": it links two different lists.

Here is a second small question before the numbers. Think about it, then check your answer against the next section.

What Do You Think?

In a library with three books labelled red, cat and hat, a visitor comes with a fur-like question. Which book do you expect it to take the most from, and does it take from only that book?

In one sentence: In cross-attention each picture patch asks a question, compares it with the label of every word, and takes a weighted mix of what the words hold.

#Queries, Keys and Values with Tiny Numbers

Now give the library its proper names. The visitor's question is the query. The label on each spine is the key. The contents of each book are the value. These are the same three roles you may have seen in self-attention. The only new fact is where each comes from.

The queries come from the picture patches.
The keys and values come from the word vectors.

The three steps, in order:

Score. For one patch, compare its query with every key. The comparison is a dot product, which is large when two lists point the same way. Divide by the square root of the list length so the scores stay in a steady range.
Share. Turn the scores into shares that add up to 1. The tool for that is the softmax function, which makes bigger scores into bigger shares.
Mix. Multiply each word's value by its share and add them up. That sum is what the patch receives.

Now do it with real numbers. Use lists of just 2 numbers. The first is "animal-ness" and the second is "colour-ness".

The keys, one per word: red is (0.2, 2.0), cat is (2.0, 0.2) and hat is (1.0, 1.0). The values, what each word hands over: red is (0, 1), cat is (1, 0) and hat is (0.5, 0.5). One patch, the fur patch, has the query (2.0, 0.2).

Step 1, the scores. The dot product of the query with the key for red is 2.0 times 0.2 plus 0.2 times 2.0, which is 0.8. For cat it is 2.0 times 2.0 plus 0.2 times 0.2, which is 4.04. For hat it is 2.0 times 1.0 plus 0.2 times 1.0, which is 2.2. Divide each by the square root of 2, which is about 1.414, and you get about 0.57, 2.86 and 1.56.

Step 2, the shares. The softmax of those three scores gives about 0.07 for red, 0.73 for cat and 0.20 for hat. They add up to 1.

Step 3, the mix. Take 0.07 of red's value, 0.73 of cat's value and 0.20 of hat's value. The result is about (0.83, 0.17). The fur patch received mostly "animal" and only a little "colour". That is what we wanted.

Now the paint patch, with query (0.2, 2.0). By the same steps the shares come out as 0.73 for red, 0.07 for cat and 0.20 for hat. The mix is about (0.17, 0.83), mostly "colour". Same words, different question, different answer. You will run this exact example in the hands-on section.

In formula form, for all patches at once, this is:

\text{output} = \operatorname{softmax}\!\Big(\frac{Q K^{T}}{\sqrt{d}}\Big)\, V

In a real model the queries, keys and values are not the raw vectors. Picture features are multiplied by one learned matrix to give queries. Word vectors are multiplied by two other learned matrices to give keys and values. Training teaches these matrices what is worth asking, labelling and handing over. The tiny example skipped that and wrote the keys and values directly.

The tool below shows the same pattern of looking back across two lists. A caution before you use it: it is a translation example, not a picture example. It shows an English sentence on the left and its French translation on the right. Each French word looks back at the English words. It is the same cross-attention idea, with words on both sides instead of picture patches on one side.

Loading visualization...

Drawn stand-in, not a trained model: the attention weights are hand-set peaks plus a little seeded noise, shown for a translation and not for a picture.

Do: Click different French words in the Target token row. Press Resample to redraw the weights.

Notice: Each chosen word sends its thickest line to the English word it translates, and the bars show a share for every English word, adding up to 100 percent.

Why: A query looks at every key and takes a weighted mix, so the strongest match gets the biggest share but not all of it. In a picture model the asking side would be picture patches, not French words.

Now the picture version. Here the asking side really is the picture: each of the 64 patches of a small latent asks the prompt "a red cat on a blue sofa" which word it is about.

Try it: where each word is looked atInteractive

Loading visualization...

Do: Click the word cat, then sofa, then on. Drag the denoising step from 1 up to 20. Press Swap red to green.

Notice: Each word lights up the patches that hold its object, and the glow is a smear at step 1 and a sharp shape at step 20. After the swap the glow for the colour word stays on the cat, and only the cat's colour in the small picture changes. Click any patch to see its shares add up to 100 percent.

Why: Every patch scores itself against every word and a softmax turns the scores into shares. Early on the picture is mostly static, so the question is vague. Drawn stand-in, not a trained model: the scene and the word keys are hand-set, but the softmax over words is computed for real.

Cross-attention is cheap in one useful way. The number of words is small and fixed, while the number of picture patches is large. The derivation below shows how that changes the cost. It is optional reading.

In one sentence: Queries come from the picture patches, keys and values come from the words, and each patch takes a softmax-weighted mix of the word values.

#Where Cross-Attention Sits in the Denoiser

Now place the mechanism inside the denoiser. In the classic Stable Diffusion design, the denoiser is a U-Net, a network shaped like the letter V that shrinks the picture grid and then grows it back. You met it in the Latent Diffusion lesson. Inside it are many blocks, and several of them contain cross-attention.

In a typical block of this kind, the order is roughly this: convolution layers that look at neighbouring cells, then self-attention among picture cells, then cross-attention to the prompt. Cross-attention is therefore not a one-time input at the start. It is a layer that repeats at several sizes of the picture grid, from coarse to fine. At the coarse sizes it can steer the big layout, such as where the cat is. At the fine sizes it can steer details, such as the colour of the fur.

Two timing facts matter for understanding cost and behavior.

The text encoder runs once. The embedding sequence is computed before the loop and reused.
Cross-attention runs at every step. The queries change with the picture, so the shares are recomputed each step. A word can matter early, when the layout forms, and matter less later.

The timestep, which says how noisy the picture currently is, goes in by a separate route. It is not part of the prompt. The denoiser therefore has three inputs: the noisy picture, the timestep, and the embedding sequence.

Newer designs change the plumbing but not the idea. Some replace the U-Net with a transformer. In those, the words may be joined into the same attention as the picture tokens, or kept in a separate stream, instead of using a separate cross-attention layer. The picture is still steered by letting picture tokens read word vectors. The Diffusion Transformers lesson covers that family.

The diagram is simplified. Real designs differ in the exact order and in how many blocks they stack. Remember the arrow into the cross-attention box: the words reach the picture there. Steering by something other than words, such as an edge map or a pose, is covered in Conditional and Controllable Generation.

In one sentence: Cross-attention is a layer that repeats inside the denoiser at several picture sizes and at every cleanup step, always reading the same embedding sequence.

#Classifier-Free Guidance and Negative Prompts

Cross-attention lets the prompt influence the denoiser. But influence alone is often too weak. A model trained on a mix of prompts may still drift from yours. Classifier-free guidance strengthens the pull. This is a recap of the Diffusion Models lesson, with one new use at the end.

At every step the model guesses the noise twice. One guess uses the prompt, called the conditional guess, written eps_c. The other uses an empty prompt, called the unconditional guess, written eps_u. The difference between the two points in the direction the prompt adds. Guidance starts from the empty-prompt guess and adds s times that difference:

\hat{\epsilon} = \epsilon_u + s\,(\epsilon_c - \epsilon_u)

This is the convention used across the track. Some papers write the same formula with a different scale, so always check which one a source uses. Here, s = 1 means no extra push and s = 0 means the prompt is ignored.

A worked example with two-number guesses. Let eps_u = (0.10, -0.20) and eps_c = (0.30, 0.10). The difference is (0.20, 0.30). At s = 1 the guided guess is (0.30, 0.10), which is exactly eps_c. At s = 3 it is (0.10, -0.20) plus 3 times (0.20, 0.30), which is (0.70, 0.70). At s = 7.5 it is (1.60, 2.05). As s grows, the guess moves further along the prompt's direction. That extra distance is why a high s follows the prompt strictly and can also look overdone.

Try the dial. The four pictures in the tool are simple drawn shapes.

Loading visualization...

Drawn stand-in, not a trained model: simple shapes that change with s.

Do: Drag the guidance scale s from 1 to 15 and press reseed.

Notice: At low s the four samples differ a lot. Near the top they look alike and overdone.

Why: A larger s pushes every sample further along the prompt direction, which removes variety.

Two guesses per step means two denoiser calls, so guidance roughly doubles the cost of each step.

Now the new part, negative prompts. Before reading how they work, make a prediction.

What Do You Think?

In classifier-free guidance, one of the two guesses is made with an empty prompt. If a tool supports a negative prompt, which guess do you expect the negative prompt to take the place of?

In common implementations the negative prompt replaces the empty prompt in the reference guess. Call that guess eps_n, made with the negative words. The formula becomes:

\hat{\epsilon} = \epsilon_n + s\,(\epsilon_c - \epsilon_n)

Add a made-up eps_n = (0.20, 0.30) to the example. At s = 7.5 the guided guess is (0.20, 0.30) plus 7.5 times (0.10, -0.20), which is (0.95, -1.20). It lands in a different place than before, (1.60, 2.05), because the reference changed. In words: the empty prompt means "away from nothing in particular", while a negative prompt means "away from this specific thing".

Some practical notes follow from the formula.

A negative prompt is not a filter. It changes a direction. It does not delete anything from the finished picture.
It works best on features the encoder can name. If it places "blurry" on the map, it can push away from it.
It costs nothing extra. The empty-prompt guess is replaced, not added to.

In one sentence: Guidance starts from a reference guess and adds s times the push toward the prompt, and a negative prompt swaps the reference so the push moves away from the unwanted words.

#Prompt Weighting and Its Limits

Many tools let you mark some words as more important. The rough idea is that the weight scales the vector of that word before it is used. A weight of 1 leaves the word alone. A weight above 1 makes its vector bigger, and a weight below 1 makes it smaller. The exact way of writing weights, and even whether the tool normalizes the result, differs from tool to tool.

A small example. Say the vector for "red" has numbers (0.2, 2.0). A weight of 1.5 turns it into (0.3, 3.0). When a patch compares its query to this larger key, the score rises, so the share for "red" rises. The patch takes more from "red" and less from the other words.

It is a useful nudge, but it has limits.

It is a blunt tool. Scaling a vector does not tell the model what you mean. It only makes that word louder.
Large weights distort. A vector scaled far from its usual size may fall outside what the model saw in training. Pictures can become strange or overdone.
It changes the shares for all words. Because the shares add to 1, a louder word takes share from the others. A heavy weight on one word can make another word seem ignored.
It does not fix a weak encoder. If the encoder blends two ideas, scaling one does not separate them.
Syntax differs. There is no standard way to write weights, so a prompt copied between tools may behave differently.

A good habit is to try the plain prompt first, then change one thing at a time. Reword before you weight, since a clearer sentence often does more than a louder word.

In one sentence: Prompt weighting scales a word's vector so it takes a bigger share, which helps a little but is blunt, varies between tools, and cannot repair what the encoder blended.

#Known Weaknesses: Counting, Binding and Text

Even with a good encoder and cross-attention, text-to-image models have known weaknesses. These appear across the field and are widely reported. This section describes the pattern and offers likely reasons. It does not claim that any one product has or lacks them, because systems change quickly and you should test the one you use.

Counting. A prompt such as "three apples" may give two, four or five. A likely reason is that the encoder stores "three" as a vector close to "two" and "four", because all of them appear in similar sentences. Closeness in meaning is not the same as an exact number. A second reason is that cross-attention hands each patch a soft mix, with no counter anywhere inside it. The picture network has no step that checks the total.

Attribute binding. Take "a red cube on a blue ball". A common error gives a blue cube or a red ball, or colours both objects. Binding means attaching the right property to the right object. In cross-attention, each patch takes a weighted mix of all the words. A patch for the cube can take a little "blue" if the share for "blue" is not very small, and the colour leaks across. A weaker encoder that blends the words together makes this worse. Stronger language encoders tend to help, as the earlier section said, though they do not remove it.

Text in pictures. Asking for a sign that reads "OPEN" often gives warped letters. Likely reasons include the following.

Tokens hide spelling. A word may be one token, so the letters inside it are not visible to the model.
Letters need exact shapes. A small error in a curve turns an "O" into something else, and people notice at once.
Training pictures that show text are a small share of the data, and captions rarely give the exact text.

Left and right. "A cat to the left of a dog" can come out reversed, since swapping left and right changes the sentence very little in embedding space.

Notice the shared cause. All four weaknesses involve exact structure: a number, a pairing, a spelling, a side. Text conditioning as described here carries meaning that is soft and blended. Exact structure is where soft blending breaks down.

What helps in practice, as general advice and not a promise:

Name each object once and keep its properties next to it in the sentence.
Ask for fewer objects per picture, and generate several pictures to pick from.
Add exact text afterwards in an editing tool.
Use layout controls from the controllable-generation lesson when position matters.

In one sentence: Counting, binding colours to objects, spelling and left versus right are known weak spots, because they need exact structure while text conditioning carries soft, blended meaning.

#Hands-On: Cross-Attention and Guidance in Numpy

The cell below runs both worked examples from this lesson in numpy. Part 1 is one round of cross-attention with three words and two picture patches, using the keys, values and queries from the tiny example. Part 2 is classifier-free guidance on three made-up noise guesses, with and without a negative prompt. Nothing here is trained. The numbers are chosen so you can check them by hand.

pythonplayground.py · Pyodide

import numpy as np

# Part 1: one round of cross-attention with tiny made-up numbers.
# Each word and each picture patch is a list of 2 numbers:
# the first says "how animal-like", the second says "how colour-like".
words = ["red", "cat", "hat"]
keys = np.array([[0.2, 2.0],    # red: mostly colour
               [2.0, 0.2],    # cat: mostly animal
               [1.0, 1.0]])   # hat: a bit of both
values = np.array([[0.0, 1.0],  # what "red" hands over
                 [1.0, 0.0],  # what "cat" hands over
                 [0.5, 0.5]]) # what "hat" hands over
patches = {"fur patch": np.array([2.0, 0.2]),
         "paint patch": np.array([0.2, 2.0])}

def softmax(x):
  e = np.exp(x - x.max())
  return e / e.sum()

d = keys.shape[1]
for name, q in patches.items():
  scores = keys @ q / np.sqrt(d)     # one match score per word
  weights = softmax(scores)          # turn scores into shares that sum to 1
  out = weights @ values             # weighted mix of the word values
  print(name)
  for w, wt in zip(words, weights):
      print(f"  weight on {w}: {wt:.2f}")
  print("  mixed result:", np.round(out, 2))

# Part 2: classifier-free guidance on two made-up noise guesses.
eps_u = np.array([0.10, -0.20])   # guess with an empty prompt
eps_c = np.array([0.30, 0.10])    # guess with the prompt
eps_n = np.array([0.20, 0.30])    # guess with a negative prompt
print()
for s in (1.0, 3.0, 7.5):
  guided = eps_u + s * (eps_c - eps_u)
  print(f"s={s}: guided guess = {np.round(guided, 2)}")
neg = eps_n + 7.5 * (eps_c - eps_n)
print("negative prompt, s=7.5:", np.round(neg, 2))

# CHALLENGE: change the fur patch query to [1.0, 1.0].
# Which word gets the biggest share now?

Tests · Check that the fur patch puts its largest weight on cat (0.73) and the paint patch on red (0.73), that each row of weights sums to 1, that s=1 returns exactly eps_c, and that the negative-prompt result differs from the empty-prompt result at the same s.

The same ideas on a real text encoder need a deep-learning library and downloaded weights, so they do not run on this page. Copy the block below to Google Colab to see the 77-slot limit and a guided picture.

pythonrunnable cell

# Copy to Colab: needs torch, transformers and diffusers, so it cannot run on this page.
import torch
from transformers import CLIPTokenizer, CLIPTextModel

name = "openai/clip-vit-large-patch14"
tokenizer = CLIPTokenizer.from_pretrained(name)
encoder = CLIPTextModel.from_pretrained(name)

# 1. The embedding sequence: 77 slots, 768 numbers each.
prompt = "a red cat wearing a small hat"
ids = tokenizer(prompt, padding="max_length", max_length=77, return_tensors="pt")
with torch.no_grad():
    seq = encoder(ids.input_ids).last_hidden_state
print(seq.shape)  # torch.Size([1, 77, 768])

# 2. The token limit: a very long prompt is cut to 77 tokens.
long_prompt = "a red cat " * 100
cut = tokenizer(long_prompt, truncation=True, max_length=77)
print(len(cut.input_ids))  # 77

# 3. Guidance and a negative prompt in a full pipeline.
from diffusers import StableDiffusionPipeline
model_id = "YOUR-CHECKPOINT-ID"  # a Stable Diffusion checkpoint you have access to
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16).to("cuda")
image = pipe(
    prompt,
    negative_prompt="blurry, low quality",
    guidance_scale=7.5,        # this is the s in the formula
    num_inference_steps=30,
).images[0]
image.save("cat.png")

#Key Takeaways

A picture maker works on numbers, so a text encoder first turns the prompt into one list of numbers per token, called the embedding sequence
CLIP trains a picture network and a text network so that a picture and its caption land close together, and its text half carries meaning that lines up with how things look
A common CLIP text encoder reads 77 tokens, about 75 usable, and in common implementations extra words are cut off
Language encoders learn sentence structure from text, and as a general trend larger ones follow long prompts better
In cross-attention the picture patches ask and the words answer: queries come from the patches, keys and values come from the words, and each patch takes a softmax-weighted mix
The text encoder runs once, and cross-attention runs at several sizes inside the denoiser at every step
Guidance is eps_u + s * (eps_c - eps_u), and a negative prompt swaps the empty-prompt guess for a guess made with the unwanted words
Prompt weighting is a blunt nudge, and counting, binding, spelling and left versus right are known weak spots because they need exact structure

#Quick Check

Quick Check1 / 4

In cross-attention for a text-to-image model, where do the queries, keys and values come from?

Where to next. Go to Latent Diffusion to see how the picture is squeezed into a small latent so that all of this runs quickly. To steer with more than words, read Conditional and Controllable Generation. To see how newer denoisers take in text, read Diffusion Transformers.

Text Conditioning: How a Prompt Steers the Picture

Learning Objectives

After this lesson, you will be able to:

Say why a prompt must become numbers, and what an embedding sequence is
Explain in plain words how CLIP learns to match pictures with captions, and name its limits, including the token limit
Say why a larger language encoder tends to follow long prompts better, as a general trend and not a claim about one product
Explain cross-attention with the library picture, then compute one round of it by hand with queries, keys and values
Say where cross-attention sits inside the denoiser and how it is used at every step
Use the guidance formula with a negative prompt, and name the limits of prompt weighting
Describe counting, attribute binding and text rendering as known weaknesses, with the likely reasons