Autoregressive and Discrete Generation

#Pictures as Sentences

A sentence is a short list of pieces from a fixed list of words. A language model is good at one thing: given the pieces so far, it scores which piece comes next. Nothing in that job says the pieces must be words. If a picture can be written as a short list of pieces from a fixed list, the same model can write pictures.

That is the whole idea of this lesson. The rest is about three choices. How do you cut a picture into pieces? In what order do you write them? And how many pieces can you write at once?

#Step One: A Tokenizer Turns Pixels Into Codes

A tiny worked number. Take a 256 by 256 picture and a tokenizer that shrinks each side by 16. The grid is 16 by 16, which is 256 cells. If the codebook has 1,024 entries, each cell is one number from 0 to 1,023. The whole picture is now a list of 256 numbers.

The recipe has two stages. Stage one trains the tokenizer on pictures. Stage two freezes the tokenizer and trains a transformer on the grids of codes it produces. To make a new picture, the transformer writes a new grid and the decoder paints it. Audio works the same way: a codec turns sound into codes, a transformer writes codes, a decoder plays them.

Try the snap below. It is the step that turns a smooth vector into a code.

• Do: Drag the point around the plane, and press the random-latent button a few times.
• Notice: The snapped entry changes only when the point crosses a border between two codes. Inside one cell, the output stays the same.
• Why: Snapping to the nearest entry is what turns a smooth value into a token. The snap is computed exactly. The panels that show how the decoded picture looks are drawn illustrations, not a trained decoder.

#Step Two: Predict the Next Code

Now the second stage. Lay the 16 by 16 grid out as a list of 256 numbers, row by row. Train a transformer to predict each number from the numbers before it. This is the same job as a chatbot predicting the next word. A word that is a token of text becomes a code that is a token of an image.

The loop is the one from the chatbot lesson. Score every code. Reshape the scores with a temperature. Roll the weighted die. Add the winner to the list. Repeat. The die decides the variety. A low temperature repeats the surest codes. A high temperature allows odd codes.

The viz below is the text version of that loop, because the drawn stand-in only knows twenty words.

Do: Pick The cat and press Generate Next Token eight times. Press Reset, switch on Greedy Decoding, and generate eight more.

Notice: With sampling, the words wander and differ from run to run. With Greedy, the same few words repeat in a loop.

Why: Each pick joins the context for the next pick. Drawn stand-in, not a trained model: the tokens are twenty words with a hand-written table that looks only at the last word. An image model runs the same loop with codes in place of words, and with a real transformer that reads every earlier code.

The same warning about greedy picking matters for images. A greedy image writer tends to produce flat, repeated areas, because the surest next code is usually the same as the last one. Some temperature is needed. Too much gives noisy patches. The run-it section later shows both ends.

#Raster Order and Its Drawbacks

The simplest order for a grid is raster order. Write row one from left to right, then row two, and so on. Here is a 4 by 4 grid, with the number showing the step at which each cell is written.

 1  2  3  4
 5  6  7  8
 9 10 11 12
13 14 15 16

This order is easy, but it has real drawbacks.

1. Many sequential steps. A 4 by 4 grid needs 16 runs. A 16 by 16 grid needs 256. A 32 by 32 grid needs 1,024. Each run waits for the one before it, so more cells means more waiting.
2. One-way context. When the model writes cell 6, it sees cells 1 to 5 above and to the left. It cannot see the cells to the right or below. An image has no natural reading direction, so this view is an accident of the order.
3. No way to undo. If cell 3 is a poor choice, cells 4 to 16 are built on it. The loop never goes back.
4. Cost of long lists. Attention compares every position with every earlier position, so the work grows with the square of the list length. Longer lists get expensive fast.

None of these makes raster writing wrong. It works, and it is simple. The next two sections show how to keep the codes but change the order and the number of cells written per run.

#Next-Scale Generation: Coarse to Fine

People draw in a different order from raster. They sketch the big shapes first, then add detail. Next-scale generation follows that habit. Instead of predicting the next cell, the model predicts the next whole grid, from small to large.

A picture helps. Think of a stack of grids of growing size.

scale 1:   1 x 1      the overall color
scale 2:   2 x 2      the big blocks
scale 3:   4 x 4      shapes
scale 4:   8 x 8      parts
scale 5:  16 x 16     fine detail

At each scale, the model writes every cell of that grid in one run, all at the same time. The cells of one scale do not wait for each other. They all wait only for the smaller scales. For the five scales above, the picture takes 5 runs. The raster route took 256.

A tiny worked number. The five grids hold 1 + 4 + 16 + 64 + 256 = 341 tokens in total. That is more tokens than the single 16 by 16 grid, which holds 256. So the tokenizer must be trained to produce all scales, and the model writes more tokens in total. The saving is in the number of runs, not in the number of tokens.

The idea is described in the original paper on visual autoregressive modeling (Tian et al., 2024), which calls it next-scale prediction. The scales above are an example, not that paper's exact list. Each run still predicts cells from the same kind of distribution. Only the grouping changes.

#Masked Generation: Fill the Surest Cells First

The best-known method of this kind is MaskGIT (Chang et al., 2022). It uses a transformer that reads the whole grid in both directions, not only left and right. That is a key difference from raster writing: a cell can use clues from every side.

A worked number. With 16 cells and 4 cells kept per round, the picture is done in 4 rounds. Raster writing would use 16 runs. A larger grid gains more. A 16 by 16 grid with 32 cells kept per round takes 8 rounds, not 256 runs. The number of cells to keep per round is a schedule that the designer picks.

Why fill the surest cells first? The easy cells fix the big layout. Later, the hard cells see their neighbors and have fewer choices. This is the coarse-to-fine habit again, found by the model's own confidence instead of a fixed set of scales.

#Discrete Diffusion: Noise That Deletes Tokens

Continuous diffusion destroys a picture by adding Gaussian noise little by little. Discrete diffusion destroys a list of tokens in a different way: it deletes them. At each step, every token has some chance of being replaced by the mask token. Early on, few are replaced. At the end, all are.

Here the "amount of noise" is the fraction of masked cells. A tiny worked number: a grid of 16 cells at noise level 0.25 has about 4 masked cells on average. At 0.5 about 8. At 1.0 all 16. The model is trained to look at a partly masked grid and guess the original token under each mask.

Generating is the reverse run. Start at noise level 1 with every cell masked. At each step, the model predicts the original tokens, you un-mask some of them, and you move to a lower noise level. That is the masked-fill loop of the last section. Discrete diffusion gives it a training recipe that covers every noise level, and a way to describe the loop in the same language as continuous diffusion.

Compare the two kinds of noise side by side.

The mask makes one thing easy that is hard in continuous diffusion. You know which cells are unknown. A user can keep some tokens fixed and mask only the rest, which is the heart of inpainting and of completing a partly written list.

Discrete diffusion for general token lists was studied in the D3PM paper (Austin et al., 2021). Masked versions have also been applied to text. The details of the training weights differ between papers, so this lesson stays with the idea.

#Three Ways Side by Side

Here is a general comparison. These are trade-offs that hold broadly across the families. They are not scores for any named product, and the numbers depend on model size, code, hardware and training.

#Run It: A Toy Image Maker

This is a toy, not a trained image model. The codebook has four hand-made entries: sky, sun, tree and grass. The training scenes are made by a short rule. The "model" is a table of counts, not a neural network. It shows the loops, not the quality of a real system.

The cell has three parts. First, it counts training scenes and samples 4 by 4 grids in raster order at three temperatures. Second, it runs the masked-fill loop and prints the round in which each cell was filled. Third, it draws eight hot samples and keeps the best one.

The first block below is the code you would use with a real model. It needs torch and a download, so it is a read-only copy for Google Colab. The second block runs in your browser with numpy.

# Copy to Colab (needs torch and a trained image tokenizer; it cannot run in the browser)
# The shape of stage two: a causal transformer over image codes.
import torch
import torch.nn as nn

class CodeTransformer(nn.Module):
    def __init__(self, n_codes=1024, length=256, dim=256, layers=4, heads=4):
        super().__init__()
        self.tok = nn.Embedding(n_codes + 1, dim)       # +1 for a start token
        self.pos = nn.Embedding(length, dim)
        block = nn.TransformerEncoderLayer(dim, heads, dim * 4, batch_first=True)
        self.body = nn.TransformerEncoder(block, layers)
        self.head = nn.Linear(dim, n_codes)
        self.n_codes = n_codes

    def forward(self, codes):                           # codes: (batch, length)
        n = codes.shape[1]
        x = self.tok(codes) + self.pos(torch.arange(n, device=codes.device))
        causal = torch.triu(torch.ones(n, n, device=codes.device), 1).bool()
        return self.head(self.body(x, mask=causal))     # scores for the next code

@torch.no_grad()
def sample(model, length=256, temperature=1.0):
    codes = torch.full((1, 1), model.n_codes)           # start token
    for _ in range(length):
        scores = model(codes)[:, -1] / temperature
        nxt = torch.multinomial(scores.softmax(-1), 1)
        codes = torch.cat([codes, nxt], dim=1)
    return codes[:, 1:]                                 # then: tokenizer.decode(codes)

Now the browser version.

Typical output has this shape. The three groups of grids are the same four tokens, written at different temperatures. At 0.3 they look almost alike. At 2.5 they look messy. The masked-fill rounds print the cells and their confidence, and then a small table of numbers, one per cell, naming the round in which it was filled. In one run the bottom row was filled first with confidence 1.0, then the second row, then the sky and grass cells with confidence from 0.6 to 0.75. Your exact numbers may differ.

Two limits to remember. The raster table sees only the cell to the left, the cell above and the row number, so it knows nothing about a tree two cells away. And the fill loop here samples a token for every masked cell, then keeps the four surest. That is the same loop as MaskGIT, shrunk to a table.

#Scaling and Test-Time Compute

Two ideas about compute apply to every family above.

A tiny worked number. If one run costs 1 unit and you draw 8 candidates, you spend about 8 units plus the cost of scoring. You hope the best of 8 beats a typical single draw. Whether it does depends on the scorer: a scorer that likes the wrong thing selects the wrong picture. The toy above used the model's own probability as the scorer, which tends to prefer tidy, common-looking grids, and the closing line of the output shows this.

#Where Multimodal Language Models Fit

A multimodal language model reads and writes more than text. There are two common designs for pictures, and a given system may use either or both.

1. Image tokens in the same list. The model's vocabulary includes image codes beside words. It writes a picture by writing image codes with the autoregressive loop, and a decoder paints them. The VQ-VAE lesson names some systems of this kind.
2. A separate image maker. The language model writes a description or a set of control values, and a diffusion model draws the picture. The two parts are trained apart.

The first design is the one this lesson is about. Its attraction is that one model and one loop handle both words and pictures, so a prompt can mix them freely. Its limit is the tokenizer ceiling from earlier. The second design keeps the strong control tools of diffusion but needs two models to cooperate. Which is better depends on the task, so look at how a given system is built before assuming.

#Key Takeaways

• A tokenizer turns a picture into a grid of codes from a codebook. A transformer then writes new grids, and a decoder paints them. The tokenizer sets the quality ceiling
• Raster order writes one code per run in a fixed direction. It is simple, but slow for large grids, sees only one side, and cannot undo a poor choice
• Next-scale generation writes whole grids from small to large, so the number of runs equals the number of scales
• Masked generation starts with every cell masked and keeps the surest guesses each round. Plain, predictable cells come first
• Discrete diffusion is noise that deletes tokens: the forward step masks tokens, and generation runs it backward
• The three families trade control, speed and quality in general ways. Test-time compute, such as best-of-N, spends extra work at the moment of use, and its result depends on the scorer

#Quick Check