When you drop a screenshot into Claude Sonnet 4.6 or GPT-4o and ask "what's wrong with my code?", a vision encoder converts the image into the same token space your text lives in. That's the whole trick. Vision-language models — LLaVA, Flamingo, Claude 3.5+, GPT-4o, Gemini 2 — are LLMs with a visual translator bolted on. This lesson shows you exactly how the bolt works.
Learning Objectives
After this lesson, you will be able to:
Understand VLM's three parts: a vision encoder (ViT/SigLIP), a projection layer (linear or MLP), and a language model (LLM)
See how image patches become visual tokens -- a ViT splits a 224×224 image into 16×16 patches, each producing a feature vector that the projection layer maps into the LLM's embedding space
Understand LLaVA-style two-stage training: first align the projection on image captions (Stage 1), then fine-tune the full model on visual instruction data (Stage 2)
Know why the projection layer is the critical architectural bridge -- and how its design (linear vs MLP vs cross-attention) affects performance
Understand visual instruction tuning: what data it uses, why the visual encoder is frozen during fine-tuning, and how hallucination arises from language priors overriding visual evidence
Compare key VLMs: LLaVA (open-source, simple MLP projection), GPT-4V (proprietary), Gemini (natively multimodal from pretraining), and Claude Vision
Try it: See how VLMs convert image patches to LLM tokensInteractive
Loading visualization...
Try it! Upload any image to ChatGPT or Claude right now and ask "What do you see in this image?" Then ask a follow-up question about a specific detail. You are using a VLM in real time -- the exact architecture described in this lesson.
Select a model to see its specific pipeline. Adjust the patch grid to see how more patches produce more visual tokens. Notice that all three stages -- vision encoder, projection, and LLM -- work together to turn pixels into language.
The user provides an image (e.g., a photo of a whiteboard with equations) and a text question ("What is the derivative of the second equation?"). These are two fundamentally different modalities that need to be brought into a common representation.
The image is passed through a pretrained vision transformer. Just like in ViT, the image is split into patches (typically 14x14 or 16x16 pixels), each patch is linearly projected, and the sequence passes through transformer encoder layers. The output is a sequence of visual feature vectors -- one per patch. For a 224x224 image with 14x14 patches, this produces 256 feature vectors of dimension 1024 or higher.
The visual features live in the vision encoder's embedding space (e.g., 1024-dim). The language model expects tokens in its own embedding space (e.g., 4096-dim for a 7B model). The projection layer bridges this gap.
In the simplest case (LLaVA v1), this is a single linear layer: W_proj : R^1024 -> R^4096. More sophisticated designs use a 2-layer MLP with GELU activation, cross-attention pooling (Flamingo), or a perceiver resampler (Qwen-VL) that compresses 256 visual tokens down to 64 or fewer.
The most influential open-source VLM. LLaVA's design is elegant in its simplicity:
Visual encoder: CLIP ViT-L/14 (frozen during initial training)
Projection: Single linear layer (LLaVA v1) or 2-layer MLP (LLaVA v1.5+)
LLM: Vicuna-7B or 13B (later versions use Llama-2, Llama-3)
Key insight: With only the projection layer trained initially (feature alignment), then the full model fine-tuned on visual instruction data, a simple architecture achieves strong performance
Architecture details are proprietary, but the system handles interleaved image-text conversations, supports multiple images, and can reason over charts, diagrams, screenshots, and photos
GPT-4o processes images natively within the same model (rather than a separate encoder), suggesting tighter multimodal integration
A VLM correctly identifies all objects in an image but says there are '3 dogs' when there are only 2. What type of error is this?
The answer reveals a key VLM challenge: the language model can hallucinate details that are not in the image. The visual encoder may correctly represent two dogs, but the LLM's language priors can override visual evidence when generating text. This is why visual instruction tuning is so important.
VLMs combine three components: visual encoder, projection layer, and language model -- the visual encoder (ViT/SigLIP) converts images to patch features, the projection maps them into the LLM's embedding space, and the LLM processes interleaved visual and text tokens
The projection layer is the critical bridge -- it translates between vision and language representation spaces, and its design (linear, MLP, cross-attention) significantly impacts performance
Visual instruction tuning teaches models to follow image-based instructions -- a two-stage process first aligns visual features with the LLM's embedding space, then fine-tunes the full model on visual question-answer pairs
VLMs inherit both LLM and vision model limitations -- they can hallucinate visual details, struggle with counting and spatial reasoning, and are constrained by the visual encoder's pretrained capabilities
What is the role of the projection layer in a VLM?
Vision Language Models give LLMs the ability to see. They represent the convergence of computer vision and natural language processing into unified multimodal intelligence. The architecture pattern -- encode modality, project to shared space, process with a foundation model -- is now being applied to audio, video, and even robotics.
The projected visual tokens are inserted into the token sequence alongside the text tokens. A typical format:
[visual_token_1] [visual_token_2] ... [visual_token_256] [USER] What is the derivative of the second equation? [ASSISTANT]
The language model now has a unified sequence of ~300+ tokens containing both visual information and the text question. Special tokens or positional cues tell the model which tokens are visual vs textual.
The language model (e.g., Llama, Vicuna, Mistral) processes the entire interleaved sequence using causal self-attention. Visual tokens attend to other visual tokens (spatial reasoning), text tokens attend to visual tokens (grounding language in the image), and the model generates a response autoregressively.
This is where the magic happens: the same attention mechanism that lets the model understand "the second equation" in text also lets it locate the second equation in the visual tokens.
The model generates text tokens one at a time: "The second equation is y = 3x^2 + 2x. Taking the derivative: dy/dx = 6x + 2." The generation process is identical to a text-only LLM -- the visual understanding was fully encoded into the hidden states during the forward pass.