What if an agent could USE your computer — clicking, typing, scrolling like a person? Claude's computer use shipped in October 2024, OpenAI's Operator followed in January 2025. They succeed roughly 70% of the time on OSWorld, fail creatively the other 30%, and cost about 10x an API call per task. They're the future of automation, just messier than the marketing suggests.
Learning Objectives
After this lesson, you will be able to:
Understand the big shift: instead of calling APIs with code, computer use agents look at the screen and click buttons like a human would
Walk through the computer use loop: take a screenshot, figure out what is on screen, click or type something, check if it worked
Compare how Claude Computer Use, OpenAI Operator, and browser agents each approach this differently
Identify the hard challenges: the AI has to figure out which pixel to click (visual grounding), pages load slowly (latency), and pop-ups get in the way (error recovery)
Know when to use computer use (no API available, legacy software) versus API-based tools (faster, more reliable, cheaper)
This is the most sci-fi lesson in the entire track. Agents that can see your screen and use any application like a human? It sounds like science fiction, but it is real and shipping today. Prepare to have your mind expanded.
Try it! Try describing a simple UI task step by step, as if you were guiding someone over the phone: "Open the browser. Type kayak.com in the address bar. Click the 'To' field. Type 'Tokyo'. Click the first suggestion." That is exactly the level of detail a computer use agent operates at. The challenge is doing this from a screenshot alone -- no HTML, no DOM, just pixels.
Every agent you have built so far interacts with the world through APIs. You define tool schemas, the LLM generates structured function calls, your code executes them, and the results come back as clean data. This works brilliantly -- when APIs exist. But most software does not have APIs. Your company's internal HR portal, the legacy accounting system, the government tax filing website, the niche industry application that only runs on Windows -- none of these expose APIs. A human uses them by looking at the screen, moving the mouse, clicking buttons, and typing text.
Computer use agents do exactly that. They perceive the screen as an image, reason about what they see, and take actions through mouse clicks and keyboard input. This is the most general-purpose form of agent interaction -- if a human can use it, a computer use agent can (in theory) use it too.
The agent takes a screenshot of the current screen state. This image is the agent's only window into what is happening. Unlike API agents that receive structured JSON, computer use agents must interpret raw pixels. The screenshot is encoded (usually as base64 PNG or JPEG) and sent to the multimodal LLM as part of the conversation.
The LLM analyzes the screenshot. It identifies UI elements: buttons, text fields, menus, labels, images, tables. It reads text rendered on the screen. It understands layout -- which elements are grouped together, which button is associated with which label, where the cursor is, which field is currently focused.
This is where the magic -- and the difficulty -- lives. The LLM must perform visual grounding: connecting its understanding of the task ("click the Search button") to a specific pixel coordinate on the screen. "Search" might appear multiple times on the page. The right "Search" button might be a small icon rather than text. A loading spinner might be covering it.
Based on its understanding, the agent issues a physical action:
Mouse actions:click(x, y), double_click(x, y), right_click(x, y), drag(x1, y1, x2, y2), scroll(direction, amount). Coordinates are in screen pixels.
Keyboard actions:type("text to enter"), key("Enter"), hotkey("Ctrl", "c"). The agent can type text into focused fields and use keyboard shortcuts.
Compound actions:click(x, y) followed by type("search query") to click a field and type into it. hotkey("Ctrl", "a") followed by type("replacement text") to select all and replace.
After each action, the agent takes another screenshot and compares the new state to what it expected. Did the click land on the right element? Did the form field accept the input? Did the page navigate correctly? Did an error dialog appear?
Verification is critical because computer use is inherently unreliable. A click might miss its target by a few pixels. A page might not have finished loading. A pop-up might have intercepted the click. The agent must detect these failures and recover -- retrying the action, scrolling to find the element, or adjusting its approach.
The agent identifies: a "From" field (pre-filled with the user's city), a "To" field (empty), date pickers, a passenger count selector, and a "Search" button. It needs to fill in the "To" field first.
The agent clicks the "To" field at coordinates (450, 230), types "Tokyo", waits for the autocomplete dropdown, identifies "Tokyo (NRT)" in the suggestions, and clicks it at (450, 310).
New screenshot shows "Tokyo (NRT)" in the "To" field. Success. The agent now clicks the departure date field, navigates the calendar to March 2026, clicks March 15, then clicks March 22 for the return date.
The agent clicks "Search." A new screenshot shows a loading spinner. The agent waits 3 seconds and takes another screenshot. Now it sees flight results: prices, airlines, times. It scans the results to find options under $1500 and identifies the cheapest: ANA nonstop at $780.
The agent can either report the result to the user ("The cheapest flight is ANA nonstop for $780, departing 10:35 AM") or, if authorized, click "Book" and proceed through the checkout flow -- entering passenger details, payment information, and confirming the booking.
What Do You Think?
A user asks the agent to 'book the cheapest flight to Tokyo on Kayak.' Should it use an API or computer use?
What Do You Think?
On OSWorld (the 2024 desktop-task benchmark with 369 real Linux tasks), what is the approximate state-of-the-art success rate as of 2026?
Computer use is the right answer. Kayak does not expose a public API for searching and booking flights. An API agent would have nowhere to send its function call. A computer use agent can navigate Kayak's website just like a human would -- slow and potentially brittle, but functional. This is exactly the use case computer use was designed for: automating tasks on software that only has a GUI. If a flight search API were available (like Google Flights API or Amadeus), that would be faster and more reliable. Always prefer APIs when they exist; use computer use when they do not.
Anthropic's Claude Computer Use shipped in 2024 with Claude Sonnet 3.5 (new) and was iterated into Computer Use 2.0 by 2025-2026 (Anthropic), now standard on Claude Sonnet 4.6 / 4.7 and Opus 4.7. The 2.0 generation made the screenshot-act loop materially more reliable -- better OCR-free element grounding, fewer hallucinated coordinates, native multimodal output, parallel tool calls, and sub-agent dispatch: the main agent can spawn smaller specialized computer-use agents for sub-tasks (e.g. a child agent that only handles the cookie-consent dialog while the parent waits). Three built-in tools the model can invoke:
computer -- Takes screenshots and performs mouse/keyboard actions. The model specifies coordinates for clicks, text for keyboard input, and can take screenshots to observe the current state. Actions include click, type, key, scroll, and screenshot.
text_editor -- Views and edits files on the local filesystem. The model can read file contents, create new files, and make targeted edits. This bridges computer use with file manipulation.
bash -- Executes shell commands and returns stdout/stderr. This gives the agent access to command-line tools, package managers, and system utilities.
The combination means a Claude Computer Use agent can interact with GUIs (via screenshots and mouse/keyboard), manipulate files (via the text editor), and run commands (via bash). It is the most complete "virtual human at a computer" implementation currently available.
OpenAI's Operator focuses on web-based tasks. It runs a browser instance and lets the agent navigate web pages, fill forms, and click elements. Operator is optimized for common workflows: shopping, booking, form filling, and data extraction from websites.
The key difference from Claude Computer Use: Operator works at the browser level rather than the screen level. It understands HTML DOM structure in addition to visual layout, which makes it more reliable for web tasks but unable to interact with desktop applications.
For developers who want full control, browser automation libraries like Playwright and Puppeteer provide programmatic access to web browsers. These are not AI-driven by default -- you write scripts that navigate pages and interact with elements. But they combine powerfully with LLMs:
LLM decides, Playwright executes. The LLM analyzes a screenshot (or DOM snapshot) and outputs a structured action: { "action": "click", "selector": "#search-button" }. Your code translates that into a Playwright command: await page.click('#search-button'). The result (new page state, errors, etc.) feeds back to the LLM for the next decision.
Advantages over raw computer use: Playwright can target elements by CSS selector, which is more reliable than pixel coordinates. It handles page loads, waits for elements, and manages network requests automatically. It is faster because it does not need to encode/decode screenshots for every action.
The hybrid approach is increasingly common in production. Use the DOM for reliable element targeting (clicking buttons, filling forms) and screenshots for visual understanding (reading charts, interpreting layouts, verifying visual state). This gives you the reliability of programmatic automation with the flexibility of visual comprehension.
The agent must map abstract concepts ("the search button") to specific pixel coordinates. This sounds easy but fails in many cases:
Ambiguous elements. A page might have three buttons labeled "Submit." Which one? The agent needs spatial reasoning: "the Submit button inside the payment form, not the one in the newsletter signup."
Non-text elements. An icon-only button (a magnifying glass for search, a hamburger menu, a gear for settings) has no text label. The agent must recognize the icon's meaning from its visual appearance.
Dynamic layouts. Responsive designs render differently at different screen sizes. An element at (400, 300) on a 1920x1080 screen might be at (200, 150) on a 1024x768 screen. Pop-ups, modals, and toast notifications shift elements around.
API tools have discrete, well-defined action spaces: search(query), book(flight_id). Computer use has a continuous action space: any (x, y) coordinate on the screen, any key combination, any text string. This vastly larger action space means more opportunities for error.
A misclick by 10 pixels might click the wrong button. Typing too fast might miss characters if the input field has not finished rendering. A scroll action might overshoot the target element. The agent must handle all of these gracefully.
Every action requires a screenshot-analyze-act cycle. Each screenshot is thousands of tokens (a 1920x1080 image can be 1500+ tokens). Each cycle involves a round trip to the LLM API. A task that takes a human 30 seconds might take a computer use agent 3-5 minutes due to this overhead.
Optimization strategies: reduce screenshot resolution (1024x768 is usually sufficient), crop to relevant regions of the screen, cache element positions for repeated interactions, and batch multiple quick actions between screenshots.
When things go wrong -- and they will -- the agent must recognize the failure and adapt:
Unexpected pop-ups. A cookie consent banner, a chat widget, or an ad overlay can appear at any time, blocking the target element. The agent must detect the obstruction and dismiss it before retrying.
Page load failures. The target page might time out, return a 500 error, or show a CAPTCHA. The agent needs retry logic and fallback strategies.
State corruption. The agent might accidentally navigate away from the target page, close a tab, or submit a form prematurely. It must detect these off-track states and navigate back.
pythonplayground.py · Pyodide
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
Tests · Implement the decision logic to navigate from homepage to booking confirmation. The agent should type a query, click search, select the cheapest result, and book it.
Let us break down exactly what happens in a single cycle of a computer use agent. Understanding this loop is essential because every computer use implementation -- Claude, Operator, Mariner -- follows the same fundamental pattern:
Step 1: Take Screenshot. The agent captures the current state of the screen as a PNG image. This screenshot is the agent's only source of truth -- it cannot inspect the DOM, read the application's memory, or access the window manager. It sees exactly what a human sitting at the desk would see. The image is typically captured at 1024x768 or 1280x720 resolution (lower than native to save tokens) and encoded as base64 for the LLM API.
Step 2: Send to Vision Model. The screenshot goes to a multimodal LLM (Claude Sonnet 5 / Opus 4.7, GPT-5, Gemini 2.5 / 3) along with the task context and action history. The model processes the image and identifies UI elements: buttons, text fields, dropdown menus, labels, checkboxes, navigation bars, error messages, loading indicators. It reads all visible text. It understands spatial relationships -- which label belongs to which input field, which button is inside which dialog, which menu item is highlighted.
Step 3: Decide and Execute Action. Based on its understanding of the screenshot and the current goal, the model outputs an action: click at coordinates (450, 320), type "Tokyo" into the focused field, press Enter, scroll down 3 units, or use a keyboard shortcut like Ctrl+C. The action is executed on the actual screen (or virtual machine), changing the application state. Critical detail: the model must output pixel coordinates for clicks, not element names. "Click the Search button" becomes "click at (720, 305)" -- and if those coordinates are off by 15 pixels, it might click the wrong thing.
Step 4: Take New Screenshot and Repeat. After the action executes, the agent waits briefly (for page loads, animations, or transitions to complete), then takes a fresh screenshot. It compares the new state to what it expected. Did the click register? Did the text appear in the right field? Did the page navigate correctly? If something went wrong (unexpected pop-up, misclick, loading error), the agent adjusts its plan and tries again. This verify-then-continue loop continues until the task is complete or the agent determines it is stuck.
Three major implementations have brought computer use to production, each with a different approach:
Claude Computer Use (Anthropic) provides the most complete "virtual human at a computer" experience. Claude gets three tools: computer (screenshots + mouse/keyboard), text_editor (file operations), and bash (terminal commands). It operates at the OS level, meaning it can interact with any application -- browsers, desktop apps, system settings, terminal. Claude Computer Use is the most general-purpose implementation: it can navigate a web form, edit a config file, and run a build command, all in the same task.
OpenAI Operator focuses specifically on web-based tasks. It runs a browser instance and understands both the visual layout (from screenshots) and the HTML DOM structure (from page inspection). This dual understanding makes it more reliable for web tasks than pure screenshot-based approaches -- it can target elements by CSS selector when visual grounding is ambiguous. The trade-off: Operator cannot interact with desktop applications, only web pages.
Google Mariner (Project Mariner) takes a browser-native approach powered by Gemini's multimodal capabilities. It runs as a Chrome extension, navigating web pages by understanding the visual layout and the underlying page structure. Mariner is designed for long-running web workflows: researching products across multiple sites, filling out multi-page forms, and extracting structured data from complex web pages. Its strength is persistence -- it can handle tasks that span dozens of page navigations without losing track of the overall goal.
All three follow the same four-step cycle. The differences are in scope (OS-level vs. browser-only), perception (pure vision vs. vision + DOM), and action reliability (pixel coordinates vs. CSS selectors vs. hybrid).
This is the most important decision you will make when building an agent that interacts with external systems:
Prefer APIs when they exist. APIs are faster (no screenshot encoding), more reliable (no visual grounding errors), cheaper (structured data instead of images), and more deterministic. If Google Flights has an API, use it. If your CRM has an API, use it. If the service offers webhooks, GraphQL, REST, or even a CLI -- any structured interface is better than computer use.
Use computer use when no API exists. Legacy enterprise systems, government portals, competitor websites, desktop applications, internal tools built by teams that left the company five years ago -- these only have GUIs. Computer use is your only option for automating interactions with them.
Use computer use for verification and testing. Even when APIs exist, computer use can verify that the user-facing experience works. An agent that logs into your web app, navigates the UI, and checks that data appears correctly is essentially an intelligent end-to-end test.
Hybrid approaches win. Use APIs for core data operations (fast, reliable) and computer use for the last-mile interactions that have no API. A travel agent might use the Amadeus API to search flights (fast, structured) but computer use to navigate the airline's booking checkout (no API for the purchase flow).
Computer use agents interact with software through screens, not APIs -- They take screenshots, understand the visual layout, and perform mouse/keyboard actions, enabling automation of any GUI-based application
The loop is screenshot-understand-act-verify -- Each cycle involves capturing the screen, analyzing elements, performing an action, and checking the result before proceeding
Visual grounding is the hard problem -- Mapping abstract intent ("click Search") to specific pixel coordinates on a cluttered, dynamic screen is where most failures occur
Always prefer APIs over computer use -- APIs are faster, cheaper, and more reliable; use computer use only when no structured interface exists, or for end-to-end verification
Production requires sandboxing, approval gates, and cost controls -- Computer use agents interact with real systems in unpredictable ways; never run them unsandboxed or without human oversight for high-stakes actions
What is the fundamental difference between an API agent and a computer use agent?
You now understand how agents can interact with any software by seeing and clicking -- the most general-purpose form of agent interaction. The tradeoffs are real: slower, more expensive, more error-prone than APIs, but capable of automating anything with a screen. Next, we will explore agents that specialize in the most meta task of all -- writing, testing, and debugging code.