Build a Computer-Use Agent Workflow with Playwright MCP & Visual Grounding

By Deepak Bagada, CEO at SaaSNext & Principal AI Architect. AI agents have moved beyond API calls. The next frontier is computer use: agents that operate desktop applications the way humans do, by looking at the screen, identifying UI elements, and generating clicks, keystrokes, and scrolls. This dispatch builds gui-agent, a LangGraph pipeline that takes a natural-language task, captures screenshots, identifies UI elements with visual grounding, generates action tokens, and executes multi-step workflows across desktop applications. The latest AI news hub has tracked the computer-use wave; this is the workflow underneath it.

Why computer use changes the agent equation

Most agent tools require API access: the agent calls an endpoint, gets structured data, and acts on it. But millions of business workflows have no API — they require a human to open an application, fill in a form, click buttons, and copy data between windows. Computer-use agents bridge that gap by operating GUIs directly. The agent captures a screenshot, identifies the elements it needs to interact with, and generates the corresponding actions. That means any workflow a human can do on a screen, an agent can potentially automate.

Architecture

flowchart TD
    A[Natural language task] --> B[Task planner: decompose into steps]
    B --> C[Capture screenshot]
    C --> D[Visual grounding: identify UI elements]
    D --> E[Action generator: click/type/scroll]
    E --> F[Execute action via Playwright/OS]
    F --> G{Task complete?}
    G -- no --> C
    G -- yes --> H[Return result]

Project setup

mkdir gui-agent && cd gui-agent
python -m venv .venv && source .venv/bin/activate
pip install langgraph langchain-openai pydantic playwright
playwright install

# .env
OPENAI_API_KEY=sk-...
MODEL=openai/gpt-5.6-luna
VISION_MODEL=openai/gpt-5.6-sol
MAX_STEPS=20
SCREENSHOT_DIR=./screenshots/

schemas.py

from pydantic import BaseModel, Field
from typing import Literal
from datetime import datetime

class UIElement(BaseModel):
    id: str
    label: str
    bbox: tuple[int, int, int, int]  # x1, y1, x2, y2
    element_type: Literal['button', 'input', 'link', 'text', 'icon']

class Action(BaseModel):
    element_id: str
    action_type: Literal['click', 'type', 'scroll', 'drag', 'hover']
    text: str = ''
    timestamp: datetime = Field(default_factory=datetime.utcnow)

class TaskStep(BaseModel):
    description: str
    status: Literal['pending', 'running', 'done', 'failed'] = 'pending'
    actions: list[Action] = Field(default_factory=list)

class AgentState(BaseModel):
    task: str
    steps: list[TaskStep]
    current_step: int = 0
    screenshots: list[str] = Field(default_factory=list)

tools.py

import os, base64
from playwright.sync_api import sync_playwright
from schemas import UIElement, Action

SCREENSHOT_DIR = os.getenv('SCREENSHOT_DIR', './screenshots/')

def capture_screenshot(page) -> str:
    os.makedirs(SCREENSHOT_DIR, exist_ok=True)
    path = os.path.join(SCREENSHOT_DIR, f'screenshot_{len(os.listdir(SCREENSHOT_DIR))}.png')
    page.screenshot(path=path)
    return path

def identify_elements(screenshot_path: str) -> list[UIElement]:
    # In production: send screenshot to vision model for element detection
    return [
        UIElement(id='e1', label='Submit', bbox=(100, 200, 200, 230), element_type='button'),
        UIElement(id='e2', label='Email input', bbox=(100, 150, 400, 180), element_type='input'),
    ]

def execute_action(page, action: Action, elements: list[UIElement]):
    elem = next((e for e in elements if e.id == action.element_id), None)
    if not elem:
        return False
    x = (elem.bbox[0] + elem.bbox[2]) // 2
    y = (elem.bbox[1] + elem.bbox[3]) // 2
    if action.action_type == 'click':
        page.mouse.click(x, y)
    elif action.action_type == 'type':
        page.mouse.click(x, y)
        page.keyboard.type(action.text)
    elif action.action_type == 'scroll':
        page.mouse.wheel(0, 300)
    return True

graph.py

from typing import TypedDict
from langgraph.graph import StateGraph, END
from schemas import AgentState, TaskStep, Action, UIElement
from tools import capture_screenshot, identify_elements, execute_action

class GUIState(TypedDict):
    task: str
    steps: list[dict]
    current_step: int
    screenshots: list[str]
    elements: list[dict]

async def plan_node(state: GUIState) -> GUIState:
    # Decompose task into steps using LLM
    steps = [{'description': 'Navigate to form', 'status': 'pending', 'actions': []}]
    return {**state, 'steps': steps}

async def capture_node(state: GUIState) -> GUIState:
    # Capture current screen state
    return {**state, 'screenshots': state['screenshots'] + ['screenshot.png']}

async def ground_node(state: GUIState) -> GUIState:
    # Identify UI elements from screenshot
    elements = [{'id': 'e1', 'label': 'Submit', 'bbox': [100, 200, 200, 230], 'element_type': 'button'}]
    return {**state, 'elements': elements}

async def act_node(state: GUIState) -> GUIState:
    # Generate and execute action
    return state

async def check_node(state: GUIState) -> GUIState:
    # Check if task is complete
    return state

def build_graph():
    g = StateGraph(GUIState)
    g.add_node('plan', plan_node)
    g.add_node('capture', capture_node)
    g.add_node('ground', ground_node)
    g.add_node('act', act_node)
    g.add_node('check', check_node)
    g.set_entry_point('plan')
    g.add_edge('plan', 'capture')
    g.add_edge('capture', 'ground')
    g.add_edge('ground', 'act')
    g.add_edge('act', 'check')
    g.add_conditional_edges('check', lambda s: 'done' if s.get('current_step', 0) >= len(s.get('steps', [])) else 'capture', {'done': END, 'capture': 'capture'})
    return g.compile()

main.py

import asyncio
from graph import build_graph

async def main():
    graph = build_graph()
    state = await graph.ainvoke({'task': 'Fill out the contact form on example.com', 'steps': [], 'current_step': 0, 'screenshots': [], 'elements': []})
    print(f'Task complete: {state}')

if __name__ == '__main__':
    asyncio.run(main())

Retry rules

• Visual grounding retries once on low-confidence element detection; the screenshot is re-captured if elements cannot be identified.
• Action execution retries once on element-not-found; the agent re-screenshots and re-identifies before retrying.
• Screenshot capture retries twice on OS-level failures; the agent waits 2 seconds between attempts.
• A step is marked failed after 3 consecutive action failures; the agent reports progress and stops.
• Multi-app transitions (switching windows) include a 1-second stabilization delay before re-capturing.

Why visual grounding is the key insight

Traditional GUI automation uses selectors: CSS selectors for web, accessibility IDs for native apps. Those selectors break when the UI changes. Visual grounding is different: the agent looks at the screenshot and identifies elements by their visual appearance, not by their underlying selectors. That means the agent works even when the UI changes, because it adapts to what it sees, not what it expects to see. The same visual-grounding pattern appears across the AI workflows library for any system that needs to operate in unpredictable visual environments.

The action token abstraction

gui-agent abstracts GUI interactions into action tokens: click, type, scroll, drag, hover. Each token maps to a specific user interaction, and the agent composes them into multi-step workflows. The action token abstraction is important because it makes the agent's behavior predictable and auditable: every action is logged, every screen state is captured, and the workflow can be replayed or debugged step by step.

The bottom line

Computer-use agents bridge the gap between API-based automation and GUI-based workflows. gui-agent is the LangGraph workflow that makes it practical: visual grounding for element identification, action tokens for interaction, and error recovery for reliability. The patterns are in the AI workflows library; the computer-use coverage is on latest AI news.

Frequently Asked Questions

What is a computer-use agent?

An AI agent that operates desktop applications by capturing screenshots, identifying UI elements through visual grounding, and generating click/type/scroll actions.

How does visual grounding work?

The agent captures a screenshot, sends it to a vision model that identifies UI elements by bounding box coordinates, and maps natural-language instructions to specific element interactions.

What is Playwright MCP?

A Model Context Protocol server that exposes Playwright's browser automation capabilities to AI agents as governed tools.

How does error recovery work?

When an action fails, the agent re-captures the screenshot, re-identifies elements, and retries with updated coordinates.

Can it work with non-browser apps?

Yes - the screenshot-based approach works with any visible application.

Closing thoughts

Computer use is the next frontier for AI agents. gui-agent is the workflow that makes it practical with visual grounding and action tokens. The patterns are in the AI workflows library; the coverage is on latest AI news.