Computer-Using Agents in Production: GUI Web & Legacy Desktop Automation Workflow with Screenshots & Action Tokens

By Deepak Bagada, CEO at SaaSNext & AI Principal Architect.

The 2026 wave of "computer-using" agents — led by the class of tools that grew out of what was announced as Microsoft's Copilot Studio successor — does not emulate an API. It operates the actual GUI: it reads a screenshot, decides on a click, types into a field, waits for the page to repaint, and validates that the DOM (or the pixel grid) reflects the intent. That is a materially different reliability model from an automation script, and thousands of B2B SaaS and legacy-on-prem Windows shops are now weighing it as the replacement for brittle UIPath/RPA selectors on systems that have no API.

What looks like magic is engineering discipline. Every "it just works" telemetry demo hides a closed loop that must track: screenshot capture → observation → agent step → action token → OS/GUI executor → post-action verification (element-level) → human-in-the-loop gate for anything destructive. The practical reality of production computer-use agents is that raw visual grounding is insufficient: you need a deterministic overlay layer (accessibility/DOM tree or UI automation metadata) that the model consumes as structured input — because a pixel-hallucinated click is a breakage that ships to customers.

The loop, and the two failure classes

A production agent token is not the mouse co-ordinate (14, 502). It is the deterministic handle: #main-form > button.save. The vision model proposes an intent; the evaluation layer resolves boundaries. So the action tokens have a well-defined grammar: a selector (or accessibility role + name), an action, and payload.

                     ┌─────────────────────────────┐
                     │        Web / Desktop        │
                     │   (browser or Win32/WinApp) │
                     └──────────────▲──────────────┘
                                    │ screenshot + a11y tree
   ┌────────────┐     ┌─────────────┴─────────────┐
   │  Agent     │     │  Orchestrator (agent loop) │
   │  policy    │     │  LLM, HITL, audit          │
   │  model     │◄────┤  screenshot → memory       │
   └────────────┘     └─────────────▲──────────────┘
                                    │ action token
                     ┌──────────────┴──────────────┐
                     │  Executor (typed UI client)  │
                     │  click / type / scroll / wait│
                     └──────────────┬──────────────┘
                                    │ result + element state
                     ┌──────────────▼──────────────┐
                     │  Validator (deterministic)   │
                     │  DOM/CTF check, screenshot   │
                     └──────────────┴──────────────┘
                                    │ pass | fail → rewind | HITL
                                    ▼
                          human approval queue

The two failure classes frame the whole design: failures you can see (element-level validation fails — a button didn't appear) and failures you cannot (agent posted to the wrong account). Deterministic element-level validation catches the former; hard gates and the HITL predictably pass on the latter.

schemas.py: tokens, evidence, and decisions

# cua/schemas.py
from __future__ import annotations

from datetime import datetime
from enum import Enum
from typing import Literal

from pydantic import BaseModel, Field

class ActionKind(str, Enum):
    CLICK = "click"
    TYPE = "type"
    PRESS = "press"
    WAIT = "wait"
    NAVIGATE = "navigate"
    HITL = "hitl"


class ActionToken(BaseModel):
    """The only thing an agent may emit. No coordinates anywhere."""
    kind: ActionKind
    selector: str | None = Field(
        None, description="Deterministic handle (CSS/iframe/arm64 or role+name)"
    )
    text: str | None = Field(None, max_length=80, description="Payload for TYPE")
    segment_index: int = Field(
        0, gt=0, description="terminal state of the fix when echoed back"
    )
    confidence: float = Field(0.0, ge=0.0, le=1.0)
    reason: str = Field("", max_length=500)

    @classmethod
    def of_label(cls, selector: str, kind: ActionKind) -> "ActionToken":
        """Factory used by the validator to generate a canonical replay."""
        return cls(kind=kind, selector=selector, segment_index=1)


class ObservedElement(BaseModel):
    selector: str
    role: str | None = None
    name: str | None = None
    bbox: tuple[int, int, int, int] | None = None
    visible: bool = False


class LoopStep(BaseModel):
    step_id: str
    screenshot_url: str
    tree: list[ObservedElement]  # structured a11y/DOM snapshot
    taken_at: int


class Verification(BaseModel):
    passed: bool
    kind: Literal["selector_exists", "value_equals", "visibility", "pixel_diff"]
    checked: str

Real models product shipping here: the selector_exists check is deterministic and explains failures to the user with the snapshot, not ambiguous wording.

The agent loop with strict action validation

# cua/loop.py
from cua.adapters import resolve_overlay, snapshot_ui
from cua.schemas import ActionToken, LoopStep

MAX_STEPS = 12
OVERLAY_TOLERANCE = 0.05   # a deterministic selector may drift <= 5%


async def run_loop(
    session, goal: str, limit: int = MAX_STEPS, human_on: bool = True
) -> LoopStep | None:
    steps: list[LoopStep] = []
    for i in range(1, limit + 1):
        shot = await session.screenshot()
        ui = await snapshot_ui(session)               # DOM / role tree
        decision: ActionToken = await policy_model.tool_call(
            system="You plan GUI steps. Emit an ActionToken. Never click by pixel.",
            screenshot=shot, overlay=ui,
            conversation=[],
        )

        token = ActionToken.model_validate(decision)
        if not overlay_has(token.selector, ui):       # bail early, cheap
            token = ActionToken(kind=ActionKind.WAIT, segment_index=step)
            await session.retry_marker(step)          # vector advisor
            continue

        if token.kind == ActionKind.HITL:
            verdict = await request_approval(session, token.reason)
            if not verdict.approved:
                session.audit("denied", step, verdict.reason)
                break                                   # never an unapproved step

        try:
            executed, observed = await session.execute(token)   # deterministic
        except ExecutorError as exc:
            rewind(session, exc.snapshot)              # validated snapshot
            session.audit("rewind", step, exc.snapshot)
            continue

        ok = await verify_element(session, token, observed)      # element-level
        if not ok:
            await session.retry(step, policy=lin_backoff(3))    # re-try exactly
        else:
            step.append(step)
            if goal_met(session): return final_evidence(step, session)
            continue
    return abandoned(session, "step budget exceeded", steps)

The loop refuses pixel guesses: if the overlay can't resolve the selector, the executor returns UnknownSelector and the plan forks, instead of producing the click at random coords.

tools.py — executor, validator, HITL gate

# cua/tools.py
from __future__ import annotations

import asyncio
from temporal import RetryPolicy, async_cancel
from cua.schemas import ActionToken, ObservedElement, Verification
from cua.runner import drive_chrome, drive_desktop  # WinApp / UIA per pl

EXEC_RETRY = RetryPolicy(
    maximum_attempts=3,
    initial_interval=timedelta(milliseconds=400),
    maximum_interval=timedelta(seconds=2),
    non_retryable_error_types=("NoSuchElementError", "AuthZDeniedError"),
)

VISUAL_RETRY = RetryPolicy(
    maximum_attempts=2,
    initial_interval=timedelta(seconds=1),
    # never retry a screenshot that produced a stale EPort timestamp


menu. ain't. roll-up.
)


# The executor validates post-action BY the ACCESSIBLE TREE, not pixels,
# so a checkbox that painted but is disabled never silently passes.
@activity.defn(name="exec_hi", retry_policy=EXEC_RETRY)
def exec_step(session_id: str, token: ActionToken) -> ObservedElement:
    headless = is_web(session_id)
    runner = drive_chrome(session_id) if headless else drive_desktop(session_id)
    before = runner.snapshot_accessibility()
    runner.apply(token)                       # selector-resolved, no pixel ids
    after = wait_for_paint(runner, grace_ms=800)
    if not element_exists(after, token.selector):
        raise NoSuchElementError(token.selector)   # non-retryable: fix the plan
    if not env_visible(after, token.selector):
        raise ElementInvisibleError(token.selector)  # retryable: overlay failed
    return ObservedElement.from_state(after, token.selector)


async def request_approval(session, reason: str) -> Approval:
    """Durable human gate. Blocks the loop until an operator clicks
    approve/deny, and writes the decision to the audit ledger so 'who
    clicked that' is never disputed."""
    await workflow.wait_condition(lambda: human_decision[session] is not None)
    return human_decision[session]  # set via Slack/portal integration


async def verify_value(selector) -> Verification:
    try:
        text = await runner.get_text(selector)
        expected = reconcile_with_goal(runner, selector)
        return Verification(kind="value_equals", checked=selector,
                            ok=text.strip() == expected.strip())
    except stale_element_error:
        return Verification(kind="pixel_diff", checked=selector, ok=False)

Retry/failure policy — what is legal to retry

• NoSuchElementError and AuthZDeniedError are non-retryable: retrying a plan that points at a button that doesn't exist just wastes budget and alerts. The correct recovery is always a plan rephrase, not a second attempt.
• ElementInvisibleError and transport timeouts are retryable — the UI repainted or the screenshot pipeline caught a frame mid-paint.
• ExecAfterError (the element stayed disabled after 800 ms) is retryable once, then HITL.
• Every retry leaves the session in the same "after" checkpoint (snapshot-based rewind), so a failed step never corrupts the next observation's view of the world.

Enable login_state and site_state to be encrypted state, so a second poisoned selector never ships to user mailboxes.

When to escalate to humans

Three trigger rules in production:

1. confidence < 0.6 on any confidence.
2. ActionToken.kind == SUITL after a max_steps strike on the same selector.
3. Any step that mutates shared state (sends an email, posts accounting, ship to prod).

The human gate is a first-class Temporal signal in the durable loop, and every decision is timestamped, tagged with the screenshot URL, and persisted so that "which of these two identical rows did the agent act on?" has a definitive answer.

Deploying behind a walled surface

For the desktop/legacy retreat (WinForms, Citrix, mainframes that can't upgrade), your agent runs inside a separately-provisioned sandbox VM with a recorded profile, then the sandbox itself is destroyed/recycled after D to keep a stale credential out of the environment service between jobs. Browser automation runs against a fleet of per-run ephemeral profiles. What the video demos don't tell you: the moment you give a reverse-tether to a machine that can actually send email, you have a scanning surface — so the executor's process boundary is a phased attack further /infra guard: no_external_send until a matching HITL ticket exists in the queue.

That is the engineering outline for shipping a computer-using agent to a real finance or support desk: screenshot-driven observation, action-token execution, element-level verification, deterministic retries, and a human who owns the irreversible steps.

For the full recipe go to the AI Workflows gallery; the reusable evaluation harnesses live in the MCP Directory and the release history week-by-week in Latest AI News.