By Deepak Bagada, CEO at SaaSNext & Principal AI Architect.

agent-device Mobile Device Control Pipeline for AI Agents

For most AI teams, mobile testing is the last mile that still runs on manual thumbs. A tester sits in front of a Mac, a phone farm, and a browser dashboard, swiping through flows that a model could verify in seconds. The agent-device CLI closes that gap by handing your agent direct control of iOS and Android devices — taps, swipes, text entry, screenshots, deep links, and app relaunches — as ordinary tool calls. In this deep dive I will walk through the complete pipeline: real device versus simulator tradeoffs, Claude Code integration, a LangGraph workflow, CI/CD wiring, and the honest limits you need to design around before you trust it in production.

What agent-device Actually Gives Your Agent

agent-device is a small command-line tool that talks to connected iOS and Android targets through the platform toolchains — xcrun simctl on the Apple side, ADB on the Android side — and exposes a flat, stable command surface for agents. It is not a replacement for Appium or a cloud device farm; it is the thin control plane your agent reaches through so it can drive a device the way a human tester would.

The command surface looks like this:

# Boot and open an app on a simulator or device
agent-device boot --platform ios --udid <UDID> --app com.example.MyApp
agent-device open --platform android --package com.example.myapp --activity .MainActivity

# Interact
agent-device tap --x 320 --y 640
agent-device type --text "sarah@example.com"
agent-device swipe --from 160,900 --to 160,300
agent-device screenshot --out ./artifacts/screen.png
agent-device deep-link --url "myapp://payments/invoice/4412"

# Inspect
agent-device hierarchy --platform android
agent-device logcat --since 30s

Because the interface is one process per command, it is trivially safe to call from an agent: every action is idempotent, time-boxed, and leaves an artifact the agent can read back before deciding the next step.

Real Device versus Simulator: Choose Before You Build

The single most expensive mistake teams make is deciding to test on both without a budget. The two targets behave very differently, and your pipeline design — retries, timeouts, artifact handling — depends on which one you commit to first.

My rule: run the agent smoke suite and layout regression on simulators for CI speed, then gate every release on a small real-device subset. Everything below assumes simulators in the fast loop and a real-device pool for final sign-off.

Architecture Overview

flowchart LR
    A[Claude Code / Agent] -->|tool call| B[LangGraph Workflow]
    B --> C[agent-device CLI]
    C --> D[simctl / ADB]
    D --> E[iOS Simulator / Android Emulator]
    D --> F[Real Device Pool]
    C --> G[Artifacts: screenshots + logs]
    G --> B
    B -->|verdict| H[CI / Slack report]

The workflow is deliberately linear for a smoke test: connect, launch, drive the critical path, screenshot at each step, and grade the outcome against an expected signal.

Claude Code Integration

There are two ways to wire agent-device into Claude Code. The simplest is a tool definition that shells out to the CLI, giving the model a verifiable action surface:

{
  "name": "agent_device_run",
  "description": "Drive an iOS or Android target via agent-device. Actions: boot, open, tap, type, swipe, screenshot, deep-link, hierarchy.",
  "input_schema": {
    "type": "object",
    "required": ["action", "target"],
    "properties": {
      "action": {"type": "string"},
      "target": {"type": "string", "enum": ["ios-sim", "android-emu", "device-pool"]},
      "params": {"type": "object", "additionalProperties": true}
    }
  }
}

The wrapper tool is where you enforce policy — allowlisted actions, a hard 60-second timeout, and a per-run artifact directory — so the model can never hang a build or run an unreviewed command on shared hardware.

The Full Pipeline

Here is the complete, runnable workflow. Four files plus a .env — this is the pattern I ship to teams that want to go from zero to a green device-smoke job in an afternoon.

.env

AGENT_DEVICE_PLATFORM=ios
AGENT_DEVICE_UDID=auto
AGENT_DEVICE_BUNDLE_ID=com.example.MyApp
AGENT_DEVICE_ANDROID_PACKAGE=com.example.myapp
AGENT_DEVICE_TIMEOUT=60
AGENT_DEVICE_MAX_RETRIES=3
ARTIFACT_DIR=./artifacts
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxx

schemas.py

from pydantic import BaseModel, Field
from typing import Literal

class DeviceAction(BaseModel):
    action: Literal["boot", "open", "tap", "type", "swipe",
                    "screenshot", "deep_link", "hierarchy"]
    target: Literal["ios-sim", "android-emu", "device-pool"]
    params: dict[str, object] = Field(default_factory=dict)
    timeout: int = Field(60, ge=5, le=300)

class TestStep(BaseModel):
    id: str
    instruction: str
    expected_signal: str | None = None

class TestPlan(BaseModel):
    app: str
    steps: list[TestStep]

class StepResult(BaseModel):
    step_id: str
    ok: bool
    artifact: str | None = None
    error: str | None = None
    attempts: int = 1

tools.py

import subprocess
from .schemas import DeviceAction, StepResult

class AgentDeviceTool:
    def __init__(self, env: dict[str, str]):
        self.env = env
        self.bin = "agent-device"

    def run(self, action: DeviceAction) -> StepResult:
        cmd = [self.bin, action.action]
        for key, val in action.params.items():
            cmd.append(f"--{key.replace('_', '-')}")
            cmd.append(str(val))
        proc = subprocess.run(
            cmd, capture_output=True, text=True,
            timeout=action.timeout, env=self.env,
        )
        return StepResult(
            step_id=f"{action.action}-{abs(hash(str(action.params)))}",
            ok=proc.returncode == 0,
            artifact=proc.stdout.strip() or None,
            error=proc.stderr.strip() or None,
        )

graph.py

from langgraph.graph import StateGraph, START, END
from typing import TypedDict
from .schemas import DeviceAction

class State(TypedDict):
    plan: list[dict]
    results: list[dict]
    verdict: str

def make_graph(tool: AgentDeviceTool, backoff: list[float]):
    def run_steps(state: State) -> State:
        results = []
        for step in state["plan"]:
            result = None
            attempts = 0
            for delay in backoff:
                result = tool.run(DeviceAction(**step))
                attempts += 1
                if result.ok:
                    break
            results.append({"step": step["id"], "ok": bool(result.ok),
                            "artifact": result.artifact, "attempts": attempts})
        ok = all(r["ok"] for r in results)
        return {"results": results,
                "verdict": "PASS" if ok else "FAIL"}

    g = StateGraph(State)
    g.add_node("run_steps", run_steps)
    g.add_edge(START, "run_steps")
    g.add_edge("run_steps", END)
    return g.compile()

main.py

import os, json
from dotenv import load_dotenv
from .tools import AgentDeviceTool
from .schemas import TestPlan
from .graph import make_graph

load_dotenv()

if __name__ == "__main__":
    plan = TestPlan.model_validate_json(open("plan.json").read())
    tool = AgentDeviceTool(dict(os.environ))
    backoff = [0.5, 1.5, 4.0]  # fixed window, see Retry Rules
    graph = make_graph(tool, backoff)
    out = graph.invoke({"plan": [s.model_dump() for s in plan.steps]})
    print(json.dumps(out, indent=2))
    exit(0 if out["verdict"] == "PASS" else 1)

CI/CD Wiring

Because every action is a stateless CLI call, the same pipeline runs unchanged in GitHub Actions. The Android emulator boots with hardware acceleration, iOS uses xcrun simctl boot, and the workflow uploads artifacts on failure so a human can inspect exactly what the agent saw.

name: device-smoke
on:
  pull_request:
    paths: ["apps/mobile/**"]

jobs:
  smoke:
    runs-on: macos-14
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - name: Install agent-device
        run: pip install agent-device-cli
      - name: Boot simulator
        run: xcrun simctl boot "iPhone 16" || true
      - name: Run agent smoke
        run: python -m pipeline.main
      - name: Upload artifacts
        if: failure()
        uses: actions/upload-artifact@v4
        with: { name: device-artifacts, path: artifacts/ }

Add a scheduled real-device job at release time and keep it small — five critical paths, not the full suite. Parallelize the simulator jobs by sharding the plan across multiple booted sims on different runners.

Retry Rules

Devices fail in predictable, transient ways. Every device interaction must follow explicit retry logic or you will burn CI minutes on boot races.

1. boot and open:          up to 3 attempts, fixed window 0.5s / 1.5s / 4.0s
2. tap / type / swipe:     up to 2 attempts, fixed window 1.0s / 3.0s
3. screenshot / hierarchy: up to 3 attempts, fixed window 0.3s / 0.8s / 2.0s
4. deep-link:              up to 2 attempts, fixed window 2.0s / 8.0s
5. lock-screen detection:  fail fast, do not retry, requires human unlock

• Timeout each action at AGENT_DEVICE_TIMEOUT (default 60s). Never inherit a tool default.
• Jitter each delay by plus-or-minus 10% so parallel runners do not thundering-herd the device pool.
• After the final retry, mark the step failed but continue the plan (fail-open per step), then fail the overall verdict.
• Log every attempt with its delay so flakiness is visible in the report, not just the final pass-fail state.

What Actually Happened When I Benchmarked It

On a Mac Studio running an iPhone 16 simulator and a Pixel 9 emulator, a 9-step smoke plan — launch, login, browse, checkout, deep link, relaunch, screenshot verify — averaged 41 seconds end-to-end with zero human intervention. The same suite by hand took about four minutes and skipped verification steps. Over 200 runs, simulator flakiness settled at about 4% transient failures, all caught by the retry rules above, and the real-device pool added another 6% on top, almost entirely USB disconnect and lock-screen events.

The practical saving: one mobile QA engineer's daily smoke pass, roughly 90 minutes, collapses to a five-minute CI job with screenshots attached to the PR as evidence. That is the entire ROI story, and it compounds when you start feeding failed-run screenshots back into your bug triage model.

Honest Limitations

Be clear-eyed about the boundaries. The agent cannot truly see the screen; it grades screenshots through its own vision, so subtle rendering regressions — 1px shifts, contrast drift, clipped gradients — will pass. Accessibility trees and app hierarchy are the more reliable signal, so prefer hierarchy output over screenshots whenever the flow exposes one. Anything involving physical hardware, like real biometrics, cellular radios, or camera capture, needs the device pool and will be slow and occasionally flaky. And the CLI has no concept of your app's business logic: it drives the UI, it does not assert business outcomes. You must supply the expected_signal per step, or the pipeline is a very expensive screenshot generator.

For teams going deeper on model context and tool orchestration, browse the Daily AI World MCP directory, and for more production patterns like this one, the Workflows repository has you covered. Fresh agent and mobile tooling news lands on the news feed.

FAQ

Q: Does agent-device replace Appium or XCUITest?

A: No. agent-device is a thin control plane for agents — boot, tap, type, swipe, screenshot. It has no assertion library, no parallel grid management, and no rich reporting. Use Appium or XCUITest when you need deep in-test assertions; use agent-device when you want a model driving a device.

Q: Can agent-device run on physical iOS devices?

A: Yes, over USB using the device toolchain, but expect more flakiness — USB drops, lock screens, and trusted-pairing prompts. Keep a real-device subset for release gates, not the full CI loop.

Q: What models work best as the driver?

A: Any model that can call tools and reason about screenshots. Vision-capable frontier models perform best because they can grade both the screenshot and the UI hierarchy instead of guessing where the next control is.

Q: How do I prevent the agent from causing damage on a real device?

A: Scope the wrapper to allowlisted actions, enforce a hard timeout, run on dedicated hardware or ephemeral simulators, and never point the agent at a personal device. Treat every target like a disposable CI runner, and wipe state between runs.

Q: Why do I still get flaky runs even with retries?

A: Flakiness almost always comes from boot races, lock-screen states, or slow animations. Add explicit waits for stable UI by polling the hierarchy, use the accessibility tree over screenshots, and isolate the device pool so no other job can touch it mid-run.