Traditional automation scripts rely on a fixed element ID or a fixed coordinate. One app redesign and the whole line broke. The first step in AI phone control is letting the model see what the screen looks like now.
Screen understanding
Two approaches are common now:
| Approach | Pros | Cons |
|---|---|---|
| OCR plus UI node scraping | Fast and cheap, suited to structured screens | Has to handle multiple languages, font scaling, and dynamic content |
| A VLM (vision-language model) looking at the image directly | Generalises well, and can read a complex layout | Higher latency, and it needs GPU acceleration or cloud inference |
iEasyRun uses a VLM plus OCR. The VLM handles the overall layout and extracts the large fields. OCR helps with small text and with filling forms. The two layers of output are aligned and checked, and only then sent on to the next planning step.
Intent planning
You say “list these products”, and the AI has to run a chain of reasoning:
- Identify the intent type. This is an e-commerce listing task, mapped to
workflow_id="list_products"if a matching template exists. - Extract the slot parameters. Price range, product category, and target platform are the blanks the workflow needs filled.
- Generate the step graph. Expand the task into an ordered sequence of steps, and mark each step with the atomic action to call and the assertion condition.
The hard part is fault tolerance. When step 3 succeeds and step 5 fails, the system locates the cause of the failure at step 5. When needed, it rolls back to the most recent successful state and retries. Each step keeps evidence: the execution, the assertion, and the result sent back.
Carrying out the actions
The execution layer has to do these three things:
- Atomicity. Each operation finishes on its own, without interfering with the others. A tap, for example, does not change the result of an input.
- Assertion. After each action, check that the screen feedback is what was expected. After a button tap, is the dialog the one you expected? After text is entered, is the cursor in the right place?
- Status reporting. Progress and results are pushed to the console, so a person can watch.
The agent on the phone does this step. The native client on each phone, an iOS Swift app or an Android Java or Flutter app, receives task_dispatch messages from the farm over WebSocket, runs them on the device, and sends the result back. The process does not go through a third-party service. The data stays on your computer.
Protocol
What connects the farm control plane and the agent is a lightweight protocol based on WebSocket and JSON. The main message types:
| Message type | Direction | Purpose |
|---|---|---|
| register | Agent → Farm | Device registration: reports ID, name, platform (ios), and capability tags |
| heartbeat | Agent ↔ Farm | Periodic keepalive; a timeout marks the device offline |
| task_dispatch | Farm → Agent | Sends a step graph or a workflow reference |
| task_progress/task_result | Agent → Farm | Uploads progress and the final result |
| task_cancel | Farm → Agent | Cancels a task that is running |
The design aims at simple, idempotent, and extensible. Each message carries a protocolVersion field for version compatibility. task_dispatch can be sent more than once, and a later send replaces one that has not finished. agentKind leaves an extension slot, so Android and HarmonyOS can be added later.
When a step fails, the system has to be able to find its way back on its own. Connecting a phone and configuring MCP is covered in AI integration and installation.
Go Deeper
Want to See It In Action?
Install the system, connect one phone, and run your first device.