AI Phone Control Capabilities: L1 Atomic, L2 Workflow, L3 AI

What each of the three layers does

To have an AI control a real phone in a stable way, three layers have to work together. Each layer answers a different problem.

Layer	Name	What it handles	Outside its job
L1 atomic capabilities	On-device Agent	Complete one checkable basic operation on a real phone (tap, swipe, input, and so on)	Understanding the user’s natural language
L2 workflows	Script / template	Turn a job you actually need into a stable step graph (for example, “publish one Douyin short video”)	Replacing IAM or the console
L3 conversation orchestration	Cloud/Eino	Pick a workflow, fill slots, create or query a task, explain a failure	Tapping the UI directly inside the chat
Task state machine	df_task	Dispatch, cancel, logs, terminal state	Running the UI automation

If the atomic capabilities are unsteady, the two layers above are unsteady too. These calls are supported now:

• launch_app: launch a specified app. The success receipt requires that the target app is already in the foreground.
• tap: tap a coordinate or a selector, with optional confirmation that the UI changed.
• swipe: a swipe gesture, with a receipt after the gesture finishes.
• input: enter text. Success means the target field matches what was expected, or the text can be read.
• wait_ui: wait until copy or a control appears. On timeout it reports an explicit error.
• album_import / pick_media: the media is ready, and it can be selected on the publish page.
• screenshot (optional): for debugging or for replanning one part. It is not written to disk by default (the product rule is that screenshots are not stored).

Every call has a success or failure receipt. The basis is three steps, each leaving its own evidence: execute, assert, and report back. When a step fails, you can see which step it was. The flow stays traceable.

A workflow stitches these calls into fixed steps. One L2 workflow has four parts.

1. workflow_id: a unique id, for example publish_short_video.
2. Slot schema: which parameters the user fills in, such as copy, video path, and target app.
3. steps or an FSM (finite state machine): the execution graph. Each step calls an atomic capability and carries an assertion.
4. Publish state and scope. Scope has two kinds. system is built in. tenant is defined by the tenant. status has to be published before it is visible outside.

The built-in example right now is “short video publish.” It has four parameters: account context, copy, how the media is referenced, and the target app. One target app is enough to start. Douyin, Xiaohongshu, or WeChat Channels. After it is created you can run it immediately, or bind one specified phone that is online. Scheduled publishing and multi-device batch, which are P1 items, are still on the schedule.

L3 takes natural language. You say one sentence, and the system splits it and dispatches it. The path looks like this:

1. The user states an intent, for example “at 9 tomorrow, use this phone to post one Douyin video, and the copy is xxx.”
2. The Eino Agent plans: it parses the intent, recognizes which workflow to call, and pulls out the slot parameters.
3. It produces a step graph and creates a task. The workflow expands into a df_task, and the state becomes running.
4. The task goes to the Agent client over WebSocket. The Agent runs the steps and sends progress and results back.
5. You read the result in the console. The state goes from pending to running, then to succeeded or failed. A failure always has an error message you can read.

By default, L3 stays at intent and dispatch. The concrete steps are already set in the L2 workflow. When the flow changes, edit the workflow JSON. The model stays as it is.

How MCP, sessions, and workflows fit together

The product is split into three pieces. They share the same workflows and the same infrastructure.

Module	What it is	What it does
MCP Server	An entry point for outside AIs such as WorkBuddy and Claude	List devices, run workflows, query tasks. Tapping the phone sits outside this entry
Agent session	The built-in chat bench in the console	Operators chat across several turns, pick a workflow, fill parameters, and create a task
Workflows	A versioned library of automation templates	Slot schema + steps/FSM + publish state; the same set inside and outside

The three pieces use one path: ws(task_dispatch) → agent client → real phone. Whether the request comes in through MCP or through an Agent session in the console, the execution path is the same.

Screenshots are not stored

Screenshots are not written to disk and not written to a database anywhere on the path. Device information and task records stay on your own computer. Studios and companies that care about sensitive data use this: if someone else gets the machine, they still cannot reconstruct what was on the screen.

On top of that there is multi-tenant separation and RBAC. Different business units see only their own devices. Operators have three permissions: devices, tasks, and AI. A read-only role cannot create tasks.

For how to start, see installation and deployment.