Developer Henry Yuan has a vision model tapping buttons, swiping screens and typing text on a stock iPhone, with no jailbreak and no developer account.
The open-source project detailed by Digital Trends on October 15, 2026, shows how a roughly $7 ESP32-S3 board acts as the AI agent’s hands. It presents itself to iOS as a standard USB mouse and keyboard. To the phone, it is just a familiar accessory.
Yuan posted the initial demo on X just days earlier. “A $7 ESP32 lets an AI agent control a stock iPhone,” he wrote. “No Mac, no jailbreak, no developer mode. Just a Linux machine.” The post spread quickly among developers chasing practical AI agents, and the GitHub repository gained thousands of stars within a week.
What sets it apart is the setup. Earlier automation tools for iPhones demanded Xcode, Apple developer certificates that expire every seven days, or full device jailbreaks. Those paths create friction. They limit who can experiment. Yuan’s approach sidesteps all of it. The board plugs straight into the charging port. USB-C models connect directly. Lightning phones need only a simple camera adapter. Power flows from the iPhone itself.
The Mechanics Behind the Control
The system splits vision from action. An iPhone mirrors its screen over AirPlay. Software on a connected computer or, in newer versions, directly on the device captures frames. Those images feed a vision-language model. The model analyzes the current display. It decides the next move. Tap here. Swipe there. Type this message.
Commands travel to the ESP32 board. The tiny microcontroller translates them into HID inputs. With Apple’s AssistiveTouch accessibility feature enabled, a visible pointer appears on screen. Mouse movement becomes finger movement. Clicks become taps. Drags become swipes. The loop closes when the model receives the next screenshot and observes the result. Visual feedback drives the entire process. No reliance on accessibility trees or app-specific APIs. It works with whatever pixels appear. Even games.
Performance sits in an interesting middle ground. One local setup using Alibaba’s Qwen model achieved actions roughly every few seconds. That is slow enough to look deliberate. In the demonstrated cases the model runs locally, so everything stays on the local network with no cloud calls.
Setup involves flashing firmware to the Seeed Studio XIAO ESP32S3 board. A Python server handles coordination. Developers point it at any OpenAI-compatible vision endpoint. Cloud services work. Local models work. The project ships with a browser interface for monitoring. Users watch the agent at work in real time.
Limitations exist. The agent cannot see iOS security screens such as passcode entry or Apple Pay, which iOS keeps out of the mirrored feed, and locking the phone stops mirroring altogether. The system also depends on AirPlay mirroring, which consumes bandwidth and introduces minor latency. Frame rates top out around five per second in tested configurations. Yet for many tasks, those constraints prove acceptable.
Yuan continued refining the project after the initial release. Later updates removed the need for a separate computer during operation, though initial setup still requires a Mac with Xcode. The ESP32 now receives the AirPlay stream directly in some configurations. An accompanying iPhone app decodes the stream, sends screenshots to the vision model, and runs the command logic. The board and phone work together. Dynamic Island shows status updates while the agent runs in the background. The entire package stays lightweight.
This approach arrives at a moment when AI agents command intense industry focus. Companies race to build systems that act across apps, complete multi-step tasks, and operate with minimal human supervision. Most demonstrations rely on simulators or web interfaces. Real phones introduce complications, from different screen sizes to app updates that break selectors and security sandboxes.
The ESP32 solution treats the phone as a black box. It sees what a user sees. It interacts exactly as a user would. That universality matters. Banking apps without public APIs. Messaging platforms with complex flows. Games that resist traditional automation. All become fair game.
Related efforts have emerged. Projects like iphone-use connect agents through USB and Apple’s testing frameworks, but they still require a Mac and developer setup. Commercial tools such as TapKit and Sightless offer agent control through different paths, some voice-driven or Mac-mediated. None match the minimal hardware footprint of Yuan’s board.
Meanwhile, smartphone makers push their own agentic visions. STEPX released its Neo device with dedicated hardware for always-on sensing. OPPO partnered to embed agents in upcoming flagships. Qualcomm outlined roadmaps for proactive on-device AI across more price points. The hardware hack from one developer suddenly looks less like a curiosity and more like an early signal of where mobile computing heads.
Developers have already begun forking the code. The MIT license invites exactly this sort of experimentation. Barriers to entry remain low. A board costs less than lunch in many cities.
The dual-use potential is obvious. The same mechanism that lets an AI order groceries could, in theory, enable unauthorized automation if paired with malicious instructions. Yet the requirement for physical access and user-enabled AssistiveTouch adds meaningful friction. It’s not a remote exploit. It’s an accessory that does what the user allows.
Apple has not commented publicly on the project. The company continues its own Apple Intelligence rollout, including regulatory approvals in markets like China for additional models. Those efforts focus on integrated, on-device experiences. Yuan’s work operates outside that walled garden. It gives tinkerers and researchers a practical way to test agent behaviors on real hardware today.
The broader shift feels unmistakable. Smartphones long served as portals to apps and services. Now they become platforms that intelligent systems can drive directly. A $7 board won’t replace sophisticated robotics or enterprise automation suites. But it demolishes the cost and complexity barriers that kept many experiments confined to virtual environments.
Watch the demos. In one, the agent opens Clock, types a time into the wheel, ticks the days and saves an alarm; in another, it finds a whiteboard app and draws a smiley face. It makes mistakes sometimes. It recovers by looking again at the screen. The process looks less like scripted macros and more like guided exploration. That’s the point. Real agents need to handle uncertainty. They need to adapt when interfaces change.
Yuan’s roadmap lists what comes next: support for landscape apps, non-ASCII text input such as Chinese, and faster steps that fit several actions into each model call.
$7 ESP32 Board Hands AI Real Control Over Stock iPhones first appeared on Web and IT News.