ESP32 iPhone Agent: AI phone automation without jailbreak
Let an AI use a real iPhone with a $7 ESP32: AirPlay for eyes, USB HID for hands, no Mac or jailbreak required.
Key Facts
- Chip: ESP32-S3 (XIAO ESP32S3 recommended)
- Control: USB HID mouse/keyboard via AssistiveTouch
- Vision: AirPlay H.264 passthrough to iOS app
- AI: Any OpenAI-compatible vision model (cloud or local)
- Requirements: iOS 18+, USB-C iPhone, cellular data
About This Project
ESP32 iPhone Agent turns a Seeed XIAO ESP32S3 into an iPhone automation bridge. The board creates its own Wi-Fi hotspot, appears as an AirPlay receiver for screen mirroring, and acts as a USB HID mouse/keyboard. An iOS app decodes the mirrored stream, sends screenshots to any OpenAI-compatible vision model, and executes tap/swipe/type actions back through the USB connection. The result: an AI agent that can open apps, fill forms, and complete multi-step tasks on your phone—without Developer Mode, a Mac tether, or a jailbreak. The architecture closes the loop with pointer tracking, wiggle-based calibration, and automatic retry when the screen changes unexpectedly.
Source & Attribution
Illustration
Available as a complete kit
Thumb-sized ESP32-S3 with native USB, ~$7
Part details in our parts libraryHeads up: some Buy links are affiliate links. If you buy through one, MCU Atlas may earn a small commission at no extra cost to you. It never changes which parts we list. How this works
Bill of Materials
Parts list
2 parts · 1 linked
Heads up: some Buy links are affiliate links. If you buy through one, MCU Atlas may earn a small commission at no extra cost to you. It never changes which parts we list. How this works
1× iPhone with USB-C (iOS 18+)
Not included: iPhone, Vision API costs, Mac (for initial build only)
Related Projects
