Benchmark
iOSWorld results
How Phone Use scores on the official 133-task iOSWorld benchmark.
Headline results
Phone Use was evaluated on the official iOSWorld suite: 133 tasks across 26 SwiftUI apps, scored by the benchmark's unmodified GPT judge. A task passes only when every rubric criterion is satisfied. The run uses Claude Opus 4.6 — the same model as the published best — driving the Phone Use CLI.
| Category | Phone Use | Published best (vision + XML) |
|---|---|---|
| Single-app | 25/27 · 92.6% | 81.5% |
| Memory | 25/46 · 54.3% | 54.3% |
| Multi-app | 20/60 · 33.3% | 36.7% |
| Overall | 70/133 · 52.6% | 51.9% |
Methodology
- Official tasks.json, app suite, and state-reset protocol; artifacts follow the benchmark schema.
- Unmodified judge (GPT, strict all-criteria pass), run with the benchmark's own judge_trajectories.py.
- 50-step budget matching the reference semantics: one step = one model generation, executing any number of device actions.
- Best-of-N across attempts per task, disclosed; every attempt's artifacts are preserved.
- Agent surface: Phone Use CLI over the accessibility tree, with selective screenshot reads (0–8 per task) instead of every-step vision.
Tiered perception
The agent reads the compressed accessibility tree by default and looks at pixels only at decision points where text is insufficient — proof screens, pickers, visual layouts. That keeps the speed and cost of a text agent with the ceiling of a vision agent: measured $2.53 median per task and ~10 minutes median end-to-end, with simple tasks completing in about a minute for roughly $0.25.