developer tools
The Agent Stopped Looking for the Button
ASIL exposes application state as structured JSON and replaces brittle pointer actions with code-executable semantic operations.
Summary
ASIL exposes application state as structured JSON and replaces brittle pointer actions with code-executable semantic operations.
Across 15 applications and 380 tasks, the structured interface exceeded 80 percent strict success with closed models while using fewer than five actions per task. Screenshot-and-click baselines remained far lower under the reported budgets, though ASIL only matched draw.io's native MCP-style content contract. Small-model fine-tuning also improved, suggesting the interface can serve as a training substrate.
Why it matters
ASIL exposes application state as structured JSON and replaces brittle pointer actions with code-executable semantic operations.
Limits and context
- Screenshot-and-click baselines remained far lower under the reported budgets, though ASIL only matched draw.io's native MCP-style content contract.
Key claims
ASIL exposes application state as structured JSON and replaces brittle pointer actions with code-executable semantic operations.
Qualification: Screenshot-and-click baselines remained far lower under the reported budgets, though ASIL only matched draw.io's native MCP-style content contract.
Evidence: source-2026-08-28-015
Sources
- arXiv preprint 2608.26991arXiv · primary research
Corrections
No corrections have been recorded for this story.