How should an LLM see a computer
Blog post from Greptile
An evaluation of seven AI models across 172 computer-use tasks compared screenshot-only input with accessibility trees and element-ID clicks, finding that interface descriptions and precise clicks substantially improved results for weaker models but made little statistically significant difference for the strongest ones. For example, GPT-5.2 rose from 15.8% success with screenshots alone to 52.8% with a tree and ID clicks, while GPT-6.1 Sol remained near 82% across setups. Accessibility trees, originally developed to help people who are blind or have low vision use graphical interfaces, can tell agents what controls are present and where they are, reducing the difficulty of locating small targets on screen. However, some applications expose little or no accessibility information, and screenshots reveal visual problems such as overflowing text, so the approach described for TREX combines screenshots with ID-based clicks when available and coordinate clicks when necessary. The authors plan to repeat the tests as models improve and adjust TREX’s screen-reading strategy accordingly.
| Trend | Post Mentions | Total Month Mentions | Posts | Companies | MoM |
|---|---|---|---|---|---|
| GPT-6.1 Sol | 6 | 131 | 32 | 22 | -48% |
| LLM | 1 | 3,355 | 742 | 191 | -49% |
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.