The Grounding Problem: How 19-Year-Old Eshaan Gulati Taught AI Where to Click

His computer-use research addressed a technical failure that kept capable models from turning visual recognition into accurate action.

An artificial-intelligence model could look at a software dashboard, identify the correct button for exporting a report, and still click somewhere else. Eshaan Gulati encountered that failure while researching computer-use agents at Purdue University in 2024. He responded by building a framework that removed coordinate estimation from the task.

The problem exposed a gap between recognition and localization. A model might understand which interface element mattered while remaining unable to identify its exact position on a real screen. For computer-use agents, that gap separates a system that can explain an action from one capable of performing it.

“The model knew which button it needed,” Mr. Gulati said. “The failure came when it had to translate that understanding into a precise location.”

Mr. Gulati was 19 and in his freshman year studying computer engineering at Purdue when he began conducting machine-learning research on computer-use agents in 2024. At the same time, he was tutoring students in senior-level C courses and assembling a team to work on the grounding problem.

One of them, Nikhil Reddy Ramolla, had placed 36th nationally among the more than 1.1 million candidates who entered India’s IIT-JEE examination process in 2017 before completing his degree at IIT Bombay. Mr. Ramolla had also worked as a high-frequency trading developer before spending roughly six months building the grounding system with Mr. Gulati.

Arbitrary desktop software offered no readable structure the model could use, making vision necessary. The system had to interpret what appeared on the screen and determine where to act.

The technical difficulty came partly from the way models produced coordinates. The model had to infer the relationship between the compressed image it processed and the real pixels on the screen. That mapping could break on screen sizes outside its training data.

Software interfaces also presented a different visual problem from ordinary photographs. Screens can be dense, with small buttons, icons, menu items, and fields that resemble one another. Models that performed well at locating objects in ordinary photographs still struggled when asked to identify precise targets inside software.

ScreenSpot was developed specifically to test whether models could locate elements in graphical interfaces. Results on the benchmark showed that strong performance in ordinary image recognition did not automatically carry over to software screens.

Mr. Gulati changed the task presented to the model. A computer-vision system cropped every interactive element from the interface. Those individual elements were then re-rendered as a clean, numbered grid.

In Mr. Gulati’s account, the model selected the number corresponding to the element it wanted to use. It no longer had to examine the full screen, understand the target, and calculate coordinates that matched the original display.

“I removed the geometry problem instead of trying to solve it through better coordinate prediction,” Mr. Gulati said. “Once the elements were isolated and numbered, localization became selection.”

His framework differed from set-of-mark prompting, a method that labels possible targets directly on the original screenshot. Those labels can help identify clickable elements, but the rest of the interface remains visible. Menus, text, graphics, and unrelated controls still compete for the model’s attention.

Mr. Gulati’s approach removed the surrounding screen from the final selection task. The model saw the available elements separately, reducing visual noise while eliminating direct coordinate estimation.

“The numbered grid changed what the model had to decide,” Mr. Gulati said. “It chose an element from a defined set rather than trying to map its understanding back onto the original screen.”

The contribution addressed a specific technical bottleneck with consequences across computer-use AI. A model understands a user’s request and correctly identifies the necessary action. That understanding has limited value when the agent cannot interact with the right part of the interface.

Later, in summer 2025, Mr. Gulati compared his vision-only framework with Claude Sonnet 4 using the original OSWorld benchmark. Anthropic cites 42.2 percent as the published record on the original OSWorld that Sonnet 4.5 later surpassed. In evaluations Mr. Gulati ran on the original OSWorld, his framework outperformed Claude Sonnet 4 by 24 percent.

The visual-grounding framework became the foundation for Mr. Gulati’s later work on browser agents.

Mr. Gulati eventually founded Argide, a browser agent for in-app customer support. The product acts inside software rather than stopping at written guidance. When a user becomes stuck, the agent can complete the requested task within the interface.

Existing support tools, among them Intercom and Sierra, answer at the text layer. The user gets a broad answer and still has to do the clicks.

“Current tools treat support as a conversation,” Mr. Gulati said. “That ultimately still forces the user to constantly think and take action themselves.”

Mr. Gulati’s research had tackled the same problem of an AI agent operating a computer, but it targeted arbitrary desktop software because that was the hardest case: no exposed structure, no fallback, vision or nothing. The software people actually get stuck in is mostly SaaS with configuration-heavy dashboards, and it runs in a browser.

Argide is built on the grounding method from the research, but without the vision. The numbered grid existed because arbitrary desktop applications expose no reliable accessibility tree. Many draw custom controls straight to a framebuffer, leaving pixels as the only queryable surface, so elements had to be detected before they could be enumerated.

A browser maintains the DOM: a live tree of nodes carrying each interactive element’s tag, role, text, and bounding box. Argide queries that tree instead of running a detector, so the interface reaches the model as text rather than as an image. Mr. Gulati said that removing the vision tokens made the system roughly four times faster, and that Argide relies on the DOM roughly 95 percent of the time.

What the research established survives the change. The model names an element and the system translates that name into a click, the same pattern DOM agents use when they address an element by index and keep the index-to-node mapping outside the model. Neither system asks it for a coordinate.

Argide brought Mr. Gulati’s research interest into a paid product used by SaaS companies, including MathGPT and Hiphip.