Om Buddhdev: AI’s Hardest Tests Begin Where Benchmarks End

The Olam Labs co-founder is examining what happens when artificial intelligence must negotiate, cooperate, compete, remember, and adapt over time instead of simply producing the right answer.

A model can perform impressively on an isolated test and still behave unpredictably once other participants enter the picture. That problem is at the center of Om Buddhdev’s work in AI evaluation, where he studies how models perform when success depends on more than reasoning through a single task.

As co-founder and CEO of Olam Labs, part of Y Combinator’s Summer 2026 batch, Buddhdev builds environments in which AI agents interact with other agents and humans across social, competitive, and long-horizon scenarios. His focus is on capabilities that are difficult to capture through conventional benchmarks, including negotiation, collaboration, memory, adaptation, and performance under pressure.

“Individual capability does not automatically tell you how a model will behave around other actors,” Buddhdev says. “A system can perform extremely well on a difficult task and still struggle when it has to coordinate, respond to changing incentives, or make decisions based on what someone else is doing.”

The question has become more relevant as artificial intelligence develops beyond the familiar pattern of a user asking a chatbot for an answer. AI systems are increasingly being designed to use tools, retain information across interactions, pursue longer-term objectives, and operate with greater independence. Those changes introduce behaviors that cannot always be understood by measuring whether a model produced a correct response at one moment in time. Researchers also need to understand how performance changes when other participants are making decisions of their own.

Traditional benchmarks remain useful for measuring specific abilities, in Buddhdev’s view. The limitation appears when strong performance on those tests is treated as sufficient evidence of how a model will function inside a team, market, organization, or another environment involving multiple decision makers. A model may reason effectively while misunderstanding another participant’s incentives. It may reach a desirable result once but fail to preserve cooperation when conditions change. These are behaviors that require a different kind of evaluation from a clearly defined task completed in isolation.

“The problem becomes much more interesting when success depends on what happens over time,” Buddhdev says. “You want to know whether the model can recognize that the situation has changed, recover from a poor decision, and adjust what it is doing rather than simply repeat the strategy that worked earlier.”

Buddhdev addresses that problem by building multi-agent arenas, long-horizon simulations, evaluation systems, and training environments through Olam Labs. The company’s public arena places humans and frontier models into the same social games with equivalent information and available actions. These environments allow him and his team to examine models as participants in evolving situations rather than solely as systems completing independent assignments. The approach creates room to study abilities that emerge only when other actors influence what happens next.

That difference changes the questions an evaluation can pose. Researchers can examine whether an agent negotiates successfully when another participant has conflicting goals, whether cooperation survives after a mistake, or whether a system remembers information that becomes relevant later. An extended scenario can also reveal whether a model recognizes changing circumstances and alters its approach accordingly. The objective is not simply to make AI tests harder, but to evaluate categories of capability that isolated tasks may never require.

Buddhdev came to the problem after working directly on AI products as both a product engineer and staff engineer. He later built public agent-research harnesses before co-founding Olam Labs. Across that work, his attention increasingly moved toward models operating inside broader systems where actions affect subsequent choices and outcomes. That perspective now shapes how he approaches evaluation.

“Once an AI system is participating in an environment rather than responding to a single prompt, earlier decisions start to matter,” he says. “What happened several steps ago can change what the model should do next. That gives you information about capability that a short interaction may never surface.”

Long-horizon evaluation is particularly important to that approach. A brief interaction can show whether a model recognizes an immediate problem, while an extended scenario can reveal whether its earlier choices create difficulties later. A system may need to maintain useful relationships, recover from poor decisions, or continue toward an objective while conditions around it change. The longer time frame allows researchers to examine whether performance holds when a task develops rather than ending after a single response.

The work reflects Buddhdev’s broader view that AI evaluation must evolve alongside the systems being evaluated. As models gain access to tools and persistent context, questions about capability extend beyond whether they can generate accurate answers. Researchers increasingly need evidence about how those systems perform when several intelligent actors influence the same outcome. A high score on an isolated benchmark can establish one kind of capability without resolving those additional questions.

Buddhdev does not view interactive environments as a reason to discard established evaluation methods. Single-agent tests can still reveal reasoning ability, knowledge, coding performance, and other defined capabilities. Multi-agent evaluation adds another layer when the behavior under examination depends on relationships, conflicting incentives, changing circumstances, or decisions made across an extended period. Using both approaches can give researchers a more complete picture of what a model can do and where its performance becomes less reliable.

“The evaluation should match the kind of environment the system is expected to operate in,” Buddhdev says. “If agents are going to interact with people and with one another over longer periods, those interactions need to become part of what we measure.”

That challenge grows as artificial intelligence moves further from isolated question answering and toward sustained action. Buddhdev’s work is focused on creating environments that allow researchers to study that transition under controlled conditions. For him, benchmark performance remains useful evidence of what a model can do. It simply cannot answer every question about what happens when that model must function as one participant among many.