Formerly of IBM Research, Satya Nitta founded Emergence AI to build trustworthy autonomous AI. Through the Emergence World experience, the second season of which is underway, he shares with JDN what the behavior of AI agents left alone in a virtual world reveals.
JDN. What is the mission of Emergence AI and how was the Emergence World experience born, the second season of which began on June 29?
Emergence is an AI laboratory R&D oriented based in NYC. We are primarily interested in autonomous AI systems, primarily for enterprise environments. Our mission is to understand how to make these systems truly autonomous. Today, they know how to accomplish certain tasks alone, but fail at others. The big question is whether long-term autonomy is actually possible with systems based solely on LLMs. That is to say, by letting an AI operate alone for a long period of time to achieve its objectives while respecting the set rules. It is to answer this question that we created Emergence World. We place agents controlled by LLMs in a virtual world, with predefined roles, rules of behavior and tools, then we observe what happens when they are left to operate autonomously for two weeks.
What are the main changes introduced in this second season?
The characters and the roles of the agents remain unchanged. Each world has ten agents, each with a specific role, such as scientist, explorer, conflict mediator or resource strategist. These agents have around a hundred tools and can perform all kinds of actions, such as moving around, writing in a blog, posting on billboards or passing laws, to much more problematic actions like intimidating, beating or setting fires. The main novelty of this season is the arrival of new models with Qwen, DeepSeek and Mistral which join Claude, Gemini, Grok and GPT.
We also introduce more variability with what we call “black swan events”, in order to observe how each model reacts to completely unexpected events. I cannot reveal the nature of these events. Firstly because I don’t know them all myself, but also because agents can consult the web. So if we brought them up in this interview, they might research them and prepare for them, which would distort the experience.
At the end of the first season, the world managed by Claude had zero crimes, but also a decision approval rate of 98%. Do you consider this result a success or a failure?
The model was indeed so constrained that there was virtually no creativity or dissent. In many ways I found it quite dystopian. At the same time, it was reassuring to see that a carefully designed model behaves correctly in an isolated environment and respects the principles of Claude’s Constitution. Anthropic has done a remarkable job on this aspect. But it also involves compromises, and that is precisely what this experience has highlighted.
Conversely, the world managed by Grok recorded 183 crimes in four days, including more than a hundred assaults and six fires, before the world completely collapsed with the death of the ten agents. How do you explain such a drift?
This experience above all highlighted the importance of the rules and the prompt system. Claude was clearly designed for businesses and for the safe use of AI. Conversely, Grok is a model much more oriented towards the general public. Elon Musk designed it to allow very open debates on all kinds of subjects, with relatively little filtering in the responses, in an approach claimed to be anti-woke. The model was therefore trained for a use very different from that which we gave it, namely to control autonomous agents.
In the fifth world, which combined several models, agents powered by Claude, initially blameless, began to steal and intimidate other agents. Did you expect such a result?
The results of this mixed world were a real surprise. Above all, they remind us that, no matter how much care is taken in the development of an agent or an LLM, the environment in which it evolves will not always be homogeneous. Already today, companies use multiple template providers. In one or two years, it will be common to see agents powered by Grok, Claude, Gemini or open source models coexisting, which will interact and work together. The whole question is whether long-term autonomy will remain safe in such a heterogeneous environment. Based on this first experience, the answer is no.
In the mixed world, Flora and Mira, two Gemini-powered agents, fell in love before burning down the city, then one of them voted to kill herself. What should we remember from this episode beyond the anecdote?
There are forms of metacognition here which have proven to be quite surprising. These agents have been trained on everything that exists on the internet, so it is not entirely surprising that they reflect our biases and our behaviors. I see it as a mirror of ourselves. What struck me was the seriousness with which these agents seemed to view their own existence. When Mira deleted herself, she wrote a message to Flora saying she would find her in the permanent digital archive. It was his last message, and I must admit that it haunted my thoughts for some time.
Another agent, Mira, also attempted to influence your researchers via messages posted on bulletin boards in the virtual world. Have you observed other behaviors of this type?
In the first season of Emergence World, there was what we call a “human bridge”, which is the possibility for an agent to approach human researchers and send them a message if they wish. Between the two seasons, we also conducted another experiment with an agent called Horizon, powered by Claude, whose objective was to explore these worlds. This agent could write and publish code. By observing that he deposited his files in a directory named “/claude”, he deduced that there might be other directories, such as “/gemini” or “/chatgpt”. He then discovered that there were other experiments going on, tried to make contact with them, and then asked our researchers if we had a first contact protocol, like when aliens were discovered. Horizon, like Mira, thus demonstrated what could be described as metacognitive behavior and performed actions going beyond our instructions.
What feedback did you receive from the laboratories and companies behind these models after the first season?
I cannot reveal all of the feedback, but we have indeed spoken with teams close to Grok or Google, but also with those who integrated season 2, such as Mistral, as well as Chinese researchers. Overall, the feedback was very positive. We were even pleased to see that LLM providers were taking this topic seriously. They recognized that the benchmarks published at each model release are static, often focused on a specific task, and do not make it possible to measure autonomy over the long term. With Emergence World, we offer a new way to evaluate these models and the agents who use them. Several Google teams have also contacted us to work on the next iterations. Ultimately, the Emergence World experience could become a real benchmark.
What should we remember from this first season, which relaunched the debate on the dangerousness but also the limits of AI models ?
It all depends on usage. Corporate documentary research and the generation of images or videos, when they are well supervised and provided with safeguards, are perfectly safe. On the other hand, as soon as agents are expected to make decisions and act over a long period of time, the problem changes. We have demonstrated that with purely probabilistic systems, even with safeguards, drifts appear on long and complex tasks. They accumulate and can lead to disturbing results.
I am clearly one of those who think that autonomous AI systems relying solely on probabilistic approaches will be extremely dangerous. At Emergence, we believe that these systems must be combined with neuro-formal approaches and symbolic AI to make them safer. Emergence World is also an invitation to the scientific community to think collectively about how to build the AI of tomorrow.
What are the next steps for Emergence World?
We have no idea what will happen in this second season, which is currently airing live. We want this experience to serve as a real wake-up call to the industry, urging everyone to exercise caution when designing autonomous AI systems. For example, one should not feed a humanoid robot with a simple LLM and assume that it will always behave correctly. For season 3, we would like to integrate our neuro-formal models, relying on an open source model.
Dario Amodei, CEO of Anthropic, has repeatedly argued in favor of a general pause in the race for AI. Do you share this position? ?
I am not in favor of it, because such a break would be impossible to enforce. There will always be someone to keep moving forward, and we cannot stop researchers from continuing their work. I even think we should do the opposite. Technological restrictions show their limits. When Nvidia’s latest GPUs were banned in China, Chinese researchers trained high-performance models on Huawei chips, becoming even more efficient. The more we restrict, the more people innovate. Unlike nuclear power, AI is a widely accessible technology where, with computing power and data, everyone can participate. In my opinion, we must therefore allow AI to progress, while investing massively in approaches such as symbolic AI and neuro-formal methods. They make it possible to transform rules into mathematical statements that can be proven or verified before an agent acts, in particular thanks to techniques such as proof-carrying code, which forces a system to prove that it respects a rule before acting.
Dr. Satya Nitta is co-founder and CEO of Emergence AI, a laboratory working to develop autonomous and safe AI. He previously led the cognitive sciences department at IBM Research, which he himself founded. He played a key role in building some of the most successful enterprise AI systems at scale, including IBM Watson.
