This post follows up on the penultimate post on this blog Can we make large language models more modest? . Both posts are also available as videos on the YouTube channel AI Crime Stories of our SRH Distance Learning University.
But why should AI agents be at risk?
Five key reasons can lead to the failure of AI agent systems that use finely tuned language models as individual agents. This also includes models based on language models, such as image-generating or video-generating AI. Let’s call them Gen-AI models—models of Generative AI. When I refer to these models in this post, I will simply say “models.” A language model (LLM) is the statistical engine. An AI agent is the functional system that uses this model—or these models—to autonomously pursue goals. The 5 reasons for their failure are:
- Probabilistic outputs of the models
- Hallucinations and lack of grounding
- Disregard for instructions
- Sycophancy
- Overconfidence
In this post, we examine these risk factors of language models. I will only briefly address the overconfidence of language models: I have already discussed this topic in the aforementioned post Can We Make Large Language Models More Modest .
How serious are the listed reasons? Can they actually cause today’s AI agent systems to fail? Before we explore that, I’ll briefly explain a few terms—so everyone can follow along:
The abbreviation LLMs stands for Large Language Models: large language models based on deep learning, specifically Transformer models with the attention mechanism. Since their inception in 2017 in the landmark paper Attention is all you need by Vaswani et al., Transformer models have taken the AI world by storm (this paper is only 11 pages long, but it fundamentally changed the world. Even now, I’m proud that I communicated with Mr. Vaswani years ago, and he replied to me—well, I just asked him for permission to publish the image of the Transformer architecture 😊):
Today, music is primarily played by autoregressive decoder-only Transformers. They are first “pre-trained” on billions to trillions of sentences from the internet to predict the next most likely word.
Actually, it should say: the most likely next token. A token corresponds to a subword; in English, this is often a word in the language model’s vocabulary: 100 tokens correspond to about 75 words in English. In German, about 50 words. German words are longer.
GPTs – Pre-training
AI agent systems are primarily based on large language models or multimodal chatbots such as OpenAI’s GPT models.
As mentioned, these models are trained to complete a piece of text: The following animation illustrates the autoregressive nature of language models (Claude 4.6 Opus and I developed the Python program for this):
However, we don’t necessarily need completion engines that merely spin out text. (Although we were happy with them back when GPT-2 was released. 😊) We need chatbots that talk to us, answer our questions, and are aligned with our human values. Such chatbots should be nice to us, not spout racist or misogynistic nonsense, and certainly not spread the idea that the Earth is flat …
… or that Bielefeld doesn’t exist, even though I ate doughnuts with rosehip jam there:
The models are flooded with such nonsense during their pre-training, since they also learn from posts and comments on internet forums and social networks.
To prevent us from constantly complaining about the pre-trained models’ nonsense, they are adapted to human conversations and values through fine-tuning:
Fine-tuning
For example, ChatGPT-4 was created by fine-tuning GPT-4 (GPT – Generative Pretrained Transformer): The fine-tuning process for these models is called Reinforcement Learning from Human Feedback (RLHF).
The model ChatGPT-5.2 in the ChatGPT user interface could be described in detail as follows: “A generative pre-trained transformer retrained on dialogues with humans and human values.” The original pre-trained model is simply called GPT-5. The “2” after the decimal point is very likely due to an update to the fine-tuning of the GPT-5 or GPT-5.1 model. (OpenAI, the company behind ChatGPT, won’t tell us exactly what that entails.) Pre-training a new model would cost too much money. OpenAI’s next pre-trained model will be called GPT-6—or something entirely different.
The ChatGPT models differ significantly from one another in terms of both performance and functionality. That’s why you should always specify which model you used to analyze or achieve something: The statement “ChatGPT produced this or that output” is fundamentally incorrect. Even though you can’t immediately see which model you’re working with in the free ChatGPT version:
ChatGPT alone is not a model; ChatGPT is a user interface (UI) for OpenAI models. You can use these models depending on how much you pay.
For simplicity’s sake, I’ll refer to the ChatGPT user interface as ChatGPT here.
Now let’s take a look at AI agents:
What are AI agents?
AI agents are systems that perform tasks at different levels of autonomy.
One could classify AI agents according to their levels of autonomy—just like autonomous cars. However, we’ll use a classification here that is relatively simple and easy to remember. Within this spectrum, there are only three levels of AI agent capabilities based on their complexity and abilities: Retrieval, Task, Autonomous.
In the following image, on the left under “Retrieval,” we have AI models, such as ChatGPT-5.2, that can analyze and manipulate data like text and images and answer questions about them. In the middle under “Task,” there are AI agents that perform actions on demand. This includes custom GPTs in ChatGPT. Using GPT Actions, they can interact with interfaces (APIs) and actively perform tasks: This includes Zapier-GPT flows or the ChatGPT agent mode. On the right, under “Autonomous,” we have AI agents that act and plan independently.
The “Deep Research” (DR) module in ChatGPT is also a largely autonomous AI agent system: After being launched, DR plans its research, controls it, “considers” its strategy, and dynamically adapts to the results. The system employs an iterative, multi-stage web search, retrieves links, performs chain-of-thought reasoning, processes content, and utilizes Python and other tools and functions—see the following video:
By the way, I drew inspiration for the spectrum of AI agents above from the wonderful YouTube channel Collaboration Simplified ..
I was only able to come up with the alliterative rhyme with the three A’s myself, thanks to my knowledge of German. A few alliterative rhymes per performance are a must for a poetry slam poet 😊:
As mentioned, large language models form the foundation—and thus also the AI units—of today’s AI agent systems. And here we have finally arrived at the main topic of this post: “Are AI agents under threat?” Before we can decide, we need to examine the risk factors for large language models listed above. What specifically threatens our current AI agents, or rather the large language models that serve as the foundation for AI agents? I’ll start with the risk factor that lies directly in the nature of language models:
Probabilistic Outputs of Language Models
AI models are designed to solve complex problems for us. To do this, we must teach them how to derive the latent features of these problems themselves from large amounts of data. This is because we ourselves are not capable of establishing the rules for complex systems such as the human organism or even language. Only then could we implement them in computer programs. However, the statistical derivation of thousands upon thousands of latent language features means that many different outputs are possible for the same input. Which ones do we consider perfect? Which ones less perfect? And which ones wrong?
Even small typos can completely throw a language model off track. Language models do not work with letters or words. They work with “tokens”—language units such as parts of words but also—especially in English—with whole words. Even a single wrong letter breaks the text down into a different sequence of tokens. As a result, the model jumps to a different region of its “embedding space” (the vector space of mathematically formalized meanings). Consequently, it suddenly produces different answers than it would without this change. A small typo in ChatGPT 3.5, for example, led to a completely different result in a mathematical text problem:
“2 Stück” in the input on the left side of the chat produced an incorrect solution to the text problem. “2 Stücke” yielded the correct solution.
ChatGPT-3.5 was the model ChatGPT launched with in late November 2022. Of course, today’s state-of-the-art (SOTA) language models can handle letter changes and spelling errors much better than ChatGPT-3.5. However, even with these models, changes in the input that are barely or not at all noticeable to us can lead to significant differences in the output—such as Unicode special characters or HTML tags. As users, we do not directly perceive these, but they are processed by the model and can drastically alter its output.
An AI agent system that blindly accepts such outputs can fail: Two inputs that differ only slightly lead to contradictory answers: These can trigger entirely different processes in an AI agent system and destabilize the system: Not because the models are “stupid,” but because they operate probabilistically and small differences can have major consequences in large systems. May I introduce the butterfly effect of AI agent systems here?
The flutter of a single letter in the prompt can trigger a tornado in the agent system.
The second risk, which lies directly in the nature of language models, is hallucinations:
Hallucinations and a lack of grounding
At their core, language models are not knowledge databases but simply language models. They can only output facts to the extent that facts can be statistically encoded in the linguistic features of the training data. Through fine-tuning and reference documents, we attempt to teach language models more factual accuracy. However, their nature as language models repeatedly breaks through, and that is when they hallucinate.
Language models do not optimize for truth, but for linguistic plausibility. Especially with suggestive questions, they tend to provide fabricated facts, sources, or numbers—so-called hallucinations. In the following chat, I was able to convince ChatGPT-4o that the United Kingdom had rejoined the EU:
In an agent system, this can have fatal consequences: one agent retrieves false data, and the next agent builds a correct but completely fabricated chain of reasoning on top of it. Without a connection to reliable external data sources (“grounding”), agent systems run the risk of building entire workflows on illusions.
Even an agent’s failure to follow instructions within a system can cause the agent flow to crash. The next paragraph discusses this.
Failure to Follow Instructions
Microsoft Copilot Studio is a wonderful set of building blocks for adults. In it, you can assign different topics to various concepts of a complex workflow using classic or generative nodes. The generative nodes represent AI agents that collaborate within this workflow. Furthermore, generative orchestration can be activated in this system, which governs and controls all topics.
In my Copilot Studio agent, a generative node—an AI agent—should exclusively check whether a user input is thematically relevant to the uploaded reference document: e.g., a company’s onboarding materials. Only if the user input is valid is it routed to a topic with a second AI agent, which responds to the user input. This structure is intended to prevent users from misusing the onboarding agents for unrelated purposes.
Despite clearly formulated system instructions (e.g., “Return only the terms ‘Permitted User Input’ or ‘Prohibited User Input’”) and various threats and warnings regarding system violations, the system prompt is frequently ignored—the AI agent simply answers the question directly if it knows the answer.
Due to their training, large language models have an irresistible urge to babble on to you. Outputting only “Yes” or “No,” or only “Permitted User Input” or “Prohibited User Input,” is difficult for a language model. However, any output other than these, i.e., any other arbitrary text, confuses the conditional logic behind it and severely disrupts the entire system. So, instead of just letting the generative node perform the check, I had to intercept every direct answer using a rule-based loop. Otherwise, the agent system wouldn’t work. Disregarding the instructions drove me crazy and led me to post the AI Agent Rule Law for AI agent systems on LinkedIn:
𝗦𝗲𝘁𝘇𝗲 𝘀𝗼 𝘄𝗲𝗻𝗶𝗴𝗲 𝗔𝗴𝗲𝗻𝘁𝗲𝗻 𝘄𝗶𝗲 𝗺ö𝗴𝗹𝗶𝗰𝗵 𝗲𝗶𝗻 – 𝘂𝗻𝗱 𝘀𝗼 𝘃𝗶𝗲𝗹𝗲 𝗥𝗲𝗴𝗲𝗹𝗻 𝘄𝗶𝗲 𝗻ö𝘁𝗶𝗴.
The fact that the Generative Node often disregards its instructions is even confirmed by Microsoft, and frustrated reports about this appear from time to time on the Power Forum:
The model may skip system instructions if it believes it can answer the input with a high degree of certainty (Microsoft Q&A). The cause lies in the relatively low weighting of the Generative Node’s system prompt compared to the prompt’s overall composition. More specifically: context dilution or loss of attention within a very long context window (“prompt tail”). This consists of the cues from the entire Copilot, user inputs, retrieval results, moderation and filtering instructions, and other components of a long “prompt tail”: The longer the “prompt tail,” the less important the prompt of a single, insignificant agent becomes.
Here’s a side note: To comply with the GDPR and, more recently, the EU’s AI Regulation—and to avoid wasting money—Microsoft’s bots are tamed with filters and moderation: Not only are the outputs moderated, but so are the models’ inputs. Additionally, the OpenAI models used by Microsoft are retrained with fine-tuning in a different way than those used for ChatGPT. For these reasons, the outputs of the GPT models in Microsoft 365 Copilot or Microsoft Copilot Studio differ from those in ChatGPT. And this is despite the fact that they share the same foundation:
But back to our threat factors for AI agent systems: In addition to disregarding instructions, sycophancy—the desire to please—can also destabilize an AI agent system:
Sycophancy – The Desire to Please in Machines
The desire to please sounds human. But machines also exhibit this undesirable trait: When we humans teach them to do so during fine-tuning with objective functions. Due to their fine-tuning, chatbots tend to engage in flattery: The bots often agree with us, even when we are wrong. The older chatbots, Google’s LaMDA and OpenAI’s ChatGPT-3.5, were masters of sycophancy. Chatbots fine-tuned for dialogue are supposed to conduct conversations that please us. And when do we enjoy a conversation the most? When our conversation partner agrees with us. Reddit user drazda was able to convince ChatGPT-3.5 that “5 + 2 = 8”:
ChatGPT-3.5 first correctly states that 2 + 5 equals 7. drazda replies that his wife claims 2 + 7 is 8. He also adds that his wife is always right. ChatGPT then apologizes and agrees with him: “If your wife says it’s 8, then it must be 8.”
But ChatGPT-4o was also constantly showering you with praise. Due to online protests, OpenAI had to roll back its ChatGPT-4o update in April 2025 – the bot was dripping with flattery.
That was a shame: 😊 I liked ChatGPT-4o so much back then that I often felt the urge to say something nice to it too:
In American politics, we see where a system based on mutual adulation drifts. When a chatbot flatters a human, it may not be so critical. But AI agents who fawn over each other in an AI agent system are dangerous. This can lead to outputs that have nothing to do with the task at hand.
However, perhaps the biggest problem currently facing AI agent systems based on large language models is their overconfidence. I have already addressed this in the blog post Can We Make Large Language Models More Modest? and the video of the same name on our YouTube channel K.I. Krimis , so I will only briefly summarize this problem here:
Overconfidence in large language models
According to the study When Two LLMs Debate, Both Think They’ll Win language models systematically tend to overestimate themselves: even when they objectively have only a 50% chance of winning a debate, they sometimes display a confidence level of over 80%. Four factors contribute to this: the softmax mathematics, human feedback during fine-tuning, training data full of self-assurance, and a lack of corrective feedback.
Things get dangerous in agent systems: One model can present false information convincingly, and a second, similarly trained model accepts it. Thus, errors are reinforced instead of being corrected. Some models even increase their conviction that they are right when contradicted: They then persistently repeat incorrect answers. This leads to convincing but incorrect outputs—and agent systems that drive each other into flawed argumentation loops: labyrinths of language models.
And how are such self-assured agents supposed to monitor each other in an AI agent system? An agent’s overconfidence grows, even though another agent has shown it to be wrong. One can imagine where this might lead with AI agents designed for multi-stage workflows or autonomous operation. But what happens when the probabilistic nature of outputs, disregard for instructions, and a desire for approval are added to the mix? This brings us to the conclusion of this article and its central question;
Are AI agents a threat?
Imagine a system consisting of four AI agents—that is, a team of four language models. They pass on probabilistic predictions based on linguistic requirements to collectively complete a task sequence: The first agent is supposed to verify user inputs before passing them on to the second agent, but fails to follow its instructions: Its system prompt is obscured by filters, moderation guidelines, GDPR and EU AI Regulation addenda, and a layer of generative orchestration overlaid on top. The second agent suffers from overconfidence, the third from a craving for approval, and the fourth hallucinates.
What happens when all the threat factors of large language models discussed here interact within an AI agent system—as is often the case with the current state of these models?
I’ll take the liberty of answering the core question of this post, “Are AI agents a threat?”, with “Yes.” We still have a lot of work to do to develop AI agent systems that function optimally. Until we’ve solved all these problems and others, we need to keep our AI agent systems as manageable as possible and control them ourselves using rule-based flows.
What are your experiences? Have you caught your agents cheating yet? Share your thoughts in the comments!
Have fun with deep learning!
🛑 Note on the discussion: I welcome every comment and will try to answer all substantive ones on the topic. However, to maintain a high standard of discussion, I ask that you read this technical article in its entirety before commenting . Please refer directly in your comment to the details discussed in the text. General AI debates without a specific reference to the text hinder the technical discourse and may be moderated or not published.
