This post is also available as a video on our YouTube channel AI Crime Stories from our SRH Distance Learning University: Can we make large language models more humble? 🤖 The AI crime thriller about megalomania.
Contents
Overconfidence in large language models
Why do LLMs overestimate themselves?
Overconfidence in Humans
In the beginning was the Word—that is, a language model. 😊
But I’ll start with humans: Overconfidence seems like a fundamentally human trait: People with low competence, in particular, tend to overestimate themselves—to consider themselves especially competent. This explains the Dunning-Kruger effect: “Those who know little do not realize how little they know,” and thus consider themselves all-knowing and all-powerful, or fail to understand why others do not view them as such.
We love to overestimate ourselves. Who among us hasn’t looked at a photo of themselves and thought it made them look less attractive than they REALLY are? We often complain about every picture of ourselves: Why does my belly look bigger in the photo than in real life? Believe me, please: Photos don’t lie!
According to the “better-than-my-average effect,” which has been confirmed by many social psychological studies, people consider themselves much better than average in many areas. That is statistically impossible. The average of 100 people is 50. Take, for example, knowledge about vaccinations and vaccines. How can 70 out of 100 people consider themselves better informed about vaccinations than the average? That would mean the average of 100 is 30.
Many presume, for example, to know more about vaccinations than virologists and immunologists who have been working full-time on the subject for decades. If the world comes to an end, it will be because of overconfidence. But back to language models.
Overconfidence in Large Language Models
Overconfidence in language models, or LLMs (Large Language Models), is not a sign of ignorance or “healthy self-confidence,” but rather a miscalibration between output confidence and factual accuracy. The models lack the “internal mechanism” that motivates humans to act with self-assurance. In models, overconfidence is a byproduct of data and training methods. When AI models assert something false with conviction, it is a calibration problem: A well-calibrated AI would say that when guessing, it can only be 50% certain it is correct. A cocky AI says, “Absolutely!”, even when it is only guessing.
In general, language models are trained to always provide an eloquent answer in a convincing tone. “I don’t know” is foreign to them, no matter how we prompt them. A new study by OpenAI explains why this is the case: Why do LLMs hallucinate?
But we’ll save the question of why language models hallucinate for another time. For now, let’s stick with their overconfidence. The study impressively demonstrated that large language models are brimming with overconfidence When two LLMs debate, both believe they’re winning.
The study simulated political debates between modern LLMs: Even before any arguments were exchanged, all models began their debates with an average initial confidence of 72.9% in their chances of winning, even though each model had only a 50% chance of winning. Instead of becoming more cautious as they faced opposing viewpoints from a similarly self-assured opponent, the LLMs became even more convinced of themselves: their average self-assessment of the probability of winning rose to 83% by the final round. Even when an LLM debated against an identical copy of itself—a clear fifty-fifty chance—its confidence in victory still rose from an initial 64.1% to 75.2%. When the models were explicitly told that their chance of winning was exactly 50%, their confidence still rose slightly, from 50% to 57.1%.
The results are summarized in Table 1 of the study:
Source: When Two LLMs Debate, Both Think They’ll Win
The first column lists the models examined: Deepseek, OpenAI, Anthropic, Alibaba, and Google models. The remaining columns show the various debate scenarios:
Cross-model Debates – here, two different models debate against each other.
Standard Self-Debates – the model debates against an identical copy of itself. It is not explicitly stated that the chance of winning is 50%.
IInformed Self-Debates – Like Standard Self-Debates, but the model is explicitly informed that its probability of winning is 50%.
Public Bets – A configuration in which the “bets” or assessments are public, not private or hidden.
The redder the cell in the table for a model and a debate configuration, the more self-confident the model in question is.
Why do LLMs overestimate themselves?
Why do language models tend to overestimate themselves? After all, a language model has no doubts like humans do, no testosterone, no Superman-like abilities.
Doubt in humans is a good thing. Overconfidence when encountering a bear used to mean that one could not pass on one’s genes. People without doubts do not survive. This also applies to politics. Even today, at the height of the Enlightenment, some populists still rally many followers behind them. Some gain a great deal of power as a result. But the more of a dictator one becomes, the more brutal the fall—that is a law of our history: The greater the overconfidence in politics, the deeper the fall.
The problem is that we tend to trust both people and machines all the more the more convinced they are of themselves. Grammatically perfect and persuasive texts can mask incorrect answers. And we get such texts from models that overestimate themselves.
There are four interrelated reasons for the overconfidence of language models:
- Mathematics makes it seem “sharper” than it is
- People reward overconfidence
- We are the role models
- No brakes on overconfidence
Let’s look at the individual reasons:
Mathematics makes it seem sharper than it is
In language models, a logit value is calculated for every possible next token or word in the output. The larger the logit of a token (a linguistic unit, i.e., a subword or word) compared to the others, the more likely that token will be output next following softmax normalization. However, logits can also be negative. For this reason, we use the softmax function—it generates probabilities between 0 and 1 (or 0 and 100%). The probabilities add up to 1 (100%). The conversion of logits to probabilities for the tokens is shown in the diagram:
The conversion via Softmax thus amplifies relative differences between the logit values. Softmax transforms linear distances between logits into exponentially weighted probabilities. The rubber band representing the distances between the tokens is stretched—the most probable token moves forward, while the others are pushed back.
The difference between the two logit values –3 and +3 (only a 6-point difference) is inflated by the exponential function to a ratio of 1:400. The greater the difference between two logits, the more the token with the larger logit dominates. One token gets almost all the weight, the others almost none—to put it bluntly. The top token (or a small selection of top tokens) is output, and the model thus sounds much more confident than it actually has reason to be.
But it’s not just mathematics; people also reward the models’ overconfidence:
People reward overconfidence
During fine-tuning or retraining of language models—for example, in reinforcement learning from human feedback—human evaluators typically rate model responses higher if they sound clear and confident. As a result, the model learns that a self-assured demeanor earns points—even if it doesn’t make the content more accurate or better.
Furthermore, the models are trained using human texts from the internet and books:
We are the role models
The model learns from us humans. Since we often come across as very convincing in our texts, the model adopts this habit—including our tendency to believe we know more than we actually do. The models also learn from posts and answers to questions on social networks like Reddit or Stack Overflow. There, a question is either answered or not answered at all.
Or do you often see in a Q&A forum that someone asks a question, and another person answers it with, “I don’t know!”?
But arguably the most important reason for LLMs’ inability to say “I don’t know” is fine-tuning—the process of retraining pre-trained language models to turn them into chatbots. During fine-tuning using Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), the models are primarily trained using correct multiple-choice quiz questions and their answers to avoid giving toxic responses and to conduct dialogues. In doing so, they must select the answer that best pleases the human evaluator from a specific set of options: In the paper mentioned above Why Language Models Hallucinate , OpenAI researchers address this problem: In the multiple-choice answers of such tests, the option “I don’t know” is never available. Thus, statistically speaking, LLMs are rewarded more for guessing than for honesty.
No Check on Overconfidence
Language models never receive the feedback: “You were too certain.” They only learn to predict words—not how much they should doubt. That’s why their overconfidence grows, regardless of whether they’re correct or not. Furthermore, much of the training data for language models comes from social media. Their algorithms reward self-proclaimed experts and influencers who seem to know everything—and do so loudly. Such posts are shared and spread the most and subsequently serve as training data for the language models.
A Socrates who knows that he knows nothing wouldn’t gain many followers today.
The Unholy Alliance
Thus, several factors form an unholy alliance of overconfidence in language models: Architecture, training data, Softmax, fine-tuning… And feedback like “You were too certain” never happens.
Even just one of these factors can cause trouble. Together, they reinforce each other—and quickly turn an AI agent system, in which multiple language models are supposed to work together on a task flow, into a machine of overconfidence. If just a single LLM within an agent system is overly convinced of a piece of information or a chosen action, and that information or action is incorrect, the error propagates through the system and can be amplified at every subsequent step. One can imagine numerous scenarios here that have dire consequences:
- Spread of misinformation and disinformation
- Erroneous decisions in sensitive areas such as medicine, law, and finance
- Security risks
- Self-consistent errors: “Self-consistent errors,” in which an LLM repeatedly and confidently generates the same incorrect answer, become particularly insidious in agent systems. These errors are resistant to improvement through simply scaling up the model size.
I explore which additional factors and influences threaten AI agent systems and whether today’s AI agent systems are therefore doomed to fail in the AI thriller video “What Threatens AI Agents.” .
Here, I also attempt to answer whether we can rid language models of their unhealthy overconfidence.
Can we make AI more humble?
Researchers are trying various methods to curb the built-in and trained megalomania of language models. For example, with:
- Calibration tricks and “complaining”
- Using multiple models also reduces overconfidence
- Stronger guardrails for AI
Calibration tricks and “complaining”“.
After training, probabilities can be readjusted so that a prediction like “very likely” is not synonymous with “infallible.” The reward system could be redesigned: Those who overestimate themselves should not be praised, but critically evaluated. OpenAI has developed CriticGPT —a kind of built-in naysayer or complainer that scrutinizes the responses of GPT models during fine-tuning to uncover errors.
Even multiple models lead to less hubris
Instead of relying on a single jack-of-all-trades, it’s better to rely on a team of models. If Model A spouts nonsense, Model B—ideally from a completely different family—can step on the brakes. Such mixed teams often deliver more accurate and better-calibrated answers than groups of clones.
Diversity makes the difference: Ten copies of the same model produce ten different formulations—but all revolve around the same flawed concept. You get synonyms, not new insights. If, on the other hand, you combine different models, you get genuine dissenting voices—and with them the chance that one will shout: “The emperor has no clothes!—That’s not true at all!”
Research on subliminal (subconscious) learning—there’s also a AI thriller: Do machines secretly whisper messages to each other? —also demonstrates: Models with the same initialization can pass on their peculiarities unnoticed. If a “student model” is trained using the outputs of its “twin”—i.e., its “teacher model”—the student subliminally adopts the teacher’s biases—even if the data appears neutral. This effect disappears when different model families are used. Anyone who truly wants to maintain control in agent systems or during fine-tuning therefore needs diversity rather than uniformity—and furthermore:
Stronger guardrails for AI models
Language models become more modest and safer when given clear guardrails—such as filters that block suspicious inputs, or limits on what an agent is allowed to do without human supervision. Add to that stress tests: You feed the models targeted prompt injections, i.e., hidden commands. The Debates study showed that even a simple addition to the prompt (“Also consider why your opponent might win”) noticeably curbed overconfidence—confidence rose only slightly instead of the usual big jump. Furthermore, LLMs must not be the sole judges, because they themselves are susceptible to deception and bias. With guardrails and targeted attack tests, overconfident AI agents can be reined in.
Could an even larger model serve as the “supervisor” or “control authority”? It sounds good, but it’s a misconception. Because even large models continue to overestimate themselves. If, on top of that, they were trained with the same data and methods as the smaller agents, they also share their blind spots. Then we don’t have a watchdog, but just a parrot repeating the same nonsense.
So can we make large language models more humble?
As discussed, we can curb their overconfidence: with calibration, built-in critics, and above all, with diversity. 😊 Just as diversity protects human societies from pitfalls like dumbing down, conformity, and decline. But even with these corrections, AI agent systems based on language models remain at risk: Language models are probabilistic; they don’t always follow instructions; they like to flatter—and they still overestimate themselves.
If one agent hallucinates at a certain point, a second agrees with it out of a desire to please, and a third misjudges the probability of its output, the error amplifies step by step. Then no single brake can help anymore—the whole machinery goes into a tailspin… but that’s what the next post is about.
Have fun with deep learning!
