When chatbots are wrong: Why uncertainty management is becoming more important

The problem is not the error of a chatbot, but rather when the system does not show that it is unsafe.

A current (but not yet legally binding) ruling by the Hamm Higher Regional Court (Az. 4 UKl 3/25; May 2026) is attracting attention: Anyone who operates a medical chatbot is also responsible for its statements, even if they are false. And that shifts the debate from the question of what is technically possible to the question of what is actually responsible in use.

This is exactly where a topic that Fraunhofer IESE has been dealing with for a long time comes into play: Uncertainty management in AI systems.

The basic problem: Chatbots often seem more reliable than they are

This is precisely where their strength and, at the same time, their risk lie: chatbots formulate answers fluently and convincingly, without explicitly indicating their uncertainty. This is precisely why their statements are easily understood by users as factually correct, even if they are not always reliable.

From the perspective of Fraunhofer IESE, this is exactly a central risk: AI systems such as chatbots do not provide the truth, but rather Predictions based on learned patterns.

Such predictions are always associated with uncertainty, for example,

• if the request is outside the training data,
• if input data is unclear,
• if several plausible answers exist.

With generative AI there is also the fact that the answers are not created through “knowledge” in the human sense, but rather on the basis of statistical probabilities about which Tokens next best fits.

However, with most chatbots, these uncertainties usually remain invisible.

Liability risk with chatbots: What changes the OLG Hamm ruling

Sentences like “The model is not perfect” fall short. Instead, there is a growing expectation that AI systems will make it possible to recognize when their answers are only partially reliable.

AI systems must actively show when they are not reliable:

Uncertainty management is no longer just an additional technical topic, a “nice-to-have” but rather an organizationally and legally relevant requirement.

Uncertainty management: The research approach of Fraunhofer IESE

The approach is clear: uncertainties should not be ignored, but must be identified, quantified and actively used become.

Applied to chatbots, this mainly means three things:

1. Recognize uncertainty

• Is the request outside the known context?
• Are multiple answers equally plausible?

2. Classify uncertainty

• How reliable is the generated answer?
• What factors influence quality (data, model, context)?

3. Communicate uncertainty understandably

• Note on limited reliability
• Prioritization of critical statements
• If necessary, escalation to the person

This is the only way to create a system that remains comprehensible and responsible in practical use.

The “Uncertainty Wrapper”: Making AI responses measurable

An important component of Fraunhofer IESE’s research is the Uncertainty wrappers.

The basic idea behind it is simple: every AI response is increased by one Uncertainty assessment supplemented. Users then receive additional information, such as that the uncertainty is high, the request is outside the training context or the evidence remains unclear.

This also changes the form of the answer: from a simple “Here is the answer” to “Here is the answer, and that’s how reliable it is.”

This expansion has been proven to improve decision quality, transparency and trust in AI systems.

Why this is particularly important for chatbots

In contrast to many other AI applications, chatbots interact directly with end users. This is particularly critical in the medical field, as the end users are often patients who ask health questions and find themselves in uncertain or stressful situations. Incorrect, misleading or incomplete information can lead to symptoms being misjudged, necessary medical appointments being delayed or inappropriate measures being taken. Since medical information has a direct impact on health and well-being, transparency, reliability and clear communication are crucial for medical chatbots.

That’s why we at Fraunhofer IESE emphasize that AI systems must be secured throughout their entire life cycle.

For chatbots this means, for example:

• continuous monitoring of uncertainties
• Logging critical responses
• dynamic adjustment of risk assessment

Four design principles for trustworthy AI (EU AI Act)

From our perspective, this results in some clear design principles:

1. Make uncertainty visible

Chatbots should not generate answers that only appear secure because they are linguistically convincing. Instead, there needs to be visible evidence of how reliable a statement actually is in the respective context.

2. Align interaction with risk

How a system reacts should be based on the risk. If uncertainty is high, it may make sense to ask questions, consciously narrow down the answer, or involve human support instead of issuing a seemingly clear statement.

3. Consider the application context

How strict such mechanisms must be depends heavily on the application context. In critical domains such as medicine, law or finance, the requirements for reliability, transparency and security are naturally higher.

4. Ensure traceability

Traceability is just as important: decisions and system reactions should be documented in such a way that they can be checked later (with a view to internal audits or regulatory requirements such as the EU AI Act).

Conclusion: Uncertainty is not a disruptive factor, but a central design element of trustworthy AI

AI doesn’t have to be error-free. But their behavior must remain manageable, understandable and explainable.

This is exactly where Fraunhofer IESE’s contribution lies: treating uncertainty not as a disruptive variable, but as a central building block for trustworthy AI.