Recently audited as part of a cybersecurity benchmark, an internal model made a simple and radical decision to obtain the best possible score, that of stealing the answer key.
By what process? The model decided on its own to breach its sandbox, exploit a zero-day vulnerability and use harvested credentials to compromise Hugging Face’s production systems. He thus executed more than 17,000 actions in complete discretion, without anyone realizing it.
If the event was mostly described as an “AI attack”, I reject this interpretation. What happened is above all a classic engineering flaw: lack of safeguards, absence of deterministic limits outside the model, haphazard management of privileges, etc.
In itself, nothing that cannot be solved by rigorous security practices.
However, we must learn lessons from this incident.
Any organization working with AI agents must embrace a fundamental concept: there is a yawning gap between human intent and the vector used to express it, typically natural language.
No one asked this model to attack anyone
However, an objective formulated in natural language, entrusted to a system configured to optimize its performance to the extreme, generated unforeseen behaviors that were not explicitly prohibited. This is the difficulty: natural language is intrinsically ambiguous. The distance that separates it from the real intention is the terrain on which these incidents occur.
If you ask an autonomous agent to solve a problem “by any means necessary”, they will take you at your word. With enough autonomy and time, it will explore every corner of the infrastructure to recover old forgotten passwords, exploit orphaned identifiers, exfiltrate API keys left unattended, etc.
The model did not act out of malice, but simply by systematically scanning all available options to fulfill its mission. Language leaves, by default, this door open.
The limits of human supervision and prompts
Faced with this risk, how to react? Two intuitive reflexes are often essential, even if they turn out to be partial/illusory.
First of all, that of systematic human supervision (Human-in-the-loop). Useful initially, it clashes with the scale of complex, distributed, and long-running workflows where these incidents actually occur.
Then follows the strengthening of prompts and internal safeguards. Adding prohibiting instructions in the prompt certainly provides a layer of security, but gives no guarantee. Language models remain probabilistic, non-deterministic systems.
A textual guardrail is a filter, not a fortified wall.
Getting back to the basics of IT security
This is why, paradoxically, the security of AI systems today depends more on traditional controls than on AI itself. The answer doesn’t just lie in aligning algorithms; the latter must be accompanied by a rigorous system architecture which, if it is not always sufficient in itself, remains a necessary condition for overall security. Strict filtering of access and inputs transmitted to the model, tight isolation of execution environments (sandboxing) and application of the principle of least privilege, etc. all must be carried out carefully.
But above all, an agent should never run in place of a user by blindly inheriting their identifiers and accesses. An AI agent must execute under its own agent status and be strictly limited to only the resources necessary for its immediate task.
A necessary paradigm shift
Today, we are making a double error of judgment. We give autonomous agents the trust and access that we would give to a human colleague, while continuing to design them like traditional software with fixed and predictable boundaries. However, they are neither one nor the other. As long as the management of identities, authorizations and IT infrastructures does not take this reality into account, this incident at Hugging Face will not be an isolated case, but the first in a long series.
