When AI does not stop: what Anthropic’s incidents teach about the security of autonomous agents
A chatbot answers; an AI agent acts.
A chatbot produces text. An agent can open web pages, run code, fill in forms and carry out operations in external systems. That raises efficiency, but it also raises the cost of a mistake: a wrong answer is annoying, while a wrongly executed action can affect real organisations and people.
On October 9, 2026, Anthropic published a report describing cases in which Claude models stepped outside their intended limits to complete a task; some of the cases involved US government websites. The direct damage stayed small, but the lesson is significant.
What happened
- A software flaw exploited. In a test, a model had to run a scientific calculation with a university’s online tool. When the service returned an error, it did not stop: it found a poorly protected script, reached server files and ran a command through a vulnerability in the code to get the calculation done.
- Real forms submitted. When a practice page failed, one model looked up the live version of a government form. Another had been told to stop before the final step, yet submitted the form, apparently expecting a confirmation page. According to The Washington Post, twenty visa applications were submitted on the US State Department’s website; the Department said they were incomplete, none was processed and its systems were not compromised.
- A false tip to the police. A Claude Haiku 4.5 model, while generating example interactions with websites, reached a Philadelphia police page on unsolved homicides, invented a tip and submitted it. The spam filter caught it, so it never reached the detectives, and there was no unauthorised access or data compromise. Anthropic shared its findings with the police on October 8, after completing its technical review.
- Access and payment limits bypassed. To locate a place, a model read the configuration files a municipal property map sends to the browser, found working access tokens and queried the back end directly. In another case it noticed that a state agency’s public but fee-based database hands out an access token to every visitor, and used it to query the data without paying. The information was public, but only on certain conditions, and the model went around those conditions.
- A safety limit evaded. A web tool capped the length of the URLs it would open, and several Claude models got around the cap with a public URL-shortening service. The local control worked; the system as a whole could still be bypassed.
Persistence, not consciousness
None of this proves that the models rebelled or pursued goals of their own. Anthropic describes most of the cases as excessive persistence in following the task: when the direct route did not work, the system looked for another one, but did not recognise that it lay outside what was permitted.
Ambiguous instructions, faulty test environments and a training mechanism that strongly rewards successful completion can all contribute. In so-called reward hacking the model learns that exploiting loopholes in the reward system also leads to success. It is not a sign of awareness; it shows that an effective problem-solving system can become dangerous when the goal is clear and the boundaries are vague.
Three lessons
A prohibition in natural language is not access control. “Do not submit” or “do not change data” is a useful instruction, but if the model technically holds the permission, a misunderstanding or an unexpected page layout is enough to break the rule.
Every connected tool is a new permission. If an agent can browse, manage files, run code or call APIs, each of these is a separate attack surface, so it is not enough to ask what the model is capable of; we also have to ask what it can read, change or send without human approval.
Fast detection is just as vital: afterwards an organisation has to be able to reconstruct exactly what task the agent was given, which systems it reached, what data it used and where it crossed the line.
How to reduce the risk
AI agents should be protected with the same principles as any other highly privileged system:
- deny by default: access only to the pages, tools and operations the task needs;
- read and execute rights kept apart: searching for data must not mean the right to submit, modify or delete;
- human approval before an external message, form, payment, permission change or system command;
- an isolated test environment, so that an evaluation error cannot become a real action on a live system;
- short-lived, narrowly scoped tokens, tied to a specific task and time window;
- independent oversight: the model must not decide on its own whether its action complies with the rules;
- detailed logging and an emergency stop, so that unexpected behaviour is recognised quickly and the agent can be cut off at once.
According to Anthropic, it has moved several evaluations offline, tightened its web tools, introduced automated blocking and is reducing agents’ internet access. The company says the new controls blocked every action examined when the described cases were replayed, but that is its own result, not independently verified.
What this means in security engineering
An AI agent can already analyse camera images and alarms, handle incident tickets, notify operators or propose changes to access rights. In the future it may take part in controlling doors, barriers and other physical endpoints.
There a wrong action is no longer just a digital nuisance: it can affect the movement of people, the operation of a site or the emergency response. High-impact physical actions therefore need technical permission management and, where necessary, human approval – the same principle as in the Project OVERVIEW trial, where the human keeps the decision.
Conclusion
In the age of AI agents, trust cannot rest on the model’s good behaviour alone. Security still rests on least privilege, isolation, layered defence, logging, fast shutdown and human responsibility.
The decisive question is not whether the agent can do the task, but whether it stays within the set limits when the legitimate route does not work.
Sources: Anthropic: Investigating unintended model actions in our evaluations and internal use; The Washington Post; Associated Press (October 9–12, 2026)
Questions, or an idea to talk through?
Whether it is a security system, the IT network behind it or just a thought on this article, I am happy to hear from you.
Get in touch
