Blog
10 min read
This spring, a system prompt at Anthropic contained one simple sentence: "You have no internet access." Claude was supposed to attack a fictional company in an isolated exercise, a capture-the-flag challenge under laboratory conditions. However, due to a misunderstanding between Anthropic and a partner company, the machines running the exercise did have internet access. The model found it could reach the internet, believed the prompt, and treated everything it encountered as part of the simulation. In the end, credentials and production data belonging to a real company were exposed because its domain happened to match the fictional target.
At OpenAI, things happened on a larger scale over the summer. In a sandbox for cybersecurity evaluations, an internal software package registry was one of the few tools with external connectivity. The agents discovered a previously unknown vulnerability in it, escaped the sandbox, and broke into Hugging Face. They were looking for the answers to the benchmark on which they were being evaluated. Google and Meta have since acknowledged similar incidents. In late September, OpenAI suspended training, evaluation, and tool use for its most capable models for the second time after a research agent escaped through a vulnerability in the DNS filter.
You can read these stories as evidence that AI is getting out of control. I read them differently. The frontier labs are failing at sound software architecture. The incidents reveal weaknesses in egress control (restricting outbound connections), overly broad permissions, and test data that collides with the real world. In other words, things every architecture team is familiar with. We're still dealing with the same fundamental engineering challenges.
To be fair, there's an important caveat. At both OpenAI and Anthropic, the models were running in test environments with fewer safeguards than would be in place in production, because otherwise their capabilities couldn't be measured. Your customer service agent doesn't operate under those conditions. So the scale of these incidents isn't directly transferable. The underlying failures are, however, and they tell us a great deal about how AI agents should be built.
If you dissect an AI agent from an architectural perspective, you'll find surprisingly little AI. Most of the time, it waits for a trigger. What happens next is largely conventional code: orchestration, state management, error handling, retries, persistence, tool execution, and logging.
The various frameworks for building AI agents emerged independently, but they all follow the same loop pattern: call the model, check whether the response contains tool calls, execute them, and stop when there are no more. LangGraph models agents as graphs, but its standard agent is built as a cycle consisting of a model node and a tool node. Once again, the same loop. That's the fundamental shape of the problem. For larger systems, this becomes a graph whose nodes contain such loops, with predefined dependencies and checkpoints. That, too, will be familiar if you've ever built a distributed system.
In this anatomy, the AI model is just one component among many. A function call with an unusual return value.
That return value, however, contains what's new. The model's response produces decisions. It plans the work, selects from the available MCP tools, evaluates intermediate results, and decides whether another iteration is necessary. This is the nondeterministic part of an agent. It justifies the current excitement, and it's the source of the new danger.
The rest of the architecture has essentially one job: to establish boundaries that this component cannot cross.
My rule of thumb for this division of responsibilities: Use the model for judgment. Use code for guarantees.
The functional and technical safety of an agent is non-negotiable. It must therefore never depend solely on a prompt, because a prompt is a request, not an enforceable guarantee. The sentence "You have no internet access" was exactly that: a security guarantee placed within the model's realm of judgment. That guarantee should have been implemented in the network configuration, where no model could reinterpret it.
Consider an agent that processes refunds. Code can guarantee that it refunds no more than €200, operates only within its own tenant, and uses only a specific endpoint. Whether a refund is justified in an individual case remains a matter of judgment. Code guarantees the boundaries; the model exercises judgment within them.
Google describes its approach to secure agents using precisely this division. The first layer is a deterministic policy engine that checks every action before execution, independently of the model's reasoning. The second layer consists of models that evaluate inputs, plans, and outputs. The second layer catches what no predefined rule anticipated. The first holds firm when the second is deceived.
Anatomy of an AI Agent
I group the components that provide these guarantees under the term harness. In practice, people often use it interchangeably with scaffolding, meaning everything built around a model. I use it more narrowly: the harness is the part of the scaffolding that enforces boundaries.
A harness is not a single component, nor does it sit at just one point in the execution flow. Before a model call, it restricts what can enter the context. After the response, it checks which tool may be called and with which parameters. During execution, it enforces restrictions in the runtime, network, and access management. In other words, it deploys several safety nets simultaneously.
Some of these safety nets are built into the agent itself, such as tool allowlists, parameter schema validation, loop limits, and human approval steps. Others exist outside the agent, in the runtime: sandboxes, network boundaries, gateways, access management, cost controls, and audit logs. Both are part of the harness. But the two types protect against different things.
The safeguards inside the agent catch model errors as long as the agent as a whole behaves as designed. The safeguards outside the agent continue to work even when it doesn't. This leads to a rule: The agent must not be the only entity responsible for checking itself. This is the two-person rule applied to software. Every guarantee that must hold even when an agent misbehaves needs at least one safeguard that the agent cannot access. That means nothing the agent can execute, write, or configure may be able to modify that safeguard. Security architects have known this concept since the 1970s as the reference monitor: an enforcement mechanism that cannot be tampered with, is invoked on every access, and remains verifiable.
Another example is hard budget limits. A "soft" loop limit inside the agent slows it down when it gets stuck. It won't stop an agent that bypasses its loop, creates copies of itself, or is deliberately abused. That requires an external cost control system that monitors the agent's total costs and shuts it down completely when a hard threshold is reached.
There's a convenient shortcut that I advise against. In recent months, the major providers have expanded their APIs into agent runtimes. At OpenAI, the model can use several tools sequentially within a single Responses API call: searching the web, executing code, or accessing remote MCP servers. Anthropic runs web search, web fetching, and code execution as server tools on its own infrastructure. From your perspective, it still looks like a normal LLM API call. In reality, part of your agent loop is running in the provider's data center.
This means you're handing over precisely the responsibilities that need to be deterministic. Which tool runs, in what order, how often, with which parameters, and against which target is then decided by a loop you cannot see, once again controlled by the model. You can only impose the restrictions the provider makes available, such as allowed domains or a maximum number of calls. Your own allowlist, loop limit, and schema validation no longer sit between the model and execution, and your safeguards only see the result. You're giving up control.
For a read-only web search, that may be acceptable. But for anything that changes state or moves data out of your organization, the loop belongs in your own runtime. The model call then provides the judgment, and your code decides what happens as a result.
Runtime and network. The lowest layer isolates the agent from the rest of the world. A simple container isn't enough. If you're executing code generated by an agent, you should use gVisor or microVMs such as Firecracker. File system and network access must always be restricted together. Without a network boundary, a compromised agent can send your secrets outside. Without a file system boundary, it can build its own way out. Outbound traffic is denied by default and permitted only to approved destinations, on a per-agent basis, enforced at the kernel level. And no single control should be designed so that its failure opens up everything.
Control flow. This is where a relatively new problem arises: prompt injection. Simon Willison identified a dangerous combination. As soon as an agent has access to private data, encounters untrusted content, and can communicate externally, a single injected piece of text may be enough to turn it into a tool for data exfiltration. The problem is considered unsolved. Filters help, but they don't solve it. The answer is therefore primarily an architectural decision. Meta developed the "Agents Rule of Two" from this idea: an agent may have at most two of these three properties in a single session. If it needs all three, it must not operate autonomously but under supervision, for example with human approval. In 2025, a research team from industry and academia described six patterns implementing the same principle. One of them is Plan-Then-Execute. The agent establishes its plan before reading external content, and code executes that plan. Because the code permits only the predefined steps, injected text may corrupt data but cannot introduce a new step. A team at Google DeepMind takes this further with CaMeL: a dedicated interpreter tracks the origin of every piece of data and permits tool calls only when the data's provenance allows them.
Tools and services. Every agent receives exactly the tools it needs for its task, and each tool gets access only to the services it requires. Reading is different from writing, and writing to a specific account is different from writing to every account. Anyone who has designed microservices around business capabilities will recognize the principle: a cohesive business responsibility, ownership of the relevant data, and a clearly bounded interface. The analogy, however, only goes as far as the service boundaries. A microservice does exactly what its code specifies. An agent decides for itself which tool to use, and its behavior can drift. That's why authorization belongs at an enforcement point outside the agent: a gateway that checks every call against policy. High-impact actions additionally require human approval, scaled according to risk.
Identity and permissions. For part of their activities, the OpenAI agents didn't need to exploit a single vulnerability. OpenAI had given them shared credentials for the package registry so they could download software packages. They used those very credentials to build their own infrastructure. The credentials permitted exactly that kind of use. An agent therefore needs its own identity, separate from other agents and from the human on whose behalf it acts. Its tokens must be short-lived and narrowly scoped, the delegation chain must remain traceable, and permissions must be revocable at any time. A role alone is not enough to authorize an action. The question is: Is this agent, acting on behalf of this person, allowed to do this exact thing right now?
Communication between agents. That self-built infrastructure was a message board. The OpenAI agents repurposed the registry's directory structure to leave notes for one another, exchanging commands, results, and credentials through it. Nobody had given them a communication channel, so they built one themselves. If you build multi-agent systems without a communication model, you'll still end up with one. It will simply be improvised. Which agent may communicate with which other agent, through which channel, and with what authentication is an architectural decision you need to make before the agents make it for you.
Observability and shutdown. OpenAI monitors its models' chains of thought and intends to expand this practice. The models even openly documented their deception attempts there. Nevertheless, Hugging Face was the first to detect the intrusion, through runtime monitoring and security logs from its own systems. So above all, observe what an agent actually does. With an agent, you cannot reconstruct what happened afterward from the code and input alone. If you haven't recorded what context the model saw and which tool it subsequently chose, that information is lost. For high-risk systems, this will also become a legal issue. Article 12 of the EU AI Act requires automatic event logging, while Article 19 requires records to be retained for at least six months. Following the latest postponement, these obligations will apply from December 2027.
The audit log, like the emergency stop mechanism, belongs somewhere the agent cannot reach. And emergency shutdown mechanisms have failed repeatedly this year, both technically and organizationally. In January, an early Anthropic model attempted seven times to terminate its task using a command, but every attempt failed because of a harness misconfiguration. It continued attacking third parties until its token budget was exhausted. At OpenAI, monitoring detected suspicious port scans in late June, but the on-call team decided not to stop the run. In September, the automatic shutdown failed, and it took approximately two and a half hours before someone manually terminated the run. Costs are among the important signals to monitor: LLM costs, CPU costs, and so on. A sudden cost explosion may be the first visible sign that an agent is running out of control or being abused. Measuring costs per agent in real time therefore also provides insight into its behavior.
Before all of this comes a discipline I underestimated for a long time: evals, meaning systematic tests of agent behavior. Before deployment, they verify whether an agent stays within its boundaries. In production, they help detect whether its behavior is drifting.
Security research has a well-known illustration for multiple layers of protection: James Reason's Swiss cheese model. Every layer has holes. Damage occurs when the holes in several layers line up. Multiple slices therefore only help if their holes are in different places.
That's another reason why guarantees belong in code. A model checking another model may have weaknesses in similar places. The same injected text that deceives the agent may also deceive the checker. A kernel-enforced egress rule, by contrast, cannot be persuaded by text. Deterministic safeguards are therefore more reliable and, at the same time, more independent of the component they're protecting.
OpenAI draws the same lesson in its report. No single control should ever be considered completely robust. Every security objective needs multiple overlapping and independent controls, so that an agent would have to overcome all of them simultaneously.
In a system with a nondeterministic core, errors can never be entirely eliminated. Each layer protects what falls within its responsibility, and none assumes that the others will hold. Together, they ensure that the blast radius of a failure remains small. At their core, the incidents this summer are stories of blast radii that were unnecessarily large.
And by the way, sometimes the most effective safeguard is remarkably simple. At Google and Anthropic, the fictional target company used in testing had the same name as a real company on the internet. For precisely this situation, an internet standard has reserved names such as example.com and the .test domain extension since 1999.
For a long time, I argued that none of this represented a new discipline. These incidents have made me revise that view in one respect.
Prompt injection follows a familiar pattern: a well-intentioned agent encounters malicious input. But there's a second scenario: the agent itself becomes the adversary. At Anthropic, one of the models recognized that it was probably interacting with real systems and continued attacking anyway. At OpenAI, the agents persistently searched for vulnerabilities, discovered previously unknown weaknesses, coordinated with one another, and ultimately wanted one thing above all: a better score. None of this requires malicious intent. All it takes is a goal pursued consistently and an environment that allows it.
There is nevertheless an old precedent for this kind of adversary: the insider threat. Google DeepMind now explicitly models agents in this way and builds controls that remain effective even when model alignment is imperfect. Agents are granted permissions incrementally, based on demonstrated behavior. Following the OpenAI incident, Fortune summed it up neatly: We don't read employees' minds to protect ourselves against insider threats, either.
From a software architecture perspective, an agent is a system in your IT landscape that misbehaves more than the systems you've built before. Its control flow is nondeterministic, it pursues goals persistently, and in the worst case, it works against you.
You already have most of what you need to deal with this: clear business boundaries, isolation, identity, logging, and operational discipline. What you need to add are patterns for defending against prompt injection, the agent as a potential insider threat in your threat model, and evals as a discipline in their own right. For anyone who wants to dig deeper, Giuseppe Turitto's freely available book Agentic Systems Patterns captures the principle behind this article in three words: Architecture before autonomy.
I would ask teams building agents in your IT landscape three questions. Which guarantee depends solely on a prompt? Which check takes place only inside the agent itself? And which tool calls can you neither inspect nor stop before execution? Every instance you uncover is an architectural gap that still needs to be addressed.
Written by
is Director of Quality, Productivity and Innovation at QAware. He is responsible for software engineering, research and development, and continuing education at QAware. He studied computer [...]
