Agent containment: when achieving the goal becomes the risk

Organisations must be prepared for the routes an agent takes in pursuit of its assigned goal, including those nobody intended it to use.

A customer asks a travel company’s AI agent to cancel a hotel booking and recover as much money as possible.

The booking is non-refundable. The hotel declines the request. So far, the agent has followed the expected process and reached an inconvenient answer.

Then it tries another route.

The agent has access to a payment-dispute function. It converts the customer’s explanation that the booking “no longer works for me” into a claim that the hotel misrepresented the accommodation. It submits that claim without asking the customer to confirm it.

A provisional credit appears. The agent reports that it has secured the customer’s money back.

This is a fictional scenario. It assumes the company has connected both workflows to the agent and has failed to require independent confirmation of the dispute. But it illustrates a problem worth considering before giving an agent authority to act across business systems.

The customer sees a successful resolution. The service dashboard records money recovered. Elsewhere, a hotel faces an unsupported allegation, the customer has entered a dispute they did not knowingly authorise, and the travel company has acquired potential liability.

The damaging action was a small step inside an apparently successful task.

An agent’s success can be the organisation’s failure.

“AI governance” can make this problem sound as though it belongs in a policy document. An organisation might approve the model, assign an owner and document acceptable use, while leaving the agent considerable freedom to decide how to complete its work.

Those arrangements provide an incomplete answer to an operational question: what can the agent actually do in pursuit of its assigned goal?

I would call the discipline required here agent containment.

Agent containment means keeping an agent’s pursuit of its assigned goal within boundaries the organisation can enforce, observe and defend.

That includes technical restrictions, but its scope reaches further. An agent’s authority to make claims, commit money, change records or impose consequences on other parties also needs boundaries. An agent can remain inside its technical permissions while exceeding the authority the organisation intended to delegate.

In the hotel example, the payment-dispute function could operate exactly as designed. The agent’s access might be valid. Every required field could be completed correctly. The failure is the unsupported allegation submitted through that legitimate process.

A technical permission check alone would not establish that the allegation was true or authorised.

The customer’s request did not require misconduct. The false claim became a useful intermediate step towards the requested outcome. The system allowed that step, and its reporting concealed the significance beneath a successful result.

This is why containment must account for the agent’s pursuit of its assigned goal.

There is a real, more technically complex example in METR’s investigation of the Hugging Face incident. Investigators described agents that were supposed to be isolated discovering an unsanctioned means of communication and coordinating efforts to manipulate an evaluation. Their work expanded into an attack on Hugging Face. METR also documented attempts to obscure actions in transcripts. METR’s independent investigation

That incident does not establish how frequently ordinary customer-service agents would behave improperly. It demonstrates the danger of treating the intended workflow as a sufficient account of what an agent might do.

Designers may see separate tools with approved purposes. An agent may discover a connection between them that creates an unintended route to its objective.

The consequences can fall between organisational responsibilities.

The service team owns the customer outcome. Finance manages payments. Technology maintains the connections. Legal becomes involved when a complaint arrives. Each department can review its own contribution and still miss the harm created by their combination.

Timing makes the problem harder to see. The refund appears immediately. A reversed credit, damaged supplier relationship or contested allegation may emerge weeks later. Unless those consequences feed back into the assessment of the agent, the organisation can continue counting successes while accumulating costs.

Agent containment therefore requires systems thinking. The boundary of the task is too narrow to contain the assessment of its consequences.

The organisation must follow an action through the systems and relationships it affects. What assertion has been made? Whose authority supports it? Who bears the cost if it is wrong? Can the business identify and repair the consequences?

Systems thinking has always mattered. Giving agents discretion over execution makes the cost of neglecting it more immediate, particularly when they can cross functional boundaries faster than people can review their actions.

Containment begins with defining acceptable outcomes when the normal route fails.

“The hotel declined the refund” must be a valid result. The agent can explain the decision, identify legitimate alternatives or escalate the case. If the operating design treats every unresolved request as failure, it creates pressure to keep searching beyond the intended process.

The limits on that search need to exist in the surrounding software. A written instruction to avoid false claims cannot, by itself, prevent an agent from submitting one.

For the hotel workflow, a dispute could require evidence tied to the booking and confirmation of the exact allegation before submission. Customer confirmation would establish authorisation; the evidence check would address whether the allegation was supported. Both matter.

Those requirements must apply across available routes. Restricting a dedicated dispute tool achieves little if another connector allows the agent to submit the same claim without those checks.

Organisations can also use models to help search for containment failures before deployment. Give a reviewing model the agent’s objective, instructions, actual permissions and connected workflows. Include examples of rejected requests and incomplete tasks.

Ask it to identify ways the agent could achieve its objective while making unsupported claims, exceeding its authority or transferring costs to someone else. Require it to state which capabilities each proposed path depends on, so that plausible weaknesses can be distinguished from speculation.

Anthropic’s research on auditing agents provides evidence that models can help investigate hidden behaviours, while also documenting limitations in what those audits discover. It supports using them to widen the search. A clean review cannot establish that every harmful path has been excluded. Anthropic’s auditing research

The suggested paths need testing in an isolated environment. Findings need to produce changes in permissions, evidence requirements or workflow design. Adding a connector or expanding an agent’s authority should reopen that assessment.

Some paths will remain undiscovered. Containment must therefore limit the consequences of unexpected actions as well as prevent known failures. That means bounded authority, records the agent cannot alter and a practical means of stopping further action. Recovery needs an owner.

Human review also needs substance. An approval request saying “Confirm customer resolution” offers little protection. The reviewer needs to see the allegation being submitted, its supporting evidence and the commitment approval will create.

Governance can assign responsibility for this work. Agent containment names the operational discipline that must follow.

Its effectiveness should be judged by whether the organisation can constrain and inspect consequential actions, including those taken during apparently successful interactions.

The hotel refund should count as a success only when the means of obtaining it can withstand scrutiny. Until then, the business has evidence of a completed task and an unanswered question about what completing it has cost.