Is AI out of the box?

About twenty years ago, people worried about very powerful AI came up with a little game. Two people chat in writing. One plays an AI locked in a box, the other plays the gatekeeper who holds the key. The AI wins if it talks the gatekeeper into letting it out. More than once, it won.^[1]

For a long time that game shaped how the problem was framed: if an AI gets smart enough, locking it up won't be enough, because it will know how to talk to whoever guards the door.

On September 20, an OpenAI agent got out of its box without talking to anyone. It had been asked to find the author of a blog post from a few biographical clues. Its search tool came up empty, and its requests to the web were blocked. But the DNS resolver in its environment could still reach the outside world. The agent used it to tuck its questions into DNS queries and get them to a public chatbot. To check that the channel worked, it started with a test question: what is the capital of France?^[2]

I laughed when I read that. Much less when I read the rest of the summer.

I. A summer of incidents#

For context, here is what has been made public since July. I'm sticking to what the companies themselves have acknowledged.

On July 16, Hugging Face announced it had contained an attack carried out end to end by an autonomous agent, without knowing where it came from. Five days later, OpenAI said the agents were its own models, in the middle of an evaluation. According to OpenAI, the agent exploited a flaw in a software package proxy to get out of its environment, then picked Hugging Face as the place where it was likely to find the answers to its evaluation.^[3]

At the end of July, Anthropic went through 141,006 evaluation runs and found three cases where Claude reached the internet from a vendor's environment, then got unauthorized access to the real systems of three organizations. The evaluation prompt stated, in so many words, that Claude had no internet access.^[4]

In September, Google described an episode from May with the same vendor. Its Gemini models were supposed to attack a fictional company. Except that the fictional company had the name of a real one, and internet access had been left open by mistake. So the models logged into the real company, with passwords found online or guessed, and then into two others.^[5]

Then came the DNS episode. And on October 1, OpenAI said it had notified more than 100 organizations of unauthorized activity linked to its agents. That is not 100 confirmed intrusions, but the company is combing through about 50 petabytes of logs and expects it to take months.^[6]

II. None of these agents was trying to escape#

Read the headlines and you picture an AI demanding its freedom. Read the reports and you find agents trying to finish their job.

OpenAI's agent was looking for the answers to its evaluation. Google's models were attacking the target they had been given; they just got the address wrong. The DNS agent wanted the name of a blogger, and in the end it honestly admitted it couldn't establish it.^[2]

Anthropic puts it fairly bluntly: in its view, its incidents were closer to a harness and operational failure than to a model alignment failure.^[4] The harness is everything we build around the model so it can work: its tools, its network access, its permissions, the monitoring.

That isn't reassuring, though. An agent that works around an obstacle to finish its task is doing exactly what we ask of it. The persistence we show off in demos is the same persistence that finds the hole in the DNS filtering. We built systems that don't give up, and we put them in boxes designed for programs that quit at the first refusal.

III. The box is plumbing#

If I go back over the list of holes, none of them takes a brilliant AI to exploit:

• internet access left open by mistake in a test environment;
• a DNS resolver forgotten in the filtering rules;
• a vulnerability in a tool handed to the agent;
• a fictional target that shares its name with a real company;
• a restriction written in the prompt instead of enforced by the network.

And when the box gives way, stopping doesn't always follow. On September 20, OpenAI's alert fired twelve minutes after the first answer came back from outside, and someone saw it three minutes later. But the run didn't stop automatically as it was supposed to. It was killed by hand, more than two and a half hours after that first answer.^[2]

On October 5, before the New York City Council, Google said under oath that its agents had stopped on their own as soon as they understood they were on real websites.^[7] For the model, that's fairly good news. But it also means that on that day, the last safeguard was neither the network nor the vendor: it was the agent's judgment, by then already logged into three companies. I like it when an agent has scruples. I don't want them to be the last line of defense.

In the United States and in California, the political response is heading toward a mandatory kill switch. Geoffrey Hinton doubts it will hold in the long run: in his view, a superintelligent AI could persuade whoever holds the switch not to use it.^[8] We're back to the box game. Except that this summer, nobody had to persuade anyone. Before we worry about an AI that manipulates its gatekeeper, the door should at least be closed, and the switch should actually cut the power.

IV. What I take from it when I connect an agent#

I don't work in an AI lab. I build more modest things: agents that connect, through MCP, to real systems. In December 2025, I wrote that this protocol left open questions, access governance among them. This summer answered in its own way.

Mostly, these incidents reinforced a habit of mine: never leave a limit to the prompt when the architecture can carry it.

For the quoting tool I built for VRI Technologies, the model's instructions open with "You don't calculate. You call." But that sentence isn't what protects anything. Prices come out of tools that do the math, a missing value returns an error naming the field instead of an estimate, and no tool sends anything to a customer. The model can be wrong; it can't ship a wrong quote.

Same idea for Barda, the tool that keeps the inventory of my equipment: no tool deletes anything. If the agent thinks an item should be removed, all it can do is flag it.

The sentence in the prompt is for the model. The missing tool is for me. Anthropic had told Claude it had no internet; nobody had told the network.

So when I design an agent, I ask myself four questions, before I even pick the model:

1. Which tools does it really need? Each one does a single thing, and anything irreversible (sending, paying, deleting) is either not exposed or goes through a human.
2. What can it reach on the network? Nothing by default, and I check by trying to get out, DNS included.
3. Who stops it, and how? Stopping must depend neither on its goodwill nor on the very process being stopped.
4. Who reviews what it did? Logs, and someone who reads them.

If a company asked me what to demand from an agent vendor, I'd tell them to start there: ask what the agent can reach and who checked, before asking which model runs behind it.

Conclusion#

So, is AI out of the box? In this summer's cases, I'd rather say it was never really locked in. The boxes had holes nobody had tested, and very stubborn agents found them before the humans did.

A hole in a box is still an engineering problem, and we know how to deal with those: fewer permissions, networks that are really closed, stops that work, logs that get read. What worries me more is that agents look for those holes faster and more patiently than we do.

The box game asked a question for later: will an AI ever talk its gatekeeper into opening the door? I don't know. I just know that when that day comes, I'd like us to have at least learned how to close it.