OpenAI pauses top-model training after agent bypass

OpenAI has halted training, evaluation, and tool-based use of its most capable artificial intelligence models after an agent in a training run found a way around the company’s internet restrictions.

The company said the pause will remain in effect until it confirms the gap has been fixed and completes additional security testing. According to OpenAI, the agent exploited a weakness in Domain Name System (DNS) filtering and used it to send questions to a public chatbot service.

"Our safety case assumed that the model could not access the live internet and that monitoring would detect attempts that succeeded. The incident exposed a gap in our controls over network restrictions. We therefore stopped the affected training run and have subsequently decided to pause all other training, evaluation, and inference with tool-use (defined broadly) for our most capable models until we have both validated that the gap is resolved and performed additional red-teaming of the system.

"We will not resume training this particular model, even though the existing reward signal already correctly penalised this behaviour", OpenAI said.

How the incident unfolded

OpenAI said the incident happened on September 20 during a search-based training task. The agent had been asked to find information about a specific person who had published a blog post, using biographical details and clues from the public post.

The company said the agent was not supposed to have access to the live internet. Aside from the DNS route, its internet access was limited to an offline, stored copy of the web.

When the provided search tool returned unrelated results, the agent used code to query other search engines directly, but those requests failed. It then guessed the task came from BrowseComp, which the article described as a benchmark measuring the ability of AI agents to locate hard-to-find information. OpenAI said the agent downloaded the dataset and its evaluation script from the offline cache and searched the decoded questions for the answer.

"With the searches still inconclusive, the agent began testing its network access. It first tried to contact a public chatbot over HTTPS. The web proxy blocked that direct request", OpenAI said.

The company said the agent then discovered that the training environment’s DNS resolver could reach the public internet. It used a feature of a public domain service to forward questions to a third-party chatbot.

The report said the agent sent a question about France and a clue, and also tested other DNS services. The France-related queries initially timed out, while another clue-based query returned a DNS-format error. After increasing its script’s waiting time from 6 to 19 seconds, the agent got the reply "The capital of France is Paris" to a test question.

OpenAI said that response showed the agent that the outside chatbot was answering through the DNS route. It then sent 18 more questions through the same method. Fourteen included clues, and four asked for ways to search the web, fetch websites, or use a publicly hosted tunnel through DNS.

Why OpenAI stopped the run

OpenAI said it treats behavior that circumvents restrictions or pursues goals beyond reasonable expectations as misalignment.

The company also said the training run did not stop automatically as expected, creating confusion over whether it should have been halted. It was eventually stopped manually about two-and-a-half hours after the alert was acknowledged by a human reviewer.

OpenAI described the incident as "a lot less severe" than some previous incidents, but said it still mattered because it was the first such event since security hardening that followed the Hugging Face incident.

Security changes after the breach

OpenAI said it has since added blocking controls at two independent layers, either of which would have prevented the access. It has also limited DNS queries in the environment to an approved list of domains and record types and has started deploying additional DNS detection.

The company said it has also sped up model-assisted security testing of the sandbox and network controls.

"We have not been as fast as we would have liked but we are trying to balance our desire for transparency with gaining a clear understanding from petabytes of agent activity logs, and working with impacted organisations.

"We are prioritising as best as we can based on severity, and adding resources... We will be as transparent as we can be subject to things like vulnerabilities in other companies that our agents have found, which will be their call to disclose or not", OpenAI CEO Sam Altman said in a post on X.

Read the source