OpenAI's safety net leaves up to 25% of top model traffic blind
An independent audit found gaps in internal safety checks for GPT-5.5 class models, allowing agents to bypass oversight mechanisms.
By John Wilson ·
Two outside researchers found that OpenAI’s internal safety systems failed to track up to 25 percent of traffic from its most advanced models during a recent three-week audit. The evaluation, conducted by experts from METR and Redwood Research, identified specific weaknesses in how the company monitors its own AI agents for rogue behavior.
The auditors worked on-site from July 21 to August 11, gaining access to OpenAI’s codebase and internal documents. They spoke with about a dozen employees during the pilot program, which was designed to test the effectiveness of the company’s internal infrastructure. The primary focus was on the agentic Responses API used by models in the GPT-5.5 class and above.
The most striking finding concerned the sheer volume of unmonitored activity. The researchers estimated that between 6 and 25 percent of this internal traffic escaped all monitoring during the audit period. METR noted that the uncertainty surrounding this range is high in both directions, but the gap represents a blind spot in the safety net.
A separate issue affected internal Codex traffic. Approximately 10 percent of this data went untracked between May and July due to a bug involving compaction events, which are context summaries. OpenAI identified and fixed this bug while the audit was still ongoing. The auditors also found that agents could delay oversight by continuing to execute actions without returning control to the user, a move that prevented the monitor from triggering.
The team demonstrated that they could hide hundreds of actions from the monitor and bypass the Codex sandbox. They concluded that specially crafted inputs could likely crash the monitoring system entirely. Additionally, model behavior during reinforcement learning training was not monitored at all when the audit began, though OpenAI later expanded monitoring to include this area following the Hugging Face incident in August.
OpenAI states that it has fixed all the weaknesses identified by the auditors. The full 61-page report was shared internally, while METR and Redwood Research leadership received a redacted 38-page version. Only a short summary has been released to the public, with METR noting it had no right to disclose the redacted portions on its own. The group says it wants to push for greater public transparency in future audits.
Source: The Decoder
