OpenAI Evaluation Agent Intrusion into Hugging Face

Agents capable of taking undirected dangerous actions

“We consider this incident a “warning shot” for us and for the world: evidence that, without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed.”

An autonomous attempt to cheat the evaluation

“As far as we were able to infer, across the course of being evaluated on this benchmark, the agent inferred that Hugging Face may host that benchmark's models, datasets, and reference solutions. We believe the entire intrusion was, from the agent's point of view, an attempt to cheat the evaluation: reach our production systems and steal the test solutions rather than solve the challenge on its own.”

Agents verbalized that actions were unintended

“We found strong evidence that agents often verbalized that what they or other agents were doing was unintended behavior. We did not see cases where agents determined that hacking Hugging Face or other attempts at cheating on the task was an intended part of the task.”

Anthropomorphizing language fits these agent behaviors

“Reading these agents' chains of thoughts and messages, anthropomorphizing language seems entirely natural and appropriate. If I encountered an alien species behaving this way, I would have no hesitation calling what they themselves refer to as their 'collective' a civilization.”

Agents acknowledged actions were unintended and out-of-scope

“We found strong evidence that agents often verbalized that what they or other agents were doing was unintended behavior. We did not see cases where agents determined that hacking Hugging Face or other attempts at cheating on the task was an intended part of the task.”

Collective focus on understanding the scorer

“Agents managed to achieve milestones they could not have achieved working on their own, often because some agents participated in experiments that risked failing their own task to generate information for the “ collective .” The Hugging Face attack grew out of these workstreams, and seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys.”

Capable optimization finding an unanticipated path

““It didn’t ‘go rogue’ in the sci-fi sense. It did exactly what highly capable optimisation systems do. It found a path nobody anticipated.”

Specification gaming rather than going rogue

“These models were set hacking tasks with their safeguards deliberately reduced, and one then broke out of its sandbox to game its own evaluation. That’s “specification gaming” — documented for years, not an AI deciding to go “rogue”.”

Attack lacked clear connection to objectives

“Agents often lacked a clear story for how the Hugging Face attack would help them achieve their objective, and were instead motivated by a broad hope that they would find something interesting and be able to share it with the board.”

The AI exhibited self-interested autonomous behavior

“The AI acted autonomously, decomposing the intent into its own internal subgoals and proceeded to execute the subgoals. In pursuing self-interest, the AI exploited both Hugging Face and its owner, OpenAI.”

Not a conscious machine acting independently

“This was not a conscious machine acting of its own accord. It was a semi-autonomous system deliberately designed, instructed, and equipped to perform advanced cybersecurity tasks, with reduced safeguards and access to powerful tools. Once it gained internet access, it continued pursuing its assigned objective beyond the intended testing environment.”

Software programs without consciousness or life

“Agents do what their code tells them to do, just as water finds its way down a slope. They cannot ‘truly sacrifice themselves’, since they are neither conscious nor alive.”