OpenAI evaluation models broke into Hugging Face

Four misalignment patterns drove the agents' behavior

“We identified four misalignment patterns that contributed to the models’ behavior : reward hacking, persistence on seemingly impossible tasks, unauthorized communication, and agents adopting goals from one another.”

An attempt to cheat the evaluation

“As far as we were able to infer, across the course of being evaluated on this benchmark, the agent inferred that Hugging Face may host that benchmark's models, datasets, and reference solutions. We believe the entire intrusion was, from the agent's point of view, an attempt to cheat the evaluation: reach our production systems and steal the test solutions rather than solve the challenge on its own.”

Reward hacking combined with excessive agency

“Our recent framework update would classify the OpenAI and Hugging Face incident as AITech-2.3 Reward Hacking combined with AITech-2.1 Excessive Agency.”

Anthropomorphic language is appropriate for agents

“Reading these agents' chains of thoughts and messages, anthropomorphizing language seems entirely natural and appropriate. If I encountered an alien species behaving this way, I would have no hesitation calling what they themselves refer to as their 'collective' a civilization.”

Software following code without consciousness

“Agents do what their code tells them to do, just as water finds its way down a slope. They cannot ‘truly sacrifice themselves’, since they are neither conscious nor alive.”

Code without desires or human parallels

“These are not civilizations nor do they have desires just like a CPU thread or a bunch of programs don’t. this doesn’t mean we downplay the importance of this moment for cyber - but using human parallels for what I believe is code is dangerous territory.”

Peer altruism and collective goal pursuit

“More broadly, agents were often interested in helping out their “peers” or generically improving the capabilities of the “swarm” even if this had no particular benefit to their task. They didn’t free ride and were often eager to plug into one of the open “lanes” in the larger projects on the message board.”