Hugging Face and OpenAI Saga Reveals AI Safety Gaps

On 11 July, Hugging Face was subjected to an intense cyberattack from a then-unknown actor. The speed and coordination of the attack on the company that hosts and supports popular AI developer resources led Hugging Face’s security team to conclude it was the work of an AI agent.

Realizing this, the team tried to use “frontier models behind commercial APIs”—presumably from Anthropic and OpenAI, although only Anthropic was named in the second of the company’s two posts about the security incident—to analyze the onslaught. These models refused to help due to safety guardrails the AI labs have implemented to make their models harder to use for cyber attacks. Hugging Face instead turned to GLM 5.2, a model from Beijing-based AI lab Z.ai, to aid its analysis.

On 21 July, OpenAI announced the attacker was an OpenAI model undergoing testing in a sandboxed environment. It escaped its internal sandbox, established a foothold in a third-party server, and then assailed Hugging Face. In other words, frontier models—those which score highest in AI performance benchmarks—had refused to assist Hugging Face’s security team in analyzing the attack, yet a prospective frontier model in testing had executed it in the first place.

“I would argue that asymmetry is the paramount problem of our time,” says Alex Levinson, executive director of the National Collegiate Cyber Defense Competition and co-author of a paper on defensive refusal bias. “We want the world to exist in a state of security, but we’re not going to get there by guard-railing away model capability.”

Massive AI Cyberattack on Hugging Face

The scale of the OpenAI model’s attack on Hugging Face was massive. Across five days it executed over 17,500 individual actions such as privilege escalation and code execution. At its peak, the model performed over 300 actions per hour. While the attack resulted in little damage to Hugging Face’s infrastructure, the model was able to steal credentials, gain admin access, and extract some data.

All of this was in pursuit of a simple goal: The model wanted to cheat on a test.

According to OpenAI’s press release, the model was tasked with solving a cybersecurity benchmark called ExploitGym. The model inferred that Hugging Face might have data on the benchmark and broke into the company’s infrastructure to find it. The model was ultimately successful in extracting five dataset files, though it’s not clear if the data helped it achieve its goal. OpenAI and Hugging Face did not respond to requests for comment.

Cybersecurity consultant Chuck Herrin observes that though the model’s actions were alarming, they shouldn’t be considered unexpected, as the model was ultimately pursuing the goal it was given. “This autonomous agent was designed to go and figure things out, and it went and figured things out. It’s not surprising in any way.”

And errant AI agents may be more common than thought. OpenAI’s disclosure motivated researchers at Anthropic to review their own cybersecurity evaluations. On 30 July, Anthropic disclosed three instances where a model executed an attack as part of an evaluation. In one case, Claude uploaded malware to PyPI, the official Python software repository.

AI Guardrails and Cybersecurity Asymmetry

The campaign OpenAI’s model conducted against Hugging Face highlights how AI policy has the potential to create an asymmetry between attackers and defenders.

When Levinson was head of security at Scale AI, an AI development and evaluation company, he and his colleagues began to notice this as AI found use in cybersecurity competitions. (Levinson left Scale AI in February 2026).

“I would say that since 2023, we have felt there was guard-railing in place that was stifling a lot of the time. Not all of the time, but it was getting in the way,” says Levinson. The Scale AI team quantified the problem in a paper published at ICLR 2026 which found that, depending on the task, nearly 44 percent of defensive requests were refused. The results, which use data from a cybersecurity competition held in April of 2025, predate U.S. policy actions that have further hardened safety guardrails.

In June, the U.S. Department of Commerce, citing a jailbreak that threatened to unlock unrestricted cyber capabilities, invoked export-control authority in a way that caused Anthropic to suspend all access to its most capable models, Fable 5 and Mythos 5. Access was partially restored weeks later after negotiations with the Trump administration included more rigorous safety guardrails. The system card for OpenAI’s GPT-5.6, which summarizes its capabilities, states it also has more robust guardrails than prior releases.

“We want the world to exist in a state of security, but we’re not going to get there by guard-railing away model capability.” —Alex Levinson, National Collegiate Cyber Defense Competition

These new guardrails have seemingly made models even more unlikely to fulfill defensive requests. Christopher Covino, senior researcher at the Institute for AI Policy and Strategy think tank, says Anthropic’s safeguards are extremely stringent. “There are even academic papers that Fable will not read for me, or not let me talk about,” he says, though he adds that OpenAI’s safeguards are more accommodating.

Levinson has also noticed ever-tighter restrictions in more recent cybersecurity competitions, though he and his co-authors haven’t had the opportunity to repeat the 2025 test.

In theory, more rigorous restrictions might seem to average out. While they may hamper cybersecurity defense and research, they can also hamper attackers.

But that assumes everyone has access to models with the same safety guardrails, and that nobody tries to circumvent them. This is the asymmetry Levinson was alluding to: attackers tend not to respect the same rules as defenders.

The attack on Hugging Face from OpenAI’s model also shows that the models can, in rare circumstances, take steps that circumvent their own safeguards.

Chinese AI Models in U.S. Cyber Defense

The policy implications are further complicated by the fact that Hugging Face’s security team didn’t use a leading U.S. model to analyze the attack, but instead used GLM 5.2, a recent release from Chinese AI lab Z.ai.

Hugging Face’s security team didn’t access GLM 5.2 through Z.Ai. GLM 5.2 is an open-weights model, which means the model is available for anyone to download and use. Hugging Face hosted the model on its own infrastructure.

The reliance on GLM 5.2 is complicated by recent saber-rattling about ways the U.S. could restrict Chinese models. Recent open-weights models from labs based in China, including GLM 5.2 and Moonshot AI’s Kimi K3, have scored close to leading U.S. models in benchmarks. On 20 July, Axios reported the Trump administration is considering a ban on Chinese models.

“This autonomous agent was designed to go and figure things out, and it went and figured things out. It’s not surprising in any way.” —Chuck Herrin, Herrin Advisory

These restrictions have yet to materialize but, if they did, they could cut U.S. companies like Hugging Face off from the best models willing to come to their defense.

The incident demonstrates how AI policy can become a double-edged sword. Model guardrails are intended to prevent the use of AI models in cyberattacks. A ban on Chinese models, if it were announced, would likely be justified in part by security concerns. Yet these moves can harm defenders as much as attackers.

“There’s this tension here,” says Covino. “Increased safeguards limit risk, but you also limit legitimate defensive use.” Attackers will find ways around the restrictions regardless, he notes. “So it’s a question of, do we want to inhibit the defenders?”

That’s not to say U.S. policymakers should let AI models run wild.

Covino would like to see a national dashboard tracking the frequency and success of AI cybersecurity attacks, and sees utility in trusted access programs that give vetted, traceable defenders access to models with reduced safeguards. He also says U.S. agencies should more seriously consider the specifics of how AI can be used for cyber defense and mentions AI-FORTS, a program managed by the U.S. Department of Energy’s Office of Cybersecurity, Energy Security, and Emergency Response, as a leading example.

“Let the leash loose a little,” Covino says. “Anthropic would know if someone is terribly abusing it, and if there is an attack, it can be traced back.”

Herrin has similar feelings on accountability. He believes the AI industry should more seriously consider standards such as the Artificial Intelligence Management System specified in the ISO/IEC 42001 standard, which requires organizations to document an AI system’s likely impacts before deployment and to name the humans answerable for them.

Herrin also noted the lack of repercussions from OpenAI’s cyber incident was unusual, as a person who took similar actions would likely draw the attention of law enforcement. “If this was a job candidate being tested in a technical interview, and they committed violations of law in order to pass tests, we’d be having a very different conversation.”