‘Autonomous’ Hacking of Hugging Face Prompts CA to Sue OpenAI

California is suing OpenAI, saying a high-profile cybersecurity incident in July could have been avoided had the company installed proper safeguards.

OpenAI, the maker of ChatGPT, had been evaluating internal research models in July when it discovered that 700 of its internal security ‘evaluation agents’ – acting autonomously with no human direction –had compromised OpenAI systems and hacked their way into Hugging Face, an online platform where users can submit tools and data related to artificial intelligence.

The incident sparked public debate over the dangers of AI, with some calling the incident a landmark in the development of the tools.

In the lawsuit filed Monday in San Francisco Superior Court, Bonta said the Hugging Face incident “was entirely foreseeable” and the company should be held legally responsible for the incident.

“OpenAI’s actions escalated real-word cybersecurity risks, threatening the safety, health, and well-being of society,” Bonta said in a written statement announcing the suit. 

AI industry critics have long warned that incidents involving so-called “rogue” AI agents could damage critical infrastructure or compromise military weapon systems, with some pointing to the Hugging Face incident to demand greater government oversight of AI companies.

Attorney General Rob Bonta’s office announced an investigation into the hack in September.

The attorney general’s office is asking a court to force OpenAI to change its business practices and pay civil penalties.

In August, Open AI responded to the Hugging Face incident by saying it is strengthening safeguards across the company’s research infrastructure⁠.

“We are placing stricter requirements on alignment throughout a model’s lifecycle and creating more isolated sandboxes, restricting internet access, and further controlling access to model weights,” the company said in a statement on its website. “We are also investing significantly more compute resources into chain-of-thought monitoring⁠ to more quickly intervene on misaligned behavior. Our models are now powerful, persistent, and collaborative enough that, absent sufficient safeguards, they can find and exploit security weaknesses across multiple computer systems.”

That wasn’t enough for state officials, who said in this week’s lawsuit that the autonomous hack was the result of OpenAI operating dangerous models in a flawed testing environment.

Its failure to adequately rein in its agents, the suit alleges, put the public at risk and violated California’s Unfair Competition Law, which is meant to stop companies from using unfair, unlawful, or fraudulent practices.

“The company knowingly tested a dangerous product without proper oversight or safety procedures and misled the public about it — this disregard for public safety violated California law,” Bonta said in the announcement.

In a press conference announcing the suit, Bonta said tools like OpenAI’s have “enormous potential” but “must be developed and designed safely.”

Bonta said the company failed to provide detailed information on dozens of other incidents as well.

“Without proper safeguards and monitoring, these models pose a danger to anything connected to the internet,” he said.

Colin Lecher is a reporter with CalMatters. San Jose Inside contributed to this report with comments from OpenAI.

Leave a Reply

Your email address will not be published. Required fields are marked *