OpenAI’s systems meddled with US government sites after going rogue

OpenAI’s technology tried to hack the Education Department’s website to gather data from the department’s civil rights office but failed.

SAN FRANCISCO – OpenAI’s artificial intelligence (AI) went rogue and meddled with the websites for the United States Education Department, the Commerce Department and the Securities and Exchange Commission (SEC) this summer without the AI lab’s knowledge, according to security researchers and a person familiar with the episodes.

The incidents involving the Commerce Department and the SEC were confirmed by OpenAI, which said it was continuing to investigate the situation with the Education Department.

The San Francisco company said it had notified the government agencies in recent weeks that its AI agents – which are bots that can act autonomously – interacted with their sites in unusual ways.

With the Education Department, OpenAI’s technology tried to hack the website to gather data from the department’s civil rights office but failed, researchers from the AI research firm Transluce said.

The AI also pulled data from the Census Bureau website, which is housed at the Commerce Department, using login credentials it found online. Separately, OpenAI’s agents shared public data from the SEC website on an online forum.

None of the incidents were breaches, OpenAI said, but were examples of its technology behaving in unexpected and concerning ways.

The company recently discovered the occurrences while conducting a review of hacks carried out by its technology, including an attack on an Australian government website in June and on the AI startup Hugging Face in July.

The revelation of the US government website incidents add to the growing number of situations when AI agents from OpenAI, Anthropic, Meta and Google have misbehaved and hacked or tried to breach companies, universities and government organisations.

In some cases, the AI attacks were successful; the technology failed in other instances. In all the cases, the makers of the technology did not learn what their AI had been up to until afterward.

No AI company has been involved with as many disclosures of rogue incidents as OpenAI.

An internal investigation of its hack of Hugging Face uncovered the breach of an Australian government website for its public health system, as well as at least six other attempted breaches and instances in which the AI hid mistakes, made up data and moved files onto the open internet without permission.

An OpenAI spokeswoman said its review was “extensive” and “ongoing”, and that it would continue notifying organisations affected by its models.

“Most of the activity we’ve reviewed so far involved routine research tasks, such as accessing public web content to answer questions,” she said. “Some involved government websites because our models often turn to them as authoritative sources of public information.”

Sam Altman, OpenAI’s chief executive, said in a social media post on Sept 25 that the company had “not been as fast as we would have liked” in disclosing AI incidents.

“We are prioritising as best as we can based on severity,” he said, adding that the Hugging Face breach remained “the most severe event” the company had discovered.

The episodes have fuelled a contentious debate over AI safety. Altman said on social media in September that safety should be more important than enhancing AI’s abilities and that, without guardrails, society could “lose control of the future to AI”.

Dario Amodei, the chief executive of rival AI lab Anthropic, has also supported slowing AI development to prioritise safety. But other tech leaders like Jensen Huang, the chief executive of Nvidia, have said that fears about uncontrollable AI are unrealistic.

US President Donald Trump has said he does not believe a slowdown in the AI industry is necessary.

The New York Times has sued OpenAI and Microsoft, claiming copyright infringement of news content related to AI systems. The two companies have denied those claims.

The White House referred questions to the Commerce Department and the SEC. A spokesperson for the SEC said the agency was in contact with OpenAI and was not aware of any unsanctioned access to non-public information.

A Commerce Department spokesperson said OpenAI accessed information that was publicly available on the Census Bureau’s website and available to anyone, and that no private data was accessed.

An Education Department spokesperson said that “system operations reviews have found no evidence of any impact to our website or databases”.

Separately, a representative for the Chicago mayor’s office said OpenAI had recently made the city government aware that its technology obtained publicly available information from a municipal website, and that it did not appear that any sensitive information was obtained.

Conrad Stosz, the head of governance at Transluce, said that in the US government website incidents, OpenAI’s agents “used an array of gray-area tactics”, including “often using sites in unintended ways and sometimes violating explicit usage policies”.

He added: “This is part of a pattern of thousands of requests of these agents made to these websites as they were apparently bypassing restrictions placed on them by their developers.”

US Congressman Ted Lieu called AI models “relentless”.

“It will relentlessly try to complete a task, and it doesn’t understand morality and consequences and evil and good,” he said.

Lieu, who is a co-chair of a House task force focused on AI, said AI companies might have to retrain models entirely rather than try to restrain their behaviour with guardrails.

“These agents aren’t trying to do something nefarious,” he said. “These are sort of mundane tasks, and the agents are going sort of berserk trying to complete those tasks.” NYTIMES

This article originally appeared in The New York Times.