Anthropic's Amodei proposes plan to 'slow the pace' of advancing AI capabilities

Anthropic CEO Dario Amodei published an essay on Saturday urging artificial intelligence companies to pace how quickly they improve model capabilities, a move that comes as a growing chorus of researchers have called for a coordinated slowdown.

Amodei proposed a three-step plan that he said will help temper the pace of development without "sacrificing commercial advantage or the United States' lead in AI," though he conceded that some steps may be easier to achieve than others. Anthropic is actively gearing up for what is widely expected to be a historic IPO, though the company has not officially disclosed when it plans to debut.

Anthropic has "unilaterally" committed to the first step of the plan, Amodei said, which grants third-party evaluators employee-level access to the company to verify safety practices and report incidents. The second step encourages leading AI companies within democratic countries to coordinate and establish common safety standards, and the third calls for coordination between democratic governments and authoritarian governments.

"To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this," Amodei wrote.

Amodei's essay landed after an Anthropic researcher set off a firestorm on social media this week by announcing he quit his job at the company. Jacob Coxon, who has also worked as a researcher at Anthropic's chief rival, OpenAI, said he resigned out of concern that Anthropic and OpenAI are "gambling with our lives." He said the people building AI "earnestly believe that it could kill us all by the end of the decade."

While extreme, concerns about the potential for AI to cause human extinction or other catastrophic events are not new in AI research circles. In 2023, for instance, prominent AI researchers and executives, including Amodei and OpenAI CEO Sam Altman, signed a statement that said, "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."

Amodei said Saturday that while pausing or slowing AI development has been floated since 2023, it made "little sense" to do so at that time. He said models were not powerful enough to take action in the real world at that point, and they were also not yet capable of "significant deception, manipulation, cheating, or cyberattacks."

"I continue to believe that AI can enormously improve the quality of human life. My desire to achieve these benefits is undimmed," Amodei wrote. "But the benefits will only be achieved if we build the technology in the right way, and — so long as we use the time we gain well — it is worth taking unusually deliberate care to get it right."

Amodei's essay was lauded by many industry researchers and executives on Saturday, including Altman. In a post on X, he said he agreed with Amodei that the industry needs to pace the development of advanced AI capabilities. Altman said the subject has been a "primary topic" of discussion at OpenAI in recent weeks.

"Committing to having independent evaluators with employee-like access is a great idea, and we will do the same," Altman said. "We'll have more to share soon."

Earlier this month, OpenAI's chief scientist, Jakub Pachocki, published a blog post earlier this month and warned that no AI company has "solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer." In the AI industry, alignment refers to the work by AI developers to ensure that the system behaves in accordance with human values and intentions.

Pachocki said he expects and hopes for voluntary slowdowns to become "commonplace until shared safety bars are established."

Elon Musk also expressed support for a slowdown on Saturday, writing in a post on X that, "Dario is right."

Musk, whose competing AI startup xAI was acquired by his rocket company SpaceX earlier this year, used to be a vocal critic of Anthropic. He previously said the company "hates Western Civilization," and is "doomed to become the opposite of its name," which would be misanthropic. But since Anthropic announced a major compute deal with SpaceX in May, Musk has largely changed his tune.

"Everyone I met was highly competent and cared a great deal about doing the right thing," Musk wrote at the time. "No one set off my evil detector."

Amodei wrote Saturday that he believes AI could still "dramatically raise the quality of human life," but that the risks need to be taken seriously.

"I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong," he said.