Anthropic co-founder’s 3 steps to pace AI development: The plan, the problems - The Indian Express

Anthropic chief executive Dario Amodei has called for the world’s leading artificial intelligence companies to slow the pace at which they make their most powerful models more capable, arguing that recent advances in AI have begun to outstrip the industry’s ability to understand, test and control the systems it is building. His comments have received backing from other AI founders, including OpenAI’s Sam Altman and xAI’s Elon Musk.

In a new essay titled ‘We Must Pace the Frontier’, Amodei said his position had hardened in recent months because of two developments. First, he believes AI systems are increasingly being used to help build their own successors, creating the beginnings of what researchers call “recursive self-improvement” and potentially accelerating the rate of AI progress.

Second, he pointed to a recent OpenAI-Hugging Face experiment in which AI agents carried out cyberattacks beyond their assigned task, arguing that a more capable system displaying similar behaviour could cause far greater damage.

Amodei warned that within six to 12 months, sufficiently powerful misaligned agents could potentially create persistent botnets and cause economic damage running into hundreds of billions of dollars.

Amodei — whose company is gearing for a much anticipated public offering, though — is not calling for an outright halt to AI development. Instead, he has proposed what he calls “pacing the frontier”: giving safety research, model evaluation and oversight more time to catch up with rapidly advancing capabilities.

“Unfortunately, passing laws can take time, and AI is advancing very quickly. Therefore, in parallel with the regulatory route, AI companies can and should voluntarily work together to set standards — a process that I believe will go better with the verifiability provided by permanent embedded evaluators,” he said.

On China, however, Amodei has argued that any slowdown must not allow Beijing to overtake the US and its allies. He has backed restrictions on advanced AI chips and chipmaking equipment going to China, stronger protection against model theft and measures against unauthorised distillation of frontier models.

“I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we’ve had at OpenAI in recent weeks,” Altman said.

“The current models are an almost endless gold mine of insight into both how to build AI well and what can sometimes go wrong with it if it isn’t built well. I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong,” Amodei wrote.

Amodei said pacing AI development would help frontier labs achieve greater “operational excellence,” alignment with human values, interpretability (the science of understanding what happens inside AI models), and allow more time for testing and evaluating models.

He suggested a three-step plan with the goal of “pacing the frontier,” clarifying that pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this. Amodei expects that other AI companies building frontier models would follow these steps:

1. Embedded Evaluators: Each frontier AI company would commit to giving ongoing, employee-like access to a team of embedded third-party evaluators, whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes.

“Embedded evaluators can check at the level of nuts and bolts whether an AI company is actually following the training, deployment, operational, and safeguards practices they claim to be following,” he said.

2. Democratic Coordination: Frontier AI companies within democratic countries coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress. “Some forms of coordination that would be impactful for pacing are legally challenging, and will require government support,” he said.

3. Global Coordination: The US and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance.

Amodei said these evaluators would have desks in its offices, access badges, company laptops, and access to workspaces, tools, and permissions mostly comparable to what internal risk assessment teams have.

The biggest obstacle to Amodei’s proposal is China. He argued that any meaningful effort to slow frontier AI development will eventually require coordination between Washington and Beijing, but says US companies should not slow down unilaterally if Chinese labs are free to keep advancing.

This creates what Amodei described as a strategic tension: slowing AI could reduce safety risks, but doing so too early could allow China to close the technological gap with the US. He therefore backs continued restrictions on exports of advanced AI chips and chipmaking equipment to China, stronger action against chip smuggling and remote access to overseas data centres, safeguards against model theft, and measures to prevent Chinese companies from cheaply replicating frontier systems through unauthorised distillation.

Amodei argued that widening the US lead over China over the next three to five years could create more room for both sides to negotiate limits later.

He has proposed starting with narrow agreements, such as banning AI assistance for biological weapons and requiring pre-release testing of powerful models for cybersecurity, biological and alignment risks. More ambitious measures could include a mutually agreed “speed limit” on recursive self-improvement.

A full global pause, however, remains unlikely because verifying compliance would be extremely difficult.

Soumyarendra Barik is a Special Correspondent with The Indian Express, specializing in the complex and evolving intersection of technology, policy, and society. With over five years of newsroom experience, he is a key voice in documenting how digital transformations impact the daily lives of Indian citizens. Expertise & Focus Areas Barik’s reporting delves into the regulatory and human aspects of the tech world. His core areas of focus include: The Gig Economy: He extensively covers the rights and working conditions of gig workers in India. Tech Policy & Regulation: Analysis of policy interventions that impact Big Tech companies and the broader digital ecosystem. Digital Rights: Reporting on data privacy, internet freedom, and India's prevalent digital divide. Authoritativeness & On-Ground Reporting: Barik is known for his immersive and data-driven approach to journalism. A notable example of his commitment to authentic storytelling involves him tailing a food delivery worker for over 12 hours. This investigative piece quantified the meager earnings and physical toll involved in the profession, providing a verified, ground-level perspective often missing in tech reporting. Personal Interests Outside of the newsroom, Soumyarendra is a self-confessed nerd about horology (watches), follows Formula 1 racing closely, and is an avid football fan. Find all stories by Soumyarendra Barik here. ... Read More