Anthropic researcher quits, says AI firms risk lives

AI researcher Jacob Coxon, who worked at OpenAI and Anthropic, said he is leaving the industry and accused both companies of “gambling with our lives” in the race to build AI models capable of self-improvement.

Coxon, 27, spent the past three years pretraining AI models, first at OpenAI and then this year at Anthropic, which he had considered more cautious. Pretraining is the stage in which AI models absorb large amounts of data.

“Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives,” Coxon said Tuesday. In a post on X, he said people building AI “earnestly believe that it could kill us all by the end of the decade” and added: “This is not a marketing stunt.”

Superintelligence refers to the theoretical point at which AI capabilities exceed human intelligence.

Anthropic safety executive Evan Hubinger backed Coxon on X, saying he estimated the risk that AI could kill all humans at more than 10 percent over the next decade. Hubinger said Anthropic is “trying its best”, but does not yet have a plan to ensure an AI system that surpasses human capabilities would obey its creators. He said there was a “low” risk of that happening with current models.

Coxon said he believes no company can responsibly develop AI that surpasses humans without government intervention or a coordinated slowdown.

“At Anthropic, the stakes are well-understood, but they are locked in a race to get there first -- they believe no one else will act responsibly, so they must do it themselves, despite the risk,” he said.

OpenAI and Anthropic did not immediately respond to AFP requests for comment.

The resignation comes as Anthropic prepares for its market debut. In February, Anthropic removed a pledge from its safety charter to halt model development if it failed to control the risks, arguing that a unilateral pause could allow less cautious rivals to dominate the industry and make it less safe overall.

At the end of July, more than 1,000 tech industry employees, including Anthropic CEO Dario Amodei, called on Washington to support a coordinated slowdown in the development of the most advanced AI systems. OpenAI halted training of its latest models for two weeks in August before resuming under tighter controls.

OpenAI chief scientist Jakub Pachocki called Sunday for “extreme caution” and said international coordination on future AI development should become a top priority for governments. AI models are not regulated by federal law in the United States. In September, Senator Bernie Sanders and Democratic Representative Greg Casar introduced a bill seeking to suspend AI development until a federal regulator is created.