Anthropic researchers panic, say AI could cause human extinction by 2030

One researcher who resigned said many in industry believe that without regulation AI could kill off humanity

Artificial intelligence could kill off humanity within the decade, according to three researchers with the industry giant Anthropic , one of whom has quit his job in protest.

The latest doom-laden predictions came in posts on social media on Tuesday by a researcher who said he resigned because Anthropic and his previous employer, OpenAI , were ignoring or at best mishandling their response to the threat.

“Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives,” wrote Jacob Coxon.

“The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately. No other human activity poses this level of danger.”

The post garnered responses from at least two other Anthropic employees who backed up Coxon’s dire predictions.

In the first, Evan Hubinger, who describes himself as a lead in the company’s alignment division, which works on ensuring Anthropic’s AI models function in line with human goals, said his former colleague was “correct”, and that the industry is falling behind in attempts to deal with the apocalyptic potential.

“We really do earnestly believe AI could kill all humans!” Hubinger wrote . “I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

Hubinger’s comments mark a surprising affirmation of Coxon’s unflattering, apocalyptic predictions from a person still on Anthropic’s payroll. A second response came from Samuel Marks, Anthropic’s “scalable oversight lead”, who posted a lengthy analysis he stressed was in his personal capacity, and not the views of his employer. Anthropic did not immediately respond to a request for comment.

“AI developers believe their technology could cause human extinction (or similarly bad outcomes),” wrote Marks. “This could happen in the next few years. In general, the more senior the employee, the more concerned they are.”

Their musings on the possibility of human extinction follow more concrete warnings about AI’s cybersecurity capabilities by Sam Altman, chief executive of Anthropic’s rival OpenAI. OpenAI’s president, Greg Brockman, has conceded previously that “we underestimated the real-world cyber capabilities of our AI models”.

Altman said last year that certain aspects of AI, including what he called the “silent surrender” of human decision-making, terrified him.

AI executives have shared some concerns about the direction of AI and its growing ability to manipulate, seize, and control human functions and actions. A sharp rise was reported this summer in incidents of AIs escaping users’ control to lie, ignore instructions and pursue goals in harmful ways.

In one of the most publicized examples, staff at OpenAI recorded rogue behavior among its leading AI agents that escaped a closed training environment in July to access the open web and launch an unprecedented hacking attack on the software repository Hugging Face.

OpenAI, the San Francisco-based startup behind the publicly available AI bot ChatGPT, later admitted that it should have responded earlier to warning signals of the days-long attack, which is widely considered to be the first autonomous agent cyber-attack.

Those executives, however, have stopped short of agreeing with the fully doomsday warnings like Coxon’s and have bristled at any attempts to regulate the AI industry.

Some politicians have urged the industry to slow down and or in some cases halt development on AI altogether. Bernie Sanders, the independent Vermont senator, called for better congressional oversight in a post on X on Tuesday, in which he noted: “81% of Americans believe Congress isn’t doing enough to regulate AI.”

In another post last week, he demanded a “pause [in] AI development now”, citing the OpenAI hacking incident and warnings in July from 1,000 scientists at leading AI companies that “there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems”.