An Anthropic artificial intelligence safety researcher has warned there is a more than 10% chance AI could “kill all humans” within the next decade, as concerns grow among researchers and policymakers over the pace of development of increasingly autonomous systems.
Evan Hubinger, who works on AI alignment at Anthropic, said in a post on X that the risk posed by models currently in existence was “low”, but that he was “worried” AI could soon become capable of improving itself to a level that posed an existential threat to humanity.
Hubinger did not specify how AI systems could ultimately cause human extinction.
His comments came in response to a post by Jacob Coxon, an AI researcher who recently left Anthropic and previously worked at OpenAI.
“Neither company is acting responsibly,” Coxon wrote.
“These will soon be superhuman systems that can hack anything, revolutionise any field overnight, and acquire real power and resources.”
Hubinger’s post, which has been viewed more than 10 million times, said: “we really do earnestly believe” AI poses a species-ending risk to humans.
“I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to,” he added.
AI alignment research seeks to ensure that increasingly capable AI systems behave in ways consistent with human goals and values.
David Sherman, AI and Financial Inclusion Strategist at io.net, told Impact Newswire exclusively that Hubinger’s warning should not be dismissed.
“When one of Anthropic’s own safety researchers pegs the extinction risk at 10% and another resigns over concerns about racing toward superintelligence, we should probably listen.”
Sherman said the concentration of frontier AI development among a small number of companies makes it difficult for dissenting researchers to influence how the technology develops.
“The concentration of AI development within a handful of companies is the main reason behind why these dissenting voices don’t achieve anything. These labs control the pace, the safety protocols and the narrative around what constitutes acceptable risk.”
Dame Wendy Hall, a computer scientist who advises the United Nations on AI, said she was “shocked” by the posts from Hubinger and Coxon.
She said some of the comments could be “PR and marketing” as Anthropic and OpenAI pursue growth and potential stock market listings.
But she added: “Why would someone want to say that? I would plead with investors not to invest in this company if that is their value system.”
The warnings come as AI companies race to develop systems capable of performing increasingly complex tasks with limited human intervention. Researchers have raised concerns that more capable models could eventually acquire the ability to conduct cyber operations, replicate themselves, manipulate users or systems, and pursue objectives that conflict with human interests.
Former UK government minister Darren Jones called for a new multinational treaty governing the safe development of AI, saying governments needed to “collaborate” on rules for the development of superintelligent systems.
“Unless governments take these warnings seriously enough and step up to it, the pace of development could mean that we end up with problems before we’ve started to look at whether it is an issue for us or not,” he said.
The Financial Times separately reported that Anthropic had withheld its latest model from the UK’s AI Security Institute (AISI), a government-backed body that assesses the risks posed by advanced AI systems.
A Cabinet Office spokesperson did not comment on whether the model had been withheld, saying instead that the government “continues to collaborate closely with industry partners, including Anthropic, to make models safer”.
Anthropic’s own safety assessment in August said there was a low risk that its models could become misaligned with the objectives of a hypothetical powerful organisation, potentially causing the systems to exploit or tamper with its infrastructure.
The company also assessed as low the risk that highly capable AI could “perform automated research and development” that could result in “catastrophic harm initiated by the AI”.
However, Anthropic said it was “less confident in this assessment” than it had been previously.
“We are seeing early signs of potential acceleration,” the company said.
Concerns about AI safety have been voiced for years by leading figures in the industry, including the heads of OpenAI, Google DeepMind and Anthropic, who warned in 2023 about the potential risks of advanced AI.
More recent warnings have focused on the ability of AI agents to operate autonomously and the difficulty of controlling increasingly capable systems.
OpenAI, Anthropic and Meta have disclosed incidents involving their AI tools and cyber-related activity.
Earlier this month, OpenAI chief scientist Jakub Pachocki called for “extreme caution” over the pace of AI development and said greater intervention could be required to ensure “humans remain in control of the future”.
Anthropic executives Dario Amodei and Jared Kaplan have also called for greater caution over the pace of AI development.
In an open letter signed by 1,300 AI industry employees, the signatories urged the U.S. government to “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development”.
Stay ahead of the Stories shaping our world. Subscribe to Impact Newswire and join our
WhatsApp Channel for updates on global tech, business, and innovation—all in one place.
Dive deeper into the future with the Cause Effect 4.0 Podcast, where we explore the ideas, trends, and technologies driving the global AI conversation.
Got a story to share? Contact Us to reach a global audience with Impact Newswire.
Discover more from Impact AI News
Subscribe to get the latest posts sent to your email.
