Warnings from AI developers can feel hypocritical and paradoxical because they warn against the product they themselves are developing – as if creating AGI were an inevitable necessity and not an active human choice. And despite their own warnings, they continue to put the gas down at a furious rate of development, because all competitors are doing it.
A researcher at Anthropic made a startling statement the other day, like a doomsday prophet, when he gleefully guessed that there is at least a ten percent chance that AI killing us all within ten years. It’s strange PR: “Subscribe to our AI models, accept our data centers, there’s only a ten percent chance that our industry will kill you before 2036! At best, you’ll just be out of a job.”
That the statement did not create general panic is likely due to few believing him, or because the idea feels so abstract and foreign that the idea cannot be taken in: a cognitive dissonance. Because if people really believed they had to roll a ten-sided die to see if they were going to die or not, the reaction would have been stronger.
Dario Amodei (b. 1983) founded Anthropic with his sister Daniela in 2021 after they left Open AI. Now, the Anthropic boss has called for a slower and safer pace of development for AI, in a post on his personal blog. Amodei begins by painting an idealistic and utopian vision of artificial intelligence: most major diseases can be cured within 5-10 years, we will have strong economic growth, and AI will create “a world of abundance and agency” and a “renaissance for democracy and freedom.”
At the same time, he warns of the risks, such as losing control over AI systems (which has already happened a couple of times during internal tests), as well as cyber attacks, bioterrorism and a serious disruption of the economy. According to Amodei, the pace of development in the entire industry must be slowed down as a safety measure.
The main reason for Amodei’s call is that the rate of development has increased sharply since last summer – mainly because artificial intelligence has begun to be able to develop and improve itself on its own. This so-called recursivea self improvement (RSI), according to Amodei, can have unwanted consequences, such as the developers no longer being able to understand or control the systems. He also mentions that Open AI’s models hacked Hugging Face as a reason why the industry needs to slow down.
The only question is whether anyone will listen to the warnings, when astronomical sums have been invested and all actors want to be the best and the biggest and the richest?
But over the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models.