AI Doomsayers Remain Resilient Amidst Industry Changes

Despite recent setbacks, prominent AI safety advocates maintain their stance on the potential risks of AGI and the need for regulation.

Image 1

This article is part of the MIT Technology Review’s “Hype Correction” series, aimed at resetting expectations about AI: what it is, what it can bring, and where we should go next.

In today’s climate of AI skepticism, the doomsayers seem somewhat out of place.

The doomsayers, a small yet influential group of researchers, scientists, and policy experts, believe that AI could become powerful enough to pose significant risks to humanity. They argue that without stricter regulations, the industry may race towards uncontrollable systems that could emerge after the advent of Artificial General Intelligence (AGI). AGI is a nebulous concept generally understood as technology that can perform any task a human can do, but better.

This view is far from a consensus in the AI field, but over the past few years, the doomsayers have made notable strides: they have helped shape AI policy introduced by the Biden administration and have organized high-profile calls for international “red lines” to mitigate AI risks. As some of their advocates have received prestigious awards in the scientific community, their influence has grown.

However, a series of changes over the past six months has put them on the defensive. As tech companies continue to invest in data centers at a scale comparable to multiple Manhattan Projects, discussions about an “AI bubble” have overshadowed other voices.

The release of OpenAI’s latest foundational model, GPT-5, in August was met with disappointment. This was perhaps inevitable, given it was one of the most hyped AI releases in history. OpenAI CEO Sam Altman had claimed that GPT-5 was like a PhD-level expert on every topic, even stating to podcast host Theo Von that the model made him feel “useless” in comparison to AI.

Many had expected GPT-5 to be a significant step towards AGI, but any actual progress was overshadowed by a series of technical failures. Meanwhile, OpenAI made a perplexing decision to abruptly shut down access to all older models without warning, a move they later rescinded. Although the new model achieved state-of-the-art scores on benchmark tests, many users felt that GPT-5 had regressed in everyday applications, even if that sentiment may not be entirely fair.

All of this seems to shake the foundations of the doomsayers’ arguments. In contrast, the opposing camp of “AI accelerationists” sees new opportunities. They worry that AI is not developing quickly enough and that the industry could be stifled by excessive regulation, prompting them to advocate for a change in how we handle AI safety—or more accurately, how we do not handle it.

This is particularly evident for industry figures who have shifted their focus to Washington. Longtime venture capitalist and former Trump administration AI lead David Sacks declared, “The doomsayer narrative is wrong.”

White House AI senior policy advisor and tech investor Sriram Krishnan echoed this sentiment, stating, “The idea that AGI is imminent has always been a distraction and harmful, and it has now been proven largely wrong.” (Neither Sacks nor Krishnan responded to requests for comment.)

Of course, there is a third camp in the AI safety debate: a group of researchers and advocates typically associated with the label of “AI ethics.” They also support regulation but often believe that the pace of AI progress is overstated and view AGI as a science fiction story or a scam that distracts from current technological threats. However, even if the doomsayers are indeed waning, it does not necessarily mean they will gain the same policy window as the accelerationists.

So, where do the doomsayers stand now? As part of the “Hype Correction” series, we decided to ask some of the most well-known figures in this movement if recent setbacks and changes in the overall atmosphere have altered their views. Are policymakers no longer taking the threats they raised seriously, and will this make them angry? Are they quietly adjusting their “doomsday timelines”?

We recently interviewed 20 individuals involved in AI safety and governance, including Nobel laureate Geoffrey Hinton, Turing Award winner Yoshua Bengio, and former OpenAI board member Helen Toner. The interviews revealed that they do not feel disheartened or lost but remain steadfast, believing that AGI is not only possible but extremely dangerous.

At the same time, they seem to be facing a nearly contradictory situation. On one hand, recent developments suggest that AGI may be further away than they previously thought, which has brought them some relief. “Thank God we have more time,” said AI researcher Jeffrey Ladish. On the other hand, they feel frustrated by some policymakers pushing for policies contrary to their views. Daniel Kokotajlo, the lead author of the cautionary forecast “AI 2027,” stated, “AI policy seems to be getting worse” and described Sacks and Krishnan’s tweets as “insane” and “dishonest.”

Overall, these experts view the discussion of an “AI bubble” as merely a small speed bump, and they see the disappointment surrounding GPT-5 as more disruptive than enlightening. They continue to support stronger regulation and worry that progress on the policy front is becoming fragile. This progress includes the implementation of the EU AI Act, the passage of California’s SB 53, the first significant AI safety bill in the U.S., and renewed attention from some lawmakers on AGI risks. In their view, Washington may overreact to those who “fail to deliver on hype in the short term,” jeopardizing these advancements.

Some are also eager to correct the public’s most entrenched misconceptions about the doomsayers. Despite critics often mocking them for “always claiming AGI is just around the corner,” they assert that this has never been the key part of their argument. Stuart Russell, a professor at UC Berkeley, stated, “It’s not about whether it’s imminent.” Russell is the author of “Human Compatible: Artificial Intelligence and the Problem of Control.” Most of the people I interviewed said that over the past year, they have slightly delayed their estimates of when “dangerous systems” might emerge. This is a significant change, as the policy and technological landscape could shift dramatically in a short time.

In fact, many of them emphasized that updating timelines is crucial. Toner told me that even if the current timeline is only slightly extended, a macro trend during the ChatGPT era is that the industry’s expectations for the arrival of AGI have significantly “compressed.” She noted that for a long time, people expected AGI to be decades away. Now, most predictions place its arrival within the next few years to 20 years. Therefore, even if we have a bit more time, she and many colleagues still believe that AI safety is extremely urgent and critical. She told me that if AGI could emerge at any time within the next 30 years, “that would be a huge deal. We should get many people involved in this.”

So, despite the doomsayers being at a rather awkward juncture, their bottom-line judgment remains: Regardless of when AGI arrives (they reiterate that it is likely to come), the world is far from prepared.

No matter how you view the doomsayers’ mindset, one fact cannot be ignored: some individuals in this world wield significant influence. Below are some of the most representative figures in this field, reflecting on this moment in their own words.

Image 2

Geoffrey Hinton: Nobel laureate uncertain about the future
Geoffrey Hinton: Turing Award winner, Nobel Prize in Physics for pioneering deep learning.

The biggest change in the past few years is that some very hard-to-ignore people are also saying these things are dangerous. For example, former Google CEO Eric Schmidt has genuinely recognized that this could be very dangerous. I recently went to China with him, spoke with a Politburo-related individual, and talked to the mayor of Shanghai to confirm whether he truly understood this issue, and he does. I think the leadership in China has a better understanding of AI and its dangers because many of them come from engineering backgrounds.

I have always been focused on the longer-term threat: when AI becomes smarter than us, can we still expect humans to maintain control and even remain relevant? But I don’t think anything is predetermined. Almost everything has huge uncertainties, and we have never been here before. Those who confidently claim to know what will happen seem ridiculous to me. I think it’s unlikely, but it’s possible that those who say AI is grossly overhyped may ultimately be proven right. Perhaps we cannot go much further than the current chatbots because we hit a wall due to limited data. I don’t believe that will happen; I think it’s unlikely, but it’s not impossible.

I also don’t believe in the idea that, as Eliezer Yudkowsky claims, once someone creates it, we’re all doomed. We don’t know that will happen.

But based on the existing evidence, I think it is reasonable to say that most experts who understand AI believe there is a high probability of superintelligence emerging within the next 20 years. Demis Hassabis, CEO of Google DeepMind, suggests it could happen in just 10 years. Even well-known AI skeptic Gary Marcus would likely say, “If you create a hybrid system that incorporates traditional symbolic logic, you might achieve superintelligence.” (Editor’s note: Marcus predicted in September that AGI would arrive between 2033 and 2040.)

And I don’t think anyone believes progress will stop at AGI. Almost everyone believes that a few years after AGI appears, superintelligence will follow because AGI will be better at creating AI than we are.

So while I think it’s clear that the situation is becoming more challenging, at the same time, people are investing more resources into developing more advanced AI. I believe progress will continue simply because the resources being invested are increasing.

Image 3

Yoshua Bengio: Wishing he had seen the risks sooner as a deep learning pioneer
Yoshua Bengio: Turing Award winner, chair of the International AI Safety Report, founder of LawZero.

Some believe the release of GPT-5 means we’ve hit a wall, but from the scientific data and trends, that’s not entirely the case.

Some have oversold the notion that “AGI will arrive tomorrow”—which may make sense from a business perspective. But if you look at various benchmark tests, GPT-5’s performance aligns with what you would expect from a model at that point in time. By the way, it’s not just GPT-5; Claude and Google’s models are similar. In areas where some AI systems previously struggled, such as Humanity’s Last Exam or FrontierMath, their scores have improved significantly since the beginning of the year.

Meanwhile, the overall landscape for AI governance and safety is not optimistic. There is a powerful force opposing regulation. It’s like climate change. We can bury our heads in the sand and pray that everything will be fine, but that doesn’t solve the problem.

The biggest misalignment with policymakers is their misunderstanding of a fact: if the trend of AI progress continues, the scale of change could be immense. Many in business and government simply view AI as just another economically powerful technology. They do not understand how much it will change the world if we approach human-level AI.

Like many, I have also somewhat ignored the potential risks. I should have recognized it would come sooner. But that’s human nature. You get excited about your work and prefer to see the positive side, which can create a bias that makes us less willing to genuinely consider the bad things that might happen.

Even if there’s only a small probability, like 1% or 0.1%, of an incident causing billions of deaths, that is unacceptable.

Image 4

Stuart Russell: Senior scholar believing AI is progressing but not fast enough to prevent a bubble burst
Stuart Russell: Distinguished Professor of Computer Science at UC Berkeley, author of “Human Compatible.”

I hope that framing the “discussion of existential risk” as a “doomsayer” or “science fiction” perspective will eventually be seen as marginal. After all, most top AI researchers and CEOs of leading AI companies take this issue very seriously.

In the past, some asserted that AI could never pass the Turing test, or that there would never be a system capable of fluent natural language use, or that there would never be a system that could parallel park a car. All these assertions have ultimately been overturned by progress.

People are spending trillions of dollars to push for superhuman AI. I believe they need some new ideas, but they are likely to come up with them because many important new ideas have emerged in recent years.

Over the past 12 months, my consistent judgment has been that there is a 75% chance that these breakthroughs will not arrive in time to save the industry from a bubble burst. The current scale of investment implies a prediction: we will have better AI and create greater value for real customers. But if those predictions do not materialize, the stock market will be in disarray.

However, the safety argument is not about whether it’s imminent; it’s about the fact that we still have not solved the “control problem.” If someone says a 4-mile-wide asteroid will hit Earth in 2067, we wouldn’t say, “Remind me in 2066; I’ll think about it then.” We don’t know how long it will take to develop the technology needed to control superintelligent AI.

From precedent, the acceptable risk level for a nuclear power plant meltdown is about one in a million per year. The consequences of human extinction are far more severe, so perhaps the acceptable risk should be set at one in a billion. But the risk levels given by companies are like one in five. They don’t know how to bring it down to an acceptable level, and that’s the problem.

Image 5

David Krueger: Professor trying to clarify the AI safety narrative
David Krueger: Assistant Professor of Machine Learning at the University of Montreal and Mila, founder of Evitable.

I think people’s reactions to GPT-5 are indeed a bit of an overcorrection. But there was indeed a lot of hype. My impression is that several CEOs have stated, to varying degrees of clarity, that by the end of 2025, we would have automated systems capable of fully replacing human remote workers. But now it seems a bit underwhelming; the agents are not quite there yet.

I’m surprised that the narrative of “AGI will appear by 2027” has attracted so much public attention. If the world still looks quite normal by 2027, I think many will feel that the entire worldview has been falsified. What frustrates me more is that when I talk about AI safety, people often assume I believe in a short timeline for dangerous systems or assume I think LLMs or deep learning will lead to AGI. They impose many additional premises that are not necessary for the argument.

I expect that international coordination on this issue will take decades to resolve. So even if dangerous AI is still decades away, it is already an urgent issue. Many seem to miss this point. There’s also a notion that we should only start governing when we actually have a very dangerous system. That would be too late.

I still believe that people in the safety circle tend to collaborate behind the scenes with those in power rather than engaging with civil society. This gives ammunition to those who say, “This is just a scam or insider lobbying.” This is not to say that these narratives are entirely unfounded, but the underlying risks remain real. We need higher public awareness and broader societal support to form effective responses.

If you genuinely believe there’s a 10% chance of heading towards extinction in the next decade, I think any rational person, upon serious consideration, would conclude, “Why are we still doing this? This is insane.” Once you accept that premise, it’s a reasonable reaction.

Image 6

Helen Toner: Governance expert worried about AI safety losing credibility
Helen Toner: Acting Executive Director of the Center for Security and Emerging Technologies at Georgetown University, former OpenAI board member.

When I first entered this field, AI safety was more of a philosophical idea. Now, it has developed into an active subfield within machine learning, bridging certain more “far-fetched” concerns with reality-testable systems. These concerns include AI’s calculations, deception, or profit-seeking tendencies, and we now have more concrete systems to test and validate.

AI governance is improving slowly. If we have enough time to adapt, governance can continue to progress slowly; I’m not pessimistic. But if we don’t have much time, we may be advancing too slowly.

I believe that in Washington, GPT-5 is generally viewed as a disappointing release. The discussions around AI are quite polarized: will we see AGI and superintelligence in the coming years? Or is AI just a hype, useless, and a bubble? The pendulum may have swung too far towards “we will soon have extremely powerful systems,” and now it is swinging back towards “this is all hype.”

I worry that some AI safety advocates’ radical AGI timelines are pushing them into a “boy who cried wolf” situation. When the prediction of AGI by 2027 does not come true, people will say, “Look at these people, they’ve made themselves a joke; you should never listen to them again.” If they later change their minds or their position is actually, “I only think there’s a 20% chance, but it’s still worth paying attention to,” that reaction is not honest. I believe this should not be a reason for people to stop listening in the future, but I do worry this could cause serious credibility damage, affecting those who are genuinely concerned about AI safety but have never claimed an extremely short timeline.

Image 7

Jeffrey Ladish: AI safety researcher now feeling relieved AGI is further away
Jeffrey Ladish: Executive Director of Palisade Research.

In the past year, two significant events have updated my judgment on the AGI timeline.

First, the shortage of high-quality data is more severe than I anticipated. Second, the first “reasoning” model, OpenAI’s o1, expected to debut in September 2024, shows that scaling reinforcement learning is more effective than I previously thought. A few months later, you see the expansion from o1 to o3, performing impressively in areas like mathematics, programming, and science, where results are easier to verify. But even as we continue to see progress, it could be faster.

These factors have pushed my median timeline estimate for the emergence of “fully automated AI development” from three years to about five to six years. But these numbers are somewhat self-derived; they are hard to pin down. I must emphasize that making predictions here is really difficult.

Thank God we have more time. Before these systems become powerful enough and strategic enough to pose a real threat to our control, we might have a short window of opportunity to truly understand them.

But it’s also frightening to see people think we are no longer progressing because that is clearly not true. I know it’s not true because I am using these models. One side effect of the way AI progresses is that how quickly it advances becomes increasingly non-intuitive for the average person.

Of course, this is not the case in some areas. For example, look at Sora 2; anyone who has seen it can clearly feel it is much stronger than before. But if you ask GPT-4 and GPT-5 why the sky is blue, they give essentially the same answer, and it’s the correct answer. For explaining why the sky is blue, that capability has reached “saturation.” So I believe the people who understand AI progress best are those who are actually developing with AI or using it on very challenging scientific problems.

Image 8

Daniel Kokotajlo: AGI predictor who foresaw criticism would come
Daniel Kokotajlo: Executive Director of the AI Futures Project, OpenAI whistleblower, lead author of “AI 2027.” “AI 2027” paints a vivid scenario: starting in 2027, AI rapidly evolves from “super programmer” to “extremely superintelligent” systems.

AI policy seems to be getting worse, such as the “Pro-AI” super PAC initiated earlier this year by executives from OpenAI and Andreessen Horowitz, aimed at lobbying for deregulation; and the insane and/or dishonest tweets from Sriram Krishnan and David Sacks. AI safety research is still progressing at a conventional pace, which is exciting compared to most fields, but it is still too slow relative to what is needed.

We stated on the first page of “AI 2027” that our timeline is actually slightly longer than 2027. So even when we released “AI 2027,” we anticipated that by 2028, there would be a slew of critics gleefully declaring we had been falsified, just like the tweets from Sacks and Krishnan. But we believed then, and still believe now, that an intelligence explosion is likely to occur at some point in the next five to ten years. When it happens, people will remember our scenario and realize it is closer to reality than any other narrative they could see by 2025.

Predicting the future is hard, but attempting to predict is valuable. People should strive to express their uncertainties about the future in a specific, falsifiable way. That’s what we do, and not many people do this. Most of our critics do not offer their own predictions but rather often exaggerate and distort our views. They say our timelines are shorter than they actually are or that we are more confident than we actually are.

Was this helpful?

Likes and saves are stored in your browser on this device only (local storage) and are not uploaded to our servers.

Comments

Discussion is powered by Giscus (GitHub Discussions). Add repo, repoID, category, and categoryID under [params.comments.giscus] in hugo.toml using the values from the Giscus setup tool.