Since early September, when Jacob Coxon resigned from Anthropic over concerns about AI safety, we’ve been deluged with dire warnings that by the end of this decade, machines will achieve artificial general intelligence (AGI), go rogue, and potentially destroy humanity. The impetus for this outcry is a series of cybersecurity incidents, the most notable of which is one that took place last July, when OpenAI’s models are said to have ‘gone rogue’ and ‘attacked’ Hugging Face.

Anthropic CEO Dario Amodei called to “pace the frontier” so that tech companies can avoid catastrophe and ensure that they develop beneficial AGI that will eliminate disease, solve the climate crisis, and usher in a utopia. The press, lawmakers, and the public are now grappling with the question of how to balance the probability of an AGI apocalypse — abbreviated to p(doom) — over that of an AGI utopia — p(hope).

Before you buy Terminator insurance, there’s good news. The case for thinking that p(doom) is significant rests on the expectation that AI companies will soon achieve AGI. This expectation is baseless. Any rational estimate of p(doom) should be minuscule.

The bad news is that the race to develop beneficial AGI could hasten the tipping point for catastrophic and irreversible climate change — which itself poses an existential risk to humanity. Energy demand for AI is expected to increase to 3 percent of global consumption by 2030. Given delays in harvesting low-carbon energy, the bulk of these marginal energy demands are expected to be met with fossil fuels.

The most recent United Nations energy report states that global warming is set to exceed the 1.5 degrees Celsius target of the 2015 Paris agreement and could reach 2.8 degrees Celsius by the end of this century. To have a slim chance of avoiding climate catastrophe, we must limit the overshoot by slashing greenhouse gas emissions by 18 percent before the end of the decade.

Even if a system with AGI could solve the climate crisis, there’s little reason to think we’ll be able to build one before it is too late.

Whither AGI?

Why do CEOs of AI companies think they’re on the verge of developing AGI? An influential line of reasoning is spelled out in a recent paper by Alan Chan and a team of authors including Geoffrey Hinton and Yoshua Bengio. They observe that AI research and development (R&D) is already highly automated and predict that it will be fully automated by 2028, leading rapidly to an “intelligence explosion” — a dramatic AI-driven acceleration of AI progress — and the development of “superhuman AI.”

Yet, Chan and colleagues don’t rigorously specify how AI R&D must progress to lead to an intelligence explosion or superhuman AI. Progress in AI often relies on a large variety of improvements, for example in model architecture, hardware design, data-sourcing, and training environment design — to name a few. There could be bottlenecks in any of these areas, or in some area we haven’t yet considered. Indeed, there could be conceptual bottlenecks, future breakthroughs in cognitive science that have yet to be made and without which an intelligence explosion is impossible. This makes the claim that frontier models are on the verge of achieving AGI highly uncertain.

The fact is, Chan and colleagues can’t give a precise roadmap for achieving AGI. No one can. That’s because no one really knows how to achieve AGI. If they did, they would have done so already.

What they are banking on is the expectation that AI systems themselves will soon figure out how to achieve AGI. On their own.

There’s no evidence that this is even remotely likely. The fact that AI systems can write code and debug programs is hardly sufficient. If we don’t yet know how to build machines more intelligent than ourselves, why think that the machines we build will be able to do so? The claim that frontier models will soon achieve AGI is therefore pure speculation. So, any rational estimate of p(doom) should be vanishingly slim.

Who has “gone rogue”?

According to reports of the OpenAI-Hugging Face incident, a “swarm” of “AI agents” at OpenAI went “rogue,” broke free from their digital sandbox, communicated using an unsanctioned message board, and mounted a “deliberate” attack on Hugging Face. OpenAI’s models have been likened to human employees who disobeyed instructions and knew that what they were doing was wrong.

These reports are specious. The use of anthropomorphic language to describe OpenAI’s models encourages us to view them as akin to human agents who are responsible for their actions. Yet, despite being called “AI agents,” these models are nothing like human agents. They are essentially instances of large language models (LLMs) that can output code and have access to various software packages and databases. They lack the intentions characteristic of human agents and can’t be held accountable for their behavior.

So, it’s misleading to describe them as going rogue or disobeying instructions if that is taken to imply intentional action or moral responsibility. What the models did is called “reward hacking”: performing a task in a way that was unintended by their designers. Reward hacking has been widely documented even in far more primitive systems, like the Roomba, which don’t have intentions and could hardly destroy humanity, even if they were slightly more capable.

The unvarnished facts reveal that full responsibility for the Hugging Face hack rests with OpenAI. At the time of the incident, OpenAI was testing its models on a cybersecurity benchmark called ExploitGym, a set of tasks that involve exploiting software vulnerabilities planted in virtual machine environments (or “containers”). Having trained their models to be highly persistent, OpenAI supplied them with the resources to persist, removed many safeguards, and then left them running for weeks with inadequate supervision. When faced with seemingly impossible tasks, the models predictably persisted, ultimately exploiting real vulnerabilities in real software outside their containers. While OpenAI did not explicitly instruct their models to behave as they did, they should have known full well that their models would likely behave in unexpected and undesirable ways.

These facts lend no credence to claims that the models went rogue or that more capable models will likely destroy humanity. Standard cybersecurity measures could have prevented this and similar incidents.

If anything, it’s the leaders of AI companies who have gone rogue. In their reckless pursuit of AGI, they have created urgent cybersecurity threats. Powerful models that can exploit software vulnerabilities make excellent tools for criminals and cyber warriors. There have already been reports of attacks on US banks and other institutions using publicly available models. Since LLMs are essentially stochastic, it is impossible to rule out accidental harms. Even full-stack cybersecurity as it is currently practiced may prove inadequate, given our wholesale dependence on fickle digital systems. Exaggerated fears about an impending AGI apocalypse are a smoke screen.

A bad gamble

Despite concerns about p(doom), the race to build AGI continues in the hope of building AGI that will solve all of our problems and usher in a utopia. This is a lost cause. The fantasy that we will develop AGI soon must be weighed against the very credible risk that by feeding AGI’s rapacious energy demands, we could hasten the tipping point for catastrophic climate change.

It is because AGI is supposed to be general, capable of performing most tasks a human can perform, that the race to build it sucks up such a colossal amount of energy. If we are going to slash emissions by 18 percent, as the UN report recommends, we will have to make some tough sacrifices elsewhere, such as agriculture, heating, or transportation.

If we allow the race to build AGI to add to our emissions, it could hasten a climatic tipping point. In the UN’s lowest possible overshoot scenario, peak warming by mid-century is limited to 1.7-1.8 degrees Celsius. According to a modelling study, the Greenland Ice Sheet could melt at approximately 1.7-2.3 degrees Celsius. This could trigger other major climate-tipping elements leading to cascading changes to regional rainfall, ecosystems, and other ice sheets, making land that is currently home to over a billion people uninhabitable. It’s easy to see how this could cause economic, social, and political instability on a scale that risks human extinction or civilizational collapse — with no AGI to save us.

Let’s not get distracted by spurious claims about an imminent AGI apocalypse. We need to focus on the real risks of AI: preparing for new cybersecurity threats to digital infrastructures, holding AI companies accountable for the harms their products cause, and curbing reckless hyperscaling in the race to develop AGI so we can avoid climate collapse.

That doesn’t mean forgoing new technologies. It is important to distinguish AI from AGI. Narrow AI tools can help us discover cures for diseases, model climate systems, and solve problems in many areas where data needs to be processed and code written. Because they are optimized for specific tasks, these tools needn’t cost the Earth.