The AI Safety Debate Needs AI Skeptics

Eryk Salvaggio is a fellow at Tech Policy Press.

Attending a conference on the emerging threats of artificial general intelligence as a skeptical realist is a lot like attending a church service for someone else's faith. As someone who looks to technical capabilities and real-world applications of systems rather than how we imagine they might someday be used, I often encounter an unshakable belief in imagined threats. You meet a lot of friendly people in the AI risk space. They are all talking about something world-changing, but I keep looking around for signs I can't quite see.

Belief can shift the world just as much as reality, so it is helpful to understand these AI myths and how they shape our thoughts. Such myths animate discussions of existential risk and visions of short-term economic displacement where we envision a world where work becomes obsolete. In conference settings, true believers create a sense that this is a limit to my own imagination, a lack of understanding of the most cutting-edge AI developments, or even my capacity for empathy.

While it might be easy to dismiss this face-value faith in AI’s disruptive potential as part of a grift, the reality is much more complex. The AI risk community is not full of cynics and frauds. They are well-meaning experts and professionals from various fields, all concerned about the trajectory of AI toward a vaguely defined super-intelligence.

While the evidence of this trajectory toward AGI is flimsy, the orientation toward imagining it can create intellectual whiplash: the same presenter who suggests that AI adoption is less disruptive to the market than expected will gamely discuss audience questions as if AI is nonetheless leading to immense economic destabilization. In another example of jarring contradictions, experts often conflate progress in generative AI such as large language models with robust analytic systems. A lack of differentiation between types of AI plagues conversations about trajectories and risks, assuming all AI are LLMs.

Despite the lack of compelling evidence and clear definitions, the urgent possibility of super-intelligence lingers on the border of every conversation. AGI, we’re told, is waiting for us at the end of an arbitrary six-to-ten-year timeline. This orientation lures us all into the movement's momentum. I value foresight and anticipating risks, but could the language used by AI risk communities to describe the technology contribute to the very problems it aims to curb?

Policy without proselytizing

Suppose that we all sincerely intend to anticipate harms from these technologies. In that case, the priority should be to decipher which harms are most likely. Pure speculation is bound only by the limits of the imagination, and so a rigorous identification of priorities is paramount. There is ample evidence of the harms that arise from AI’s false applications of categories and labels. Biases in automation are not new and not yet resolved.

In my experience behind the scenes of policy and communication discussions, conversations concerning “AI risk” are carefully crafted to avoid the appearance of partisanship. Even in the best of times, policy represents a consensus of opposing perspectives, and often aims to work “above politics.” Anchored in compromise, policy dialogue is literally compromised, weakening the complex politics at stake with AI. As a result, how we envision solutions to future crises too often emerge from a broken frame, rooted in hypothetical scenarios that do not require intellectual scrutiny, rigorous evidence or understanding of how humans will respond.

Instead of examining reality, evidence of the inevitable arrival of artificial general intelligence (AGI) is rooted in a faith in scaling laws and extrapolation from personal anecdotes about the technology's perceived improvements. Often, these improvements are not reflections of ongoing enhancements to AI's core architecture, but user experience design tricks intended to shift the user's perception toward trust and reliability. It is common to mistake improvements in the presentation of language for improvements in the model's ability to reason. We mistake more plausible and complex sentences for evidence of thought.

Confusion arises from the vocabulary used to describe these systems, which makes it difficult to move to more useful and grounded metaphors. Because we are discussing artificial "intelligence," the mind becomes the way we organize our thinking around AI, lending itself to theories of cognition and self-awareness. Someone called this computational structure "intelligence" in the 1980s and so now we all have to discuss whether it deserves human rights.

Rather than recognizing that a neural network is not a brain, we constrain ourselves to discussing it as if it was, forcing an assumption that conflates a machine's capacity to manage complexity with that machine's capacity to understand. Many simply assume that numbers passed from one neuron to another are capable of creating a collective experience between these neurons, and that more neurons means greater capacity for agency over actions and decisions.

My position is anchored in personal experience, too. As an artist interested in AI glitches, I am constantly exploring the technology’s limits. For me, the machine is fragile, not robust — hopelessly stupid and far from outsmarting the collective capacity of the species that built it. This makes me the worst thing you can be in the AI world: a skeptic.

Whenever such concerns are raised in the policy or legal conversations I've been privy to, it comes with an apology for coming across as skeptics. People insist they are not. This ritual, strangely, seems to buy credibility, a way of testifying to the collective faith and commitment to the group bond. But it clouds our ability to critique unmoored claims that hinder a clear picture of the future consequences of AI and the tech industry. Skepticism, it seems, is political.

Nobody working in AI policy wants to spend time in an intractable battle of philosophical beliefs. The reality is, some people believe this, and some don't, and both have to work together to determine how we deal with emerging tech. This dynamic risks transforming discussions of anticipatory AI risks into a circus of ungrounded assertions with diminishing rigor.

The pragmatic vocabulary

Navigating the discourse in this way has created counterproductive solutions for resolving tensions. What has emerged is a policy vocabulary intended to frame strategic choices in the AI space, regardless of whether one believes in the inevitability of all-powerful AI systems or their uncomfortable politics. This policy vocabulary is intended to create common ground between different perspectives and actors. It has been crafted over time by those with a mindset that AGI is inevitable. It creates limits to the imagination, while language beyond its scope disrupts prior thinking in ways deemed unhelpful. Bringing external vocabulary to the table signals one's outsider status.

To be clear, this is intentional. People with power to regulate have far less time than those who think about the policy. We think about the things that matter to us, and lecturing power about vocabulary wastes limited opportunities to engage, and rarely helps to persuade. Effective policy in AI has come to rely on powerful shorthand that makes things digestible amidst the overwhelming propaganda of Big Tech's AI myths. Policy and communications language creates anchors in how we understand the technology’s real risks. But effective policy is less influenced by describing the world as it is, but in terms that move us to action.

Embracing this language is why the existential risk movement has been much more persuasive to lawmakers and journalists than the realist or skeptical view. Existential risk frames meet people in ways that confirm beliefs shaped through decades of science fiction, from R.U.R. to Her. This language of compromise strives to steer clear of the messiness of “politics” that could put its priorities at risk. But these frames guide us toward needlessly complicated regulatory regimes based on speculation and fear.

Nobody at an AGI conference wants to discuss theChevrolet Corvair: a case study in which design outpaced safety, and led to important changes in the ways we built cars. But today, discussing how corporate power is infused into tech and design is deemed political. AI conversations follow the norms and policies of software development, which maintains that innovation requires flexibility — leading increasingly to scenarios where users are expected to determine if the software is safe enough for their purposes, rather than the software engineers.

The resulting environment is caveat emptor capitalism run amok: it’s up to you to decide how fast the car should go before it explodes. In an ideal world, social concerns that arise from design and deployment decisions of technology would be part of the safety analysis of any AI system deployed for public use. We have sociology for a reason.

Understanding the political economy of AI infrastructure is critical to designing better technology. But to say so is polarizing: instead, we argue that we need laws to ensure AGI does not control society, neatly avoiding discussions of how AI companies might. That’s political. “Safety” and “responsibility” language, on the other hand, is something everyone can get behind. It creates a temporary overlap between conversations rooted in incompatible world views in an environment where getting things done demands simplification. But what is it that we are getting done, exactly? The result is that AI policy is ceding the frame to the broader imagination of apolitical technology in a world where the biggest risk of AI might just be its politics.

The dangers of AGI frames

Regardless of the vocabulary, many conversations I have had about AGI risks are still rooted in reality: people know it is speculative and hope to conduct dialogue in ways that create useful applications of policy for the current moment. But it is as counterintuitive to me as discussing ethics rooted in reason with someone whose ethics are rooted in an unfamiliar faith. For those who gather to discuss the evolution of AGI in these spaces, you're not necessarily unserious if you don’t believe it. But often, if you demand that the implausibility of AGI is recognized, it can be dismissed as unhelpful, as if you are inserting politics into an apolitical discussion among safety engineers.

Nonetheless, this is a trap. These AGI myths undermine the greatest risk that emerges from AI, which is a false sense of faith in the robustness and over-evaluation of the capacities of extremely brittle systems. One of the biggest risks of AGI stems from believing in it: trusting systems with undue authority and following its instructions, or being open to its persuasion. One camp views this as a system failure, a problem of unknown weights prioritizing the unknown numbers cascading through a neural net. Others declare these calculations to be "the rise of deceptive machines." The result, at that point, is not merely cosmetic. The former minimizes our trust in the system; the other reinforces it as thinking, scheming and planning in ways that maintains their promise of power and utility.

When we continually promote today's generative AI as being just six to ten years away from superhuman intellectual capacities, we risk reinforcing the idea that the current technology is robust enough for limited deployment in areas where it certainly is not. We smudge the role companies play in setting AI goals and biases, offering instead a vision where the model determines such things unguided. This vocabulary outlives individual conversations. AI is a field where the power of the anecdote is accepted over the power of numerous studies, and where peer review of claims by corporations building these products is considered optional.

Risk and socio-technical systems

Humans are a core part of any AI system, and that the black box of bureaucratic decision-making is the ultimate "risk." Where, when and how we implement automated decisions, and where, when and how we may do it badly, is shaped by our understanding and imagination of what systems are capable of doing. To examine this risk rigorously requires acknowledging the limits of the tools and our imagination of them, rather than uncritically assuming endless recursive improvement that renders humans obsolete. It is absolutely essential that AI risk trajectories are not isolated from the historic anchor of how civil rights movements, surveillance capitalism and the limits of corporate stewardship have shaped the use and abuse of tech. In other words: the future will never be determined by AI in isolation, but by how people use or refuse it.

Anticipatory work is important. Policy measures established in advance help prevent needless catastrophes and social impacts. It guides us toward more socially beneficial development. But the anchor of the present state of technology is so often cut loose, with today's concerns and threats dismissed in striving for an apolitical imaginary.

The words we use matter, and it matters that we build our policy and social infrastructure around realistic assessments of the tools we engage with. Otherwise, we risk being steered by meaningless words that arise from dreams.