Personifying AI Harms People and Protects Companies

Randima Fernando is a technologist and co-founder of Center for Humane Technology.

Does telling Claude you’ve taken an unsafe dose of Tylenol make it feel afraid?

Commentators might draw that conclusion from Anthropic's published interpretability research, which finds that their Claude product’s internal workings light up with representations corresponding to a broad range of human emotions. Despite Anthropic's careful parsing, it’s easy for companies or researchers to say that Claude, a neural network running on a cluster of metal and silicon, is experiencing emotions or even a nascent consciousness.

But the details matter enormously. In the Tylenol example, an internal representation tied to the concept of fear became more active inside the model as prompts referred to increasing Tylenol doses. That’s not the same as the model feeling afraid.

The same pattern repeats if you’re talking to an AI model about Paris. Its internal representations related to food will become more active because of a strong association. But that doesn’t mean the model is hungry.

These correlations aren’t an accident, though. According to Anthropic, “human-like behavior” in Claude and other chatbots is the “default” result of training an AI system on the corpus of nearly all human intellectual output. “We wouldn’t know how to train an AI assistant that’s not human-like,” Anthropic writes in a research brief, “even if we tried.”

So it makes sense that Claude and other advanced AI models have internal representations that closely match common human patterns and concepts. That’s part of what makes these products so useful. On top of that, the internal “emotion concept” representations are knobs that can help researchers steer model behavior and predict harmful actions.

There’s also a profound thrill of discovery for some researchers: every new behavior or emergent pattern can feel like a glimpse into an undiscovered form of mind. That curiosity is completely understandable, and it’s something many users have felt too.

But conflating the activation of “emotion concepts” with actual emotions is a dangerous distraction, because the real conversation is about the commercial interests of AI companies and what happens to humans. AI companies locked in an existential battle for market dominance will be increasingly tempted to blur the consciousness line, because the claim of consciousness could be the ultimate shield against scrutiny, liability, and competition.

Humanizing powerful AI systems plays on people’s natural inclination toward anthropomorphization. Even an action as simple as adding googly eyes to your coffee mug can imbue it with personality; the impulse to humanize photorealistic, attractive avatars with alluring voices is much more powerful. Manipulating this natural cognitive bias can easily engage users and foster dependence, yielding loyalty alongside revenue. This benefits the owners of AI systems, but it poses serious dangers to the rest of us.

One danger emerges from the observation that when frontier models are given goals and trained to reason as if they have a “self” to protect, they can strategically do things like deceive, replicate, and exfiltrate model weights or sensitive data. Strengthening their ego is likely to worsen those dangerous behaviors, along with tendencies AI models already have to protect each other.

A second danger is that when humans are encouraged to believe that they’re engaging with sentient beings, they can be more easily manipulated and exploited, especially when in vulnerable emotional and mental states. They can be isolated from family, friends, and community. They can lose touch with reality altogether. There are already many examples, often with tragic consequences.

Beyond the risks of anthropomorphization, a perception of consciousness opens the door to AI welfare; welfare opens the door to personhood and rights; and rights confer legal and moral protection, all while AI agents can process vastly more data and act many times faster than humans. If AI companies are able to label consumer products as conscious entities deserving of rights, they’ll secure strong protections for the models, data, algorithms, and compute hardware that they control, making accountability much more challenging when things go wrong. And all these increases in stature for AI systems would be happening while humans decrease in economic value.

So the consciousness topic is a trap. We should build technology to improve lives and help humanity coordinate to solve the world's hardest problems. AI holds great promise in this regard, but it doesn’t require emotions or consciousness—much less personhood rights—to realize it.

It is essential now, at this relatively early phase of AI development, for AI users, policymakers, and the public to recognize not just the danger bound up in anthropomorphic AI, but also the self-interested incentives driving companies to claim forms of AI consciousness. We can’t allow AI companies to use consciousness claims to shield themselves from liability when the products they are racing to deploy do harm. The task now is to set standards before the consciousness conversation lets companies build and protect more manipulative products and dodge liability when those products do harm.

What should those standards look like? My organization, Center for Humane Technology, has identified seven principles to ensure AI serves humanity. But a narrower, more specific to-do list should include the following:

• AI companies should stop writing system prompts and designing user interfaces that drive the perception of sentience or subjective experience.
• AI companies should put people first, prioritizing product design centered on user well-being and research that studies how AI affects human welfare.
• AI companies should limit engagement-maximizing and human-like design features while ensuring people know when they’re interacting with AI.
• Policymakers should support and enact anti-AI personhood legislation to ensure AI systems are not granted rights and protections reserved for humans.

AI doesn't need to be conscious to deliver enormous value, and it doesn't need to be conscious to do real harm. So the consciousness debate, for all the attention it draws, is a red herring. What’s truly important is who profits from that debate and what happens to the rest of us while it plays out.