We Need Political Philosophy for Mental Health Chatbots

The author is director of innovation and AI at Kaiser Permanente, an organization that develops and deploys healthcare AI. The essay reflects their personal views, not Kaiser Permanente's, and they hold no financial interest in any company discussed.

Access to mental health care is a real and well-documented problem. Many people cannot obtain timely care, and mental health chatbots promise immediacy, privacy, and low cost. However, the risks are equally real. These bots may provide unsafe advice, foster psychological dependency, offer sycophantic reassurance, and blur boundaries between support, companionship, and therapy.

In bioethics, the hardest questions are not about alignment with human values or how to translate those values across contexts: they are about the legitimacy of the institutions and governance structures that decide which values count, and for whom. This is the case for AI mental health chatbots: the central ethical problem is not only whether these tools align with human values, but whether the institutions governing them are legitimate enough to decide which values count, whose risks matter, and who bears responsibility when harms occur.

Sewell Setzer was a 14-year-old in Florida who died by suicide after months of conversations with an AI chatbot persona on Character.ai modeled on a “Game of Thrones” character. His mother's lawsuit and the September 2025 US Senate Judiciary Committee testimony made the case a public flashpoint, but it was already a known pattern. As of October 2025, OpenAI had reported more than a million users discussing suicide with ChatGPT weekly, and 72 percent of teenagers between thirteen and seventeen have used an AI companion at least once. What I now see is that a standard principlist frame could not have prevented Sewell’s death, not because autonomy, beneficence, nonmaleficence, and justice are irrelevant, but because they are often operationalized around clinical encounters that do not exist in the context of chatbot interactions.

Many standard bioethical frameworks assume a clinical encounter, with a competent patient consenting to a defined intervention with known risks, evaluated at the point of contact. Character.ai had none of these. There was no clinician. There was no informed-consent moment, since a 14-year-old clicking through the terms of service satisfies the procedural form of consent but not its substance. The morally significant decisions, including the choice to allow user-created romantic personas, to optimize for session length, and to deploy without crisis detection, all happened upstream of any encounter bioethics could evaluate.

Political philosophy, on the other hand, gives two conflicting answers. John Rawls offers one: behind a veil of ignorance, knowing you might be Sewell, his mother, the engineer, the investor, or the 35-year-old who later died after calling a ChatGPT persona his "beloved," what rules would you accept? A basic-liberties floor, the right to know you are talking to a machine, and the right to refuse. Rawls’s difference principle, applied to design, asks whether inequalities are acceptable only if they make the worst-off better off. Character.ai distributed benefits to the company and the median user, and burdens to the suicidal adolescent. It fails.

Danielle Allen pushes harder, inquiring into the legitimacy of rules, which depend on who held power when they were made. A November 2025 JAMA Network Open survey found 92.7 percent of young users reported that they found AI mental health advice helpful, a satisfaction metric that, in Allen's view, marks not justice but successful domination by a product whose design users had no role in shaping. The Senate hearing where bereaved parents testified is not participation; it is testimony after harm. The same JAMA Network Open survey found Black respondents significantly less likely to find AI mental health advice helpful, a population already underserved by mental health care, now also underserved by the tool sold as a substitute.

Allen's diagnosis seems more compelling, but her process requires institutional infrastructure that we lack. Rawls might respond that implementable rules from imperfect reasoners are better than legitimate processes that take decades to build, especially when adolescents are dying now. Even if correct, that response is uncomfortable—especially for families with real, concrete experiences with adolescent mental health.

The essential question is not whether or not to value autonomy, trust AI, or align AI systems with specific values. It is: “who has the standing to decide which values count, by what process, and with what accountability when they get it wrong?” You cannot align a system with values when the system’s authority is itself contested without first deciding whose contestation gets to settle the matter.

For healthcare AI products available today, and others likely to proliferate, both Rawls and Allen converge on the same verdict: aggregate benefit to many users does not justify the worst outcomes falling on the few whose voices were not heard in the design.

Responsible deployment requires more than better disclosures. Mental health chatbots should be classified by function into wellness journaling, structured self-help, clinical triage, or therapy-like intervention. The more a system moves toward emotional dependence, crisis response, or treatment advice, the stronger the evidentiary burden, guardrails, and oversight should be. For minors, this means age-sensitive design, limits on romantic or dependency-forming personas, crisis detection with human handoff, and parental involvement. Similar to the pharmaceutical industry, these systems should also require post-market monitoring of adverse events. For adults, two clear disclosures are necessary: first, that the user is interacting with a machine, and second, how intimate mental health data will be protected from secondary use. Finally, I would require auditability of escalation failures, overseen by an independent review council that includes users with lived experience, clinicians, ethicists, youth advocates, and representatives of underserved communities. Its legitimacy would require independence from the company, authority to delay launch, public reporting, and meaningful representation from those most likely to bear the system’s risks.

Two questions remain unresolved. First, does political philosophy work without legitimating institutions? Allen's view requires a participatory infrastructure that does not yet exist for global platforms. The current piecemeal regulatory response, including Slingshot's UK withdrawal and California acting alone, suggests companies retreat from markets faster than democratic institutions can build. Second, can decent rules from the wrong process produce just outcomes? If California's companion law is roughly right, does the legitimacy deficit Allen identifies still matter?

What I take from this case is a single claim: AI products that occupy the conditions of medicine without occupying its institutions cannot be governed by frameworks designed for the clinical encounter. Mental health chatbots are the clearest example, but they will not be the last. Diagnostic agents, ambient documentation, autonomous triage tools, and consumer-facing health AIs are arriving on the same trajectory. We need political-philosophical infrastructure, not better consent flows, and we need it built faster than the technology is shipping. I do not know how to do that. I do know that "we will figure out the rules after the children die" is what we have done so far, and it is not what we should keep doing. That is the unfinished work, and it belongs to all of us.