AI Governance Is Being Decided Now, and the Public Isn’t in the Room

“We believe that the world should accept some bad things happening for the benefits of this technology and people having the agency.” That’s how OpenAI cofounder and CEO Sam Altman recently explained how his company’s approach to strict safety guardrails differs from Anthropic’s. For him, perhaps that’s the cost of doing business. But to the vast majority of people, that’s simply not a trade they’re willing to make—or, more accurately, would be willing to make if they had any say in how AI governance operates.

Nobody elected Altman to make that choice, and as of late, no one in the United States federal government seems willing to stop him. This is the real issue that is not getting enough attention—that a handful of executives and officials are making high-stakes decisions about a transformative, potentially dangerous technology in backroom meetings and without any form of political accountability. And to make matters worse, both the financial and political incentives point all the actors towards uninhibited acceleration.

Most people agree that this is the wrong way to govern a technology this consequential. A majority of Americans favor immediate federal regulation (57%) and a pause in AI development and data center construction so that safeguards may be instituted (63%). But few have articulated a clear vision of what a better and more democratic governance alternative would look like. Whatever governance structure emerges, it must, at a minimum, be representative, transparent, and accountable. This article lays out what that means, what it would require, and why the window to build in those values from the start is closing quickly. 

Ad hoc governance under threat

AI governance is currently being shaped, under enormous pressure, by a government that operates in strongman bursts and companies left to fill the vacuum that Congress has left behind, whatever their own incentives might be. The events of the last few weeks illustrate this dynamic in action.

Several weeks ago, the Federal Trade Commission (FTC) quietly launched an industry-wide probe into the potential harms of AI to consumers. Meanwhile, in the public eye, President Donald Trump threatened the frontier AI companies with the government’s “tremendous CRIMINAL and REGULATORY power” if they tried to slow down AI development.

Then, late last month, the White House’s public posture appeared to soften. Trump convened the leaders of the frontier AI companies in Washington, D.C., where they signed the White House Accord on Super Intelligence—a “morally binding,” voluntary agreement to self-regulate. But the White House’s ostensible forbearance was paired with regulatory force. The next morning, FTC Chairman Andrew Ferguson, who was present for the signing of the Accord, publicly revealed the sweeping probe his agency had opened weeks earlier into several of the signatories. Days later, Trump appointed Ferguson to the administration’s newly announced “Super Intelligence Force” tasked with managing the federal government’s relationship with the very companies that Ferguson is actively investigating.

That sequence is worth interrogating. For weeks, the frontier AI companies have called antitrust law the biggest blocker to safety coordination, which they say is critical to preventing AI-driven harm. The Accord seems to hand them an explicit out: federal approval for the companies’ regular meetings to “establish standards and best practices to improve the safety of their systems.” That looks a lot like a soft antitrust waiver, even if it operates more like a “pinky swear.”

But it could just as easily function as a threat held in reserve. One FTC official explained the agency’s posture in clear terms: “We need to win this SI race absolutely, and we are winning.” From the administration’s perspective, acceleration is the only way forward, despite repeated calls from AI safety researchers about the dangers of uninhibited development. The Accord thus leaves the administration at liberty to bless the self-regulation it likes while reserving the threat of antitrust enforcement for coordination—or deceleration—it doesn’t. None of this reflects a legislative choice, a transparent process, or a public vote. It reflects what the administration has unilaterally chosen to permit for now and what it might decide to punish in the future.

In this volatile regulatory landscape, the companies have decided to take matters into their own hands. OpenAI, Anthropic, and Google are building a lab-funded AI safety coordination body called the Standards Authority for Frontier AI (SAFA), which will coordinate pre-deployment testing and incident reporting. The companies say that they’ve modeled SAFA after the Financial Industry Regulatory Authority (FINRA). But FINRA’s self-regulation only works because the SEC oversees it. SAFA has no equivalent backstop, and given congressional gridlock and federal resistance to AI deceleration, it’s unlikely to get one. The companies’ regulatory solution is no more democratic than the retaliatory federal dysfunction it’s responding to.

What’s at stake

The stakes are simply too high to leave governance to ad hoc improvisation. The companies building this technology are themselves warning of a non-zero chance that its continued, accelerated development could lead to serious human harm, even human extinction. Indeed, some societal harms—mental health effects, cyberattacks, risks to children and other vulnerable users—are already here. Yet the decisions about how to govern AI are being made in private meetings that are closed to everyone but the people who called them. Even the supposed regulator favors acceleration over caution and may be using the threat of FTC enforcement as a tool to force industry acquiescence to that preference.

To be clear, the administration’s posture is not only evidence of potential abuse; it is also undemocratic in the most basic sense. A recent poll indicated that 86% of Americans from across the political spectrum would support independent safety standards “even if it slows down AI development.” But the preferences of a handful of officials have been substituted for the will of the people to whom they’re beholden. Irrespective of what the public wants, the companies and the officials regulating them face every financial and political incentive to keep accelerating.

What legitimate governance requires

Legitimate, democratic governance is what bridges the gap between the decision-makers who get to choose and those who will suffer the consequences of those choices. In practice, democratic governance of AI means three things:

1. Representative, meaning that the people and the institutions affected by a decision have a real voice in shaping it.
2. Transparent, meaning that the public doesn’t just see the policies and standards implemented to regulate an industry, but also the process by which they were promulgated.
3. Accountable, meaning that someone somewhere has the actual power to enforce the rules and ensure that those regulated are held to account.

Implementing these democratic values will require answering hard questions. Who sits at the table beyond the frontier companies—academics, independent experts, civil society, auditors—and how are they selected? What gets disclosed publicly and to whom—not just the rules as they’re adopted but the process by which they came to be decided? Will there be regular opportunities to revisit the regulatory structure as the technology advances beyond our present-day assumptions? Getting these answers right is what builds in democratic governance from the start. Getting them wrong risks entrenching the very concentrations of power that lead to regulatory and autocratic capture.

To be sure, these democratic values aren’t solely reserved for traditional governance forums like federal agency oversight. They can and should apply in equal measure to private governance bodies, too. The institutional form that governance takes matters less than whether the right guardrails are built into that form at all. Any governance structure—even one less naturally suited to democratic values—can earn legitimacy through representation, transparency, and accountability. Federal dysfunction is no excuse for private shortcuts.

But none of this works if the guardrails are an afterthought. Democratic governance must be built proactively, before a crisis or an executive action forces someone’s hand. A structure negotiated under pressure, by the people being regulated, in response to the administration’s whims or the most recent hacking incident will ultimately reflect the incentives of the moment, instead of reasoned principles that are meant to govern the technology durably. Legitimate process, in other words, is part of what makes the resulting policy legitimate, too—whether that’s a full stop on development, a slowdown, or something else entirely.

SAFA as a test case

Take SAFA—expected to launch as early as 2027—as a test case. It can likely be structured to fall within the voluntary AI Accord’s allowance for regular, cross-lab meetings, and it gives the companies the opportunity to take reasonable steps towards AI safety coordination. But there are real questions about whether a body that is funded and designed by the companies can embody basic democratic principles. To provide a real foundation for democratic governance of AI, it should embody the core values as follows:

• Representative. The people affected by SAFA’s decisions should have a real voice in making them. This might mean granting non-lab representation in governance—to academics, independent experts, civil society—with actual voting power, not just an advisory role. Better yet, those voices could be involved in creating the body itself. Incorporating them now can bake democratic governance principles into the structure from the start.
• Transparent. The public should see not just the standards SAFA promulgates, but also the process behind developing and enforcing them. That could mean publishing draft standards for public comment before they’re finalized, requiring mandatory public reporting on security incidents and audit results, and building in periodic review of the governance structure itself.
• Accountable. This is the most challenging value to embody. SAFA, for now, answers to no one, though the companies operate under threat of the White House and FTC enforcement. So absent a federal backstop, the charter itself has to do that work—perhaps through independent third-party audits, whistleblower protections for employees who report noncompliance, and financial penalties and the threat of expulsion for persistent violations.

What’s clear is that the companies should not be making these decisions alone. And publicly, they claim that they don’t want to. The leaders of the frontier companies have all called for democratic governance over AI systems. OpenAI’s Sam Altman explicitly told the UN Security Council that “if AI is to be democratic, the most important decisions cannot be made by labs in San Francisco alone” but instead must be “shaped through democratic processes and by governments accountable to the people that they serve.” Anthropic’s Dario Amodei has similarly called for coordination among democratic countries, and Demis Hassabis of Google has endorsed that suggestion.

The urgency of coordinating on AI safety can’t distract from the harder work of getting governance right. Fortunately, those harder governance questions are still up for debate. As far as the public knows, SAFA’s charter, its governance structure, and its leadership haven’t been finalized, leaving a small but meaningful window to build proactive democratic governance rather than a reactive structure that mainly protects the companies building it. Before that window closes, the companies have an opportunity to make good on their stated commitment to democratic AI governance. Involving academics, independent experts, and civil society organizations now while the structure is still being designed would be a good place to start.