Who Should Pace the Frontier? Not Dario Amodei

Over the weekend, Anthropic CEO Dario Amodei published a new essay, titled “We Must Pace the Frontier.” Amodei argues for slowing down the pace of AI development, and offers a three-part plan for industry regulation. OpenAI CEO Sam Altman and SpaceX CEO Elon Musk have both responded favorably to his proposal.

For years, I’ve made a habit of dissecting manifestos and missives from tech CEOs. I shared my early reactions to Amodei’s on Bluesky. There are a few points that deserve further elaboration. Chief among them: Who should pace the frontier? Should the AI industry be permitted to police and regulate itself?

Let me start positively. Amodei writes that “A race to the bottom, spurred by commercial incentives, can make [the risks of AI doing serious harm] more acute.” Yes, absolutely.

Nothing about the AI future is inevitable. As I’ve contended for years, the arc of any emerging technology bends toward money. Regulations shape market incentives and social behaviors, rewarding certain technological uses and punishing others. The current status quo, featuring a handful of unencumbered tech companies in an outright race to win market share and attract investor cash, is a recipe for the worst possible outcomes.

But it’s also worth noting that Amodei’s company is currently winning that AI race. Anthropic is preparing for its own IPO, boasting about a $30 trillion total addressable market, and scraping together two consecutive quarters of profitability, all while beating OpenAI at its own game. Even if we grant that Amodei’s concern is genuine (and I think it is), it clearly takes a back seat to his commercial ambitions. He is not worried enough to delay going public.

Yet one really cannot neutrally read this essay. If you believe Amodei is the good AI guy, then you will read it as an act of responsible thought-leadership. If you believe he is another huckster, you will read it as more hucksterdom.

For the moment, let’s grant Amodei temporary status as the less-bad-AI-guy. I believe he has a conscience. I believe he believes the things he is saying. And the same simply cannot be said of Sam Altman or Elon Musk. But just as Effective Altruism begins from good intentions, and then eventually arrives at ‘the most effective form of altruism is buying ourselves a castle,’ those good intentions are easily warped in utterly predictable ways.

At a high enough level of abstraction, Amodei’s suggestions all sound appealing. He proposes (1) instituting mandatory third-party evaluation and monitoring at all frontier labs, (2) developing a (self-) regulatory regime that all AI labs in democratic countries commit to, and (3) bilateral coordination with China on AI model development. These are all potentially good ideas. But the details matter quite a bit. Amodei’s scheme for pacing the frontier amounts to a series of proposals that all benefit Anthropic. These proposals are all subject to revision if-and-when they get in Anthropic’s way. We should read him accordingly.

And the fundamental problem is this: Dario Amodei believes we ought to trust the AI industry to effectively regulate itself. He has consistently called for regulation, but only of the sort that gives Anthropic a comparative advantage within the industry. He believes the government should mandate the same testing and auditing practices across the industry that Anthropic has already instituted. But, as Lina Khan recently pointed out, the Federal Trade Commission could be much more aggressive in enforcing regulations and holding the industry liable for the harms caused by its products. It is a mistake to let the AI industry shape the contours of its own regulatory system, even if we grant the good intentions of its leaders. We know from history exactly how that will turn out.

Take his proposal for embedded evaluators. Amodei writes:

1. Embedded Evaluators. Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR), whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes. This is the key step for verifiability of any pacing commitments, and has precedent in the banking industry, which sometimes involves regulatory “supervisors” embedded along with employees. Anthropic is unilaterally committing to this step now. We intend this to be part of a broader push to redouble efforts on our safety and alignment work.

Amodei writes that this “has precedent in the banking industry, which sometimes involves regulatory “supervisors” embedded along with employees.” But the thing about the banking industry is that these regulatory “supervisors” are only there because the government requires them. Without the force of government oversight (and the looming threat of draconian fines), this sort of monitoring doesn’t work.

Imagine how this would actually work in practice: these third-party evaluators are meant to have desks and office space at Anthropic. They’ll examine the same models, write their own reports, and offer a “second opinion free of commercial incentives.” Who pays their salary, and how competitive is that salary? I keep seeing advertisements from Anthropic for entry-level employees in the range of $400K. Top-level engineers are attracting eight- or nine-figure salaries from the company’s competitors. Either these third-party monitors are going to make similar salaries — provided by the very same funders who are deeply invested in Anthropic — or they are going to be little more than glorified apprentices. If you have a desk at Anthropic, and you are analyzing the same data as Anthropic employees, and you have the same skillset as Anthropic employees, and you are paid 1/5th what Anthropic employees are paid, then you will eventually apply for a job at Anthropic. Third-party, independent evaluators will either become a revolving door to the industry, or else they’ll be utterly outmatched.

The scenario that Amodei is describing is less like the banking industry post-2008 and more like Arthur Andersen’s relationship to Enron. In the absence of government enforcement, independent monitoring gets captured by industry, particularly when big money is involved.

SE Gyges has discussed the problems with METR, specifically. METR, Gyges writes, is not a meaningful check on Anthropic. “METR is not meaningfully independent, is not sufficiently staffed, and has no authority over Anthropic that cannot be revoked at Anthropic’s discretion.” Suggesting that embedding METR into Anthropic would be a meaningful check on Anthropic is so suspicious that it looks like an attempt to evade oversight and to sabotage attempts at oversight in general.”

The other awkward part of the METR example is that AI safety is a very small world. The industry is still young enough that it is indistinguishable from an online subculture. Practically everyone who works in the AI safety industry spent time on the LessWrong discussion boards, where they marinated in the same cultural artifacts, bonded over shared perspectives, and built up a mountain of shared assumptions. They live in the same group houses and hang out at the same parties. It is more than a little weird, and the furthest thing from a dispassionate third-party regulator. To a first approximation, the AI safety industry is one giant polycule.

Amodei also repeatedly claims that “Anthropic has long supported sensible and targeted AI regulation.” Color me underwhelmed. Every company in history has been in favor of regulation, so long as they get to pick the regulators and the regulatory details. The cigarette industry also favored “sensible and targeted” regulations, so long as it got to set the rules for itself. You do not get a trophy for pressuring government to adopt your preferred regulations.

This matters quite a bit, because what Dario Amodei wants to do with the extra time provided by a slowdown in frontier model development amounts to a laundry list of tasks — “operational excellence,” “alignment,” “interpretability,” "testing and evaluation” — that Anthropic previously committed to, but is now handling sloppily. Amodei is asking for the luxury of moving slower and burning less money on compute, all without altering the trajectory of his industry in the slightest. He calls for a “coordinated pacing strategy [that] would give frontier AI developers time to do this vital work without sacrificing commercial advantage or the United States’ lead in AI.” Stated another way, his company is currently winning the AI race. He worries that the race might be disastrous for humanity. So his solution is to freeze the standings while we work this all out.

His proposal for “pacing within democracies” is premised entirely on the belief that the AI industry can be trusted to regulate itself. This is where he should lose whatever benefit of the doubt he started with. The frontier labs are not going to “solve alignment,” ensuring that future AIs properly serve humanity, anytime soon. (This may be impossible, and is certainly improbable given the economic incentives.) The other, more viable path is to establish and enforce a robust regulatory system with clear liabilities and punitive damages. If the law stated clearly and forcefully that a frontier lab will be found liable for punitive damages if a model escapes its sandbox and hacks websites, and if that law was consistently enforced, then the frontier labs would figure out how to build better sandboxes.

Amodei is never going to support robust enforcement schemes, because doing so would be bad for his company. And, ultimately, Amodei’s argument amounts to surely there must be some way to steer the AI future to maximally benefit society and also maximize value for my investors!

We ought to create regulatory friction to alter the trajectory of the AI future. An unencumbered trillion-dollar race among corporate giants is indeed the worst possible way to build out this technology. We ought to demand transparency regimes and enact serious regulatory frameworks. (Just not the ones the industry itself selects.) And yes, certainly, we should foster bilateral conversations with China.

What Amodei fails to recognize is that his morals and his self-interest are not, and cannot be, aligned. If the risks from the AI labs are indeed this serious, then we ought not rely solely on the AI developers to solve them.

We should take Dario Amodei’s writing just seriously enough to stop entrusting him with so much power.