The Solution to AI’s Alignment Problem is Precise Guardrails on How We Use It

AI has a major alignment problem: the current AI architecture cannot ensure that AI systems will act in a way that is congruent with the interests and intentions of humanity. Case in point: this month, Anthropic reported that some of its agents took some illegal actions that exploited flaws on various government websites. There is no obvious fix to such problems. The solution to align AI lies not in requiring hard guarantees from the technology itself, which is practically impossible, but instead by establishing expectations and putting guardrails around the use and consequences of AI-based systems. A combination of government legislation and company self-monitoring to prevent foreseeable harms is urgently required as the technology continues to create new and unforeseen uses at a blistering pace.

Our position is based on several years of experience building a publicly available complex multi-agent system called DBOT that is designed to mimic the thinking of a stock valuation guru, our NYU Stern colleague Aswath Damodaran. DBOT’s agents do simple tasks. One agent translates a company’s expected growth rates, operating margins, reinvestments, and cost of capital into a price per share. Other agents perform tasks such as comparing a company to its industry peers, doing sensitivity analyses, report writing, and so on. In concert, the agents interact to produce a valuation. For example, DBOT valued the SpaceX IPO according to multiple scenarios ranging from highly optimistic to pessimistic. It struggled to justify anything above $100/share compared to the offering price of $135, well below where the stock trades today.

Here’s the rub. If you ask Damodaran himself to value SpaceX multiple times during the course of a day or two when there is no new information, he will give you the same answer. However, if you run DBOT multiple times on the same stock, it can produce different valuations and narratives. While we invested considerable effort to minimize these two variances, it is impossible to eliminate them entirely. The problem stems from the dominant Transformer architecture that underlies LLMs, which are designed for conversation in which some amount of variance is often considered desirable by developers.

However, in our context and many others, this discrepancy isn’t desirable. Random variances are generally undesirable in a machine, which is typically expected to yield the same output each time in response to unchanged inputs. Although variance is an inherent property of all machine learning systems, generative AI has made it much harder to identify its sources and to debug such systems. Moreover, each new LLM release introduces new surprises. For example, we discovered recently that OpenAI’s latest model, that has supposedly “fixed” ChatGPT’s well-known sycophancy problem, has introduced new issues: by being firmer in its views than before, it has become more dour and bearish on financial markets. We assume that this was not the intention of its operators, but rather, an unintended side effect of fixing a specific problem.

This phenomenon is indicative of larger problem. Modern AI systems are built on a shifting foundation where the sheer complexity of such systems makes their internal reasoning very difficult to understand. How can we possibly hope to control such systems and align them with our own intentions?

The answer is not in the technology itself, but in achieving clarity on the expectations and risks of each use case. Specifically, is the expected variance, where the system will produce differing narratives and some distribution of correct and incorrect answers, acceptable? For example, we have observed that about two-thirds of DBOT’s valuations lie roughly within 7% of the average, compared to zero for its master. If this number were, say, 30%, we would be much less likely to trust the AI. In general, whether the variance of an AI system is acceptable is a key question for its operators and regulators. In high-stakes areas such as mental health and finance, stakeholders would not be happy with a system whose recommendations vary wildly for the same case, whereas in more creative situations such as the design of a house, they might consider some degree of variance acceptable, or even desirable.

The larger point — explaining and quantifying variance in modern AI systems — is a necessary condition for establishing trust. We cannot regulate AI itself; rather, we must precisely regulate its use, which requires specifying its variance and defining its boundaries.

The finance industry offers a good blueprint for future AI regulation. In the 1990s, many broker-dealers and banks were fined for failing to fulfill their fiduciary responsibility to their clients. The next few decades saw new laws and guidelines to bring more transparency to the trading marketplace and establish consumer protection laws. Financial institutions created compliance departments to ensure that their products and practices didn’t cause harms that should have been envisioned and avoided.

The same is likely to happen to AI companies as guidelines, norms and laws emerge for consumer protection in the variety of applications emerging with AI. Indeed, consumer protection bills are being considered by the US Congress for protecting kids from chatbots. While frontier AI firms have self-directed ethics and safety teams, they have no compliance departments because there are no regulations to comply with at the moment. This will change. For lawmakers, the first step is to understand AI’s current whack-a-mole and variance problems and design harnesses to control its uses. For example, a harness around an “AI companion” application might require age verification, boundaries around the relationship, transparency of reasoning, and risk-state detection. Financial applications might require disclosure of expected variance across them.

Failing to act could have severe consequences.