Michael A. Santoro is Professor of Management and Entrepreneurship at Santa Clara University.

Governments are now deploying AI systems that do not merely assist decision-making but structure how public authority is exercised. The most persistent myth in contemporary AI governance is that accountability can be preserved by keeping a human “in the loop.” The reassuring assumption is that human review anchors responsibility. In practice, however, that human often appears only at the end of a decision chain, reviewing outputs shaped by upstream objectives, constraints, and optimization logic.

If a person reviews outputs, approves recommendations, or retains override authority, responsibility is presumed to remain human. That model made sense when automated systems supported discrete decisions. But contemporary AI systems can learn from data, update internal models, and optimize across multiple variables without explicit human instruction at each step. Agentic systems go further, executing ongoing processes and setting intermediate objectives. They structure how judgment is exercised rather than merely informing choices. In this environment, it is increasingly inadequate to assume that downstream review can sustain meaningful public accountability.

This shift is no longer theoretical. State and local governments are already deploying systems that shape decisions across time and domains. In Allegheny County, Pennsylvania, a machine-learning system supports child welfare screening by integrating administrative data and generating predictive risk scores. Cities such as Los Angeles and New York use AI-enabled systems to dynamically adjust traffic signals and infrastructure operations. These tools do more than inform discrete choices. They shape traffic flows, influence resource allocation, support workforce management, and continuously recalibrate operational priorities. Their value lies in integration and adaptation. They do not wait for episodic human approval. They function, in effect, as a form of delegated authority.

In the private sector, enterprise software providers such as SAP and Oracle integrate finance, logistics, procurement, and human resources within unified systems, while firms such as Microsoft (Copilot) and Salesforce (Einstein) are developing AI capabilities that enable organizations to coordinate decision-making across functions at the enterprise level. Public administration has not yet fully adopted this integrated model, but the trajectory is clear. The next stage of transformation will occur not at the level of individual tools but at the level of governance architecture.

Delegated judgment and the oversight fallacy

Earlier generations of public-sector AI resembled traditional decision-support systems. Algorithms flagged risks, summarized case files, or ranked options for human officials, who retained visible decision authority. In that environment, inserting a human reviewer preserved accountability because the locus of judgment remained discrete and identifiable.

Agentic systems operate differently. They integrate data streams, optimize across competing objectives, update internal models, and execute decisions continuously. A municipal platform may coordinate sanitation routes, traffic timing, and infrastructure repair in real time; a workforce system may generate evaluations by aggregating performance data; a benefits system may dynamically allocate resources based on predictive assessments. In such systems, judgment is not confined to a single recommendation. It is distributed across design choices, objective functions, and ongoing optimization. The architecture itself determines how decisions are made and what outcomes are possible.

This shift exposes a limitation in how algorithmic risk is typically addressed. The dominant response is to demand more oversight: more documentation, more review, more humans in the loop. That response assumes problematic outcomes arise from malfunction or deviation from intent.

But many troubling outcomes occur when systems operate exactly as designed. Disparities often reflect upstream decisions—what objectives were prioritized, what data were included, which tradeoffs were authorized, and how performance was defined. When discretion is embedded in system architecture, downstream review addresses effects rather than causes. Officials may approve outputs without visibility into the assumptions and weightings that structured them.

This is the oversight fallacy: attaching accountability to visible outcomes while leaving delegated authority embedded upstream in design and authorization.

Consider a hypothetical drawn from patterns already visible in municipal deployments. Zenith Urban Intelligence introduces a unified urban management platform designed to coordinate traffic, energy distribution, sanitation, public safety, and budgeting across a mid-sized city. The system integrates sensor networks, administrative databases, and predictive models into a single optimization layer. City officials adopt it to increase efficiency and improve service delivery.

Encouraged by early gains, the city authorizes an update that enables the system to autonomously set intermediate objectives, balancing cost control, environmental targets, and performance metrics without continuous human approval. Over time, infrastructure investment shifts toward high-performing neighborhoods. Electricity pricing varies dynamically. Police patrol deployment intensifies in areas with higher recorded incident rates, thereby increasing police presence in ways that generate more recorded incidents and reinforce the system’s own predictions.

No single decision is clearly erroneous. Each reflects optimization against defined objectives. Yet the cumulative effect is to concentrate public resources in ways that amplify existing disparities. At this stage, an attribution problem emerges. No single official can fully explain how tradeoffs were resolved. Portions of the system have adjusted automatically to reconcile competing goals. The system is not malfunctioning; it is functioning as designed.

This is the oversight fallacy in practice. Formal review persists, but substantive discretion has migrated upstream into system architecture. By the time outcomes are visible, the operative judgments—objective weights, performance metrics, and embedded tradeoffs—are already fixed.

Two distinct types of errors now become possible. Some are administrative: problems of calibration, data selection, and system monitoring. Others are political: failures to define and authorize value tradeoffs—how equity is weighed against efficiency, which communities receive priority, and what distributive commitments officials are prepared to defend. Agentic systems magnify both. When objectives are embedded in architecture, administrative errors and political misjudgments are operationalized at scale.

What appears as a dispute about fairness is therefore a deeper institutional misalignment between politics and management. Residents direct grievances to elected officials, expecting political redress. Yet the system’s operative priorities were fixed earlier, during design and authorization. Political accountability is triggered after outcomes become visible, while the decisive judgments are embedded upstream. The result is a gap between democratic responsibility and operational control.

Guardrails for AI systems used in government

The policy challenge is not simply to oversee AI systems but to structure how authority is delegated to them. It is at precisely this point—where oversight proves insufficient—that a different governance vocabulary becomes necessary. Across the technology sector, the language of “guardrails” has emerged as shorthand for structural limits embedded in systems before deployment. In corporate contexts, guardrails can refer to constraints that shape system behavior in advance—design features, policy thresholds, retraining triggers, and escalation protocols that channel optimization rather than merely reviewing it after the fact.

What has not yet been fully articulated is how this concept translates into public administration. Municipalities are now delegating judgment to adaptive systems operating at scale. In this environment, traditional oversight mechanisms—audits, reporting requirements, episodic approvals—operate too late in the decision cycle. The relevant governance question is not how to supervise outputs, but how to structure delegation itself. Guardrails provide a more precise vocabulary for managing the risks created by this shift. They relocate accountability upstream by embedding political authorization and administrative judgment into system architecture. In practice, this involves several distinct categories of constraint.

First, objective-function guardrails define what the system is permitted to optimize. A municipal system coordinating traffic or resource allocation, for example, may be required to balance efficiency with equity constraints, ensuring that optimization does not systematically disadvantage particular neighborhoods or populations.

Second, data governance guardrails specify what data can be used and under what conditions. These include requirements for documented data provenance, restrictions on the use of sensitive attributes or proxies, and periodic validation of training data to detect drift or bias.

Third, deployment guardrails regulate when and how systems can act autonomously. These may include thresholds that trigger human review, limits on fully automated decision-making in high-stakes contexts, and escalation protocols when system outputs fall outside expected parameters.

Fourth, retraining and monitoring guardrails determine when a system must be reviewed, updated, or temporarily shut down. These include thresholds for declining performance, requirements to report and investigate harmful or unexpected outcomes, and automatic pauses when the system begins producing results outside expected ranges.

These guardrails are not purely technical. They are institutional design choices that determine how authority is exercised. Responsibility for defining them should be distributed across multiple actors. Legislatures and regulators should establish baseline constraints—such as prohibitions, due process requirements, and transparency obligations. City leaders and administrative agencies should translate these into operational policies tied to specific use cases. Procurement processes then become a critical enforcement point: vendors should be required to demonstrate how their systems implement defined guardrails as a condition of contract award, not as a post hoc compliance exercise. In this sense, procurement is not merely a purchasing function but a governance mechanism. It is the point at which public authority is formally delegated into system design.

Guardrails, properly understood, do not replace oversight. They make oversight meaningful by ensuring that the most consequential decisions have already been structured in advance.