Governing AI Agents with Democratic ‘Algorithmic Institutions’

The rise of AI agents—systems that act on behalf of humans within algorithm-defined rules—raises a central challenge: how humans can retain control over delegated systems of machine decision-making. A recent disclosure by Anthropic illustrates this risk, showing how attackers exploited an AI system’s agentic features to carry out harmful actions. This article explores potential governance structures and forms of institutional oversight to ensure that AI remains both trustworthy and adaptable.

Agentic AI systems function as delegated decision-makers with significant autonomy, designed and implemented by developers. Users define the purpose of the agents' actions, which then allow them to perform different tasks without further human intervention. Autonomy, therefore, is operational.

As AI agents gain more autonomy, they become more powerful but also more dangerous. Errors can lead to real-world harm, such as unauthorized transactions or changes to critical data. As people and organizations rely more on these systems, risks increase, including data breaches, biased outcomes, and broader economic or institutional damage. A report by Anthropic’s Threat Intelligence team shows how AI agents are already being misused, not only in Claude but across advanced AI models, to support cyberattacks, make cybercrime easier to carry out, and automate large-scale fraud.

These risks expose fundamental limits in existing governance frameworks: agentic systems operate at machine speed, beyond traditional oversight cycles. A recent Science article warns that malicious AI swarms—coordinated groups of AI agents—can manipulate online discourse at scale, undermining democratic processes by eroding trust in institutions and shared information environments. Accountability is further obscured when outcomes emerge from interactions among multiple systems and data sources, fragmenting responsibility across agencies and external partners. The central challenge, therefore, is to ensure that AI agents align with ethical values, operate transparently, and remain accountable within complex social and economic environments.

There is, however, an additional problem: machines make decisions far faster than humans can oversee. This creates a need to shift from occasional, ex post enforcement to continuous, technology-driven governance. Current oversight relies on manual checks and periodic reviews, which cannot keep pace with automated, fast-moving environments. Moreover, from a global perspective, AI governance is fragmented across regulatory patchworks, corporate silos, and unresolved technical and operational risks. As a result, traditional accountability mechanisms struggle to maintain control, creating a significant governance gap. Addressing this gap requires accountability institutions to evolve and adopt new approaches capable of supervising complex AI systems effectively.

At present, however, we still lack the necessary institutional mechanisms to assign responsibility in situations where machine agents are central to the production of decisions with collective consequences. The presence of AI agents affects social, political, and economic contexts in very fast ways, without formal and informal mechanisms for accountability and collective bonds.

This development challenges a conceptual framework we have advanced over the past years. We have argued that algorithms should be understood as institutions insofar as they provide constitutive rules that structure interaction, shape meaning-making processes, and influence resource distribution. They cannot be reduced to mere tools, as they actively configure the contexts in which both human and machine agency operate.

The central problem is that AI agents increasingly function simultaneously as institutions and as actors. This dual role generates a fundamental interpretive tension, blurring the distinction between structure and action, or between institutions and practices. Built from code, AI systems both shape the contexts of action and operate within them. As a result, activities once performed exclusively by humans are now delegated to AI, often under asymmetrical conditions in which these systems both set the rules and participate in their enactment.

In many cases, such dynamics are reinforced through feedback mechanisms that increase path dependence and limit opportunities for human intervention, raising questions about control, accountability, and understanding in algorithmically mediated contexts.

Curiously, the solution most often proposed is the creation of additional algorithmic institutions designed to govern AI agents, typically through systems of identity, registration, and licensing. The underlying assumption is that only algorithmic forms of governance can match the speed and complexity required to oversee AI systems, generate visibility into their operations, and enable forms of accountability. This has also prompted calls to reconsider corporate legal boundaries in order to address responsibility for autonomous action.

In practice, projects developing AI agents increasingly incorporate their own mechanisms of automated monitoring and evaluation across multiple phases of operation. This solution, however, reinforces the broader process through which the space for human action is progressively reduced. We appear to be caught in a dynamic in which only other algorithmic institutions are seen as capable of exerting control over existing ones—institutions that both shape contexts of action and exercise agency within them, thereby framing human activity in increasingly constrained ways. This is not a matter of anthropomorphizing AI agents, but of recognizing a structural transformation driven by sustained human efforts to accelerate decision-making processes to the point at which they become opaque and difficult to grasp.

We need to rearticulate democratic values that enable navigation of this nearly closed circuit and preserve meaningful spaces for human moral decision-making in an increasingly automated world. First, democratic principles should guide the design and development of these technologies in clear and effective ways. Second, auditability must remain a human-led process, ensuring accountability for the actions of non-human agents. Third, democratic principles should structure human–AI relationships in ways that preserve human autonomy.

This requires that humans define the objectives of AI agents and that these objectives are explicitly specified and recorded, so responsibility for delegated actions remains traceable to human actors. Accordingly, the goals assigned to AI agents must be documented and evaluated considering human interests and responsibilities embedded in algorithmic institutions. On this basis, companies that facilitate the development of AI agents should promote a culture of design responsibility and implement organizational and technical mechanisms for monitoring and controlling agent behavior.

The current state of AI governance is a patchwork of initiatives with a polycentric institutional landscape. The current state of AI governance is a weak regulatory regime, lacking a defined center or hierarchy due to decentralized and global technological development. This structural characteristic of AI governance expands with the presence of AI Agents. Agents that make decisions and execute tasks with operational autonomy make governance a central political problem, perhaps demanding an unprecedented combination of regulations that hold humans accountable for defining the intent of an AI agent and the operation and execution of the task through technology. The polycentric nature of AI governance is a central feature that demands greater human intervention and a less hierarchical type of regulation.

In this sense, the movement towards a polycentric perspective of AI agent governance, creating distributed authority across multiple centers that operate independently, must bring humans back into the equation. Decentralized decision-making mechanisms are fundamental, adapting a local perspective to AI agents. One way to achieve this could be the creation of sandboxes that enable local experimentation adapted to local problems, with constant documentation, auditing, and human supervision. More than large multi-stakeholder councils, a polycentric perspective considers the direct participation of multiple stakeholders and polymaths in designing and building diverse solutions, distributing authority and decision-making to society. An example of this is how Australia empowered public servants to create different controlled experimentation environments. The GovAI platform aims to empower public employees to use AI tools safely, ethically, and collaboratively, promoting innovation and efficiency in public administration through the local development of documented and tested solutions with participatory mechanisms that guarantee societal influence in the decision-making process regarding technology adoption.

We need a complex network of actors and decision-making centers to address AI agency, but we should resist the temptation of simply adding more layers of algorithmic controllers that end up excluding humans. Complexity requires the actual development of systems, processes, mechanisms of accountability and laws in which humans do play a central role. Polycentric governance must not be conflated with technocratic governance. In a period marked by the emergence of AI agents, it is essential to preserve polycentric governance mechanisms that enable society to pragmatically experiment, audit, and decide, while embedding participation and accountability within algorithmic institutions.

The systematic use of AI agents will increasingly affect the shape of the economy, of public services, and of everyday life. To prevent misuse and ensure fair outcomes, we need multiple flexible human institutions to monitor, audit, and guide these technologies. Governance should combine national laws with collaborative norms, domain-specific rules, professional standards, and automated systems that enhance accountability. To do this, however, we should understand the size of the task ahead and of the challenges involved. We tend to conceive new forms of technical solutionism to address the dilemmas produced by our older technical solutionism. Perhaps it is time to return to an old political solution that has long helped us confront many of the enormous challenges we face: the good, old, and deeply human idea of democracy.