Researchers Detail How AI Systems Can Enable Authoritarianism

Technologies cannot typically be accurately described as intrinsically authoritarian. However, as many Tech Policy Press readers know, technology can, and is, used to advance authoritarian goals. The various forms of AI are no exception. In the US, there has been recent scrutiny of tools being developed for ICE to identify deportation targets, and longstanding issues have been identified with predictive policing systems that purport to identify where crime is likely to occur or who is likely to commit a crime. Emerging from a fully-authoritarian context, China’s DeepSeek LLM has been shown to adhere to its homeland’s communications policy regarding Taiwan and the 1989 events at Tiananmen Square.

As illustrated by the policing examples above, authoritarianism-enabling AI goes beyond LLMs and frontier models, and includes systems that have been in place for some time. What then makes AI of particular interest in this context? In a new preprint study, “From Democracies to Autocracies: How AI Systems Enable Authoritarianism by Design,” policy researchers Jeba Sania, Marta Ziosi, and Fazl Barez draw together common threads amongst AI systems of concern, noting that they include “specific features such as statistical learning, extensive data collection and analysis, and black-box algorithmic decision-making that resists explainability and thus evades effective oversight.”

Having identified this area of specific concern, the authors developed and applied a systematic approach to analyzing how AI systems can “exacerbate” authoritarian practices, “extending reach, reducing cost, and obscuring accountability.”

The study

To investigate the problem of AI-enabled authoritarianism holistically, the paper adopts a practice-oriented model of authoritarianism rather than restricting the scope to systems built or implemented in fully authoritarian regimes. In keeping with the premises mentioned above, the team sought to identify “enabling features” throughout the full product lifecycle, as opposed to merely looking at the systems’ capabilities in abstract. They identified ten distinct stages, from ideation and initial public communications to oversight and current state of deployment, with a particular emphasis on the influence of stakeholders at many of these stages.

The enabling features were classified according to a framework of six characteristics, developed from a review of the literature about authoritarianism:

• “Coercive Capacity is defined as the extent to which force and other forms of compulsion are monopolized by executive powers.
• “Accountability Erosion is defined as the removal or weakening of formal or informal oversight mechanisms.
• “Symbolic Safeguards are defined as accountability processes without genuine or meaningful enforcement that can be exploited or co-opted, and thus become largely symbolic.
• “Information Control is defined as the monopolization and manipulation of information channels, including invasive monitoring and surveillance practices.
• “Anticipatory Repression is defined as preemptive acts of coercion, intimidation, and harassment to maintain control.
• “Boundary Control is defined as drawing exclusionary social and political categories that violate the political participation and human rights of specific populations, particularly minorities.”

Six systems were selected for the study, developed and deployed in different countries with different political contexts, including both democracies and authoritarian regimes. Each of the systems was identified as: using AI; fitting into one of three domains (mass surveillance, predictive policing, and information manipulation); and with sufficient reliable data accessible, including investigative reporting, public sources, and leaked information. They include, as quoted from the paper:

• FlockSafety Automated License Plate Recognition (ALPR): Neighborhood and business-level license plate recognition camera system developed by a private US company, FlockSafety.
• Live Facial Recognition (LFR) Vans: Real-time facial recognition vans used by 13 local UK police departments for suspect identification.
• SlimmeCheck: Welfare fraud detection system developed by the Amsterdam municipality government's Work, Participation, and Income (WPI) department.
• Lavender: Militant identification system developed by the Israeli Defense Force's Unit 8200.
• Integrated Joint Operations Platform (IJOP): Aggregated policing system used to monitor and identify security threats in Xinjiang, developed by the Xinjiang Public Security Bureau.
• Sfera: Biometric (facial recognition) payment system used in the Moscow Metro developed by Russian private-public partnerships.

Findings

The authors present their full set of enabling features for all systems in the paper, organized by characteristic of authoritarian practice. For example: under “Accountability Erosion,” the authors identify “Over-reliance on Automated Decision-Making” in the Lavender system when it allows its military users latitude to proceed with its recommendations without significant human review, thus limiting pathways to contest the decisions. Or, under “Anticipatory Repression,” the “Co-optation of Administrative Systems” is found in Sfera, which the paper says has been used by the police to identify those who participated in earlier anti-government protests so that they could be arrested in advance of planned actions.

Many of the features identified, including both of those in the examples above, are broad enough to not be restricted to a particular characteristic or lifecycle stage. Rather they reappear in various forms throughout the findings, as can be observed from the summary table reproduced here.

The researchers identified a number of patterns in the data. On the most basic level, there were the most frequently occurring feature clusters: “a) over-reliance on automation, b) mass data integration and centralization, c) co-optation and integration of administrative and military infrastructures, d) oversight and regulatory gaps, e) weak user compliance, and f) harms to protected groups.” Beyond that, the following stands out:

• Distinct risks emerged not only from centralization but also from fragmentation. Whereas centralizing systems might escape checks on power from the broader network of processes and stakeholders, fragmented systems can make it harder to identify accountable parties and maintain safeguards.
• Mirroring common moves towards authoritarianism outside of the technology sphere, a “preceding security crisis” was influential in the introduction of four of the systems.
• Four of the six systems were initially developed by private companies or through public-private partnerships.
• Information gaps were most prevalent in the design, development and testing stages, and, for IJOP, Sfera and Lavender, regarding oversight and safeguards.

Take aways

As one might expect from the framing of the study, the researchers highlight that authoritarian-enabling features can be identified across the product lifecycle and across political contexts, covering many of the same characteristics (though it does appear that, intuitively enough, Information Control and Anticipatory Repression were enabled in the Chinese and Russian systems, whereas Symbolic Safeguards were more of a concern in systems from the largely-democratic US and Israel).

It follows from this that interventions to prevent authoritarian outcomes must also take place at multiple stages of system development, implementation and usage. For example, safeguards must be embedded in the technical systems, not reliant on operational restrictions, especially as systems are subject to repurposing after their initial deployment. The testing stage is also highlighted as a promising point of intervention, as long as suitable performance thresholds (e.g. for accuracy and bias) have been established.

Transparency is another critical building block for accountability, and varied greatly between systems. The researchers noted that they had access to more detail on the technical stack of China’s IJOP than Flock’s license plate reader! The presence of private companies with corporate secrecy concerns plays a significant role here. Finally, legislation and regulation can constitute an important safeguard, so long as it is sufficiently clear and carries significant penalties.

The findings are somewhat tentative due to the gaps in available information on the systems mentioned above. And, as the authors acknowledge, only a few systems were reviewed, so the conclusions cannot be thought of as comprehensive or generalizable across the scope of AI systems used by government actors across the world.

However, this contribution to the investigation of how AI can enable authoritarian practices in any country may be particularly valuable for its design of a holistic, systematic approach to analyzing individual systems. It is also yet another set of evidence of the gaps in safeguards that defenders of democracy in industry, government and civil society need to be taking steps to remedy now.