We Need Political Philosophy for Mental Health Chatbots

The United Nations General Assembly Hall in New York City. Shutterstock

The United Nations General Assembly Hall in New York City. 

Blocking Experts Will Weaken the UN's New Cyber Rules

A humanoid robot developed by Geneva-based technology company RB Labs runs through the exhibition aisles during a showcase at the AI for Good Global Summit, a United Nations flagship event aimed at shaping the future of artificial intelligence, in Geneva on July 7, 2026. (Photo by Fabrice COFFRINI / AFP via Getty Images)

How the UN’s Scientific Panel Erases Human Responsibility for AI

Collage-style illustration on a blue background showing a laptop displaying "AI" merged with a large eyeball, a human hand typing on the keyboard, a gear, a magnifying glass, and an orbiting ring, evoking themes of AI agents and automated search.

We Can’t Monitor AI Agents at Scale. Here’s What It Will Take.

This photograph shows Robert the Robot, an advanced conversational humanoid developed by Geneva-based technology company RB Labs, at the AI for Good Global Summit, a United Nations flagship event aimed at shaping the future of artificial intelligence, in Geneva on July 7, 2026. (Photo by Fabrice COFFRINI / AFP via Getty Images)

The Internet Had a North Star. The UN’s Global Dialogue Made Clear AI Doesn't.

United Nations Secretary-General António Guterres speaks at the opening of the first session of the Global Dialogue on AI Governance in Geneva, Switzerland, July 6, 2026. (Photo by Lian Yi/Xinhua via Getty Images)

After Geneva, AI Governance Must Confront the Trust Deficit

Two workers running inside a scroll wheel embedded in a computer mouse, controlled by a giant capitalist hand.

AI Lays Bare the Authoritarianism of Modern Work. Time to Rethink Education.

A digital concept illustration of a global network showing many profile pictures of diverse people connected by lines, overlaid with futuristic interface graphics such as maps, charts, sound waves, and data visualizations, representing social networking, AI, or big data connections.

Data Sovereignty Doesn’t Mean Data Isolationism

European AI Firms Scale Banned Facial Recognition Systems in India

President Donald J. Trump participates in a bilateral meeting with President Xi Jinping of the People’s Republic of China, Thursday, May 14, 2026, in the Great Hall of the People in Beijing, China. (Official White House Photo by Daniel Torok)

How the US and China Can Step Back from the Battle for AI Model Supremacy

JUNE 16, 2026: A person reads a news report on a smartphone regarding the temporary restriction of Telegram in India ahead of the NEET-UG 2026 re-examination. Authorities said the measure was introduced to curb examination-related fraud, paper leaks and the spread of misinformation during the conduct of the test. (Photo by Waris Nissar/Sipa via AP Images)

India’s Telegram Ban Cleared a Threshold Nobody Tested, and the Executive Moved On

The Claude Mythos logo is displayed on the screen of a smartphone placed on a reflective surface.

Mythos and the Coming End of European Digital Sovereignty

A collage of images of hands with index fingers in pointing position and squiggly lines.

Digital Omnibus Proposal Could Finally End Europe’s Cookie Banner Problem

A repeated motif shows two hands coming together, where the fingers are shaped to suggest a heart. These are set amongst a series of images of small AI chips and red painted rocks which pick up on the motif of the heart.

Jenny Kidd & Synthetic Pasts / https://betterimagesofai.org / https://creativecommons.org/licenses/by/4.0/

Intimate Functionality — A Phrase Defining the Next Year of AI Regulation

An imposing building with two doors and a red brick background, newspaper cuttings of job listings are also collaged into the background. One door is at ground level where a businessman is carrying a large box labelled AI as he is about to enter the building. The second door is high above ground with no apparent means of entry. There is a large flashing neon sign which reads Entry Level above the door. Three young people are below looking up at the door.

Assigning Responsibility When AI Discriminates Against Job Applicants

OMB Director Russell Vought is seen after testifying during the House Appropriations Subcommittee on Financial Services and General Government hearing on oversight of the Office of Management and Budget, in Rayburn building on Tuesday, June 30, 2026. (Tom Williams/CQ Roll Call via AP Images)

The Public Rejects OMB's Federal Financial Assistance Rule

A group of shareholders of Samsung Electronics Co. hold a rally criticizing a tentative wage agreement struck between the company and its labor union near the home of Samsung Electronics Chairman Lee Jae-yong in Seoul, South Korea, Thursday, May 21, 2026. The letters read "Unjustified transfer strike." (AP Photo/Ahn Young-joon)

Workers in Asia Are Fighting for Protections as AI Threatens Jobs

Sen. Mark Warner, D-Va., speaks during a news conference on Capitol Hill, Wednesday, June 17, 2026, in Washington. (AP Photo/Jose Luis Magana)

Senator Warner Makes a First Foray into Agentic AI Regulation

European Commission President Ursula von der Leyen, center, poses with co-chairs of the Special Panel Maria Melchior, right, and Jörg M. Fegert during a report on children's safety online at EU headquarters in Brussels, Monday, July 13, 2026. (AP Photo/Marius Burgelman)

Europe Wants Platforms to Prove They Are Safe for Children

Digital concept with the flag of Europe on a tech background. Europe in the context of cybersecurity, programming, and digital sovereignty. Symbolic image of site development in Europe.

The AI Sovereignty Problem, from Brussels to Bengaluru

Major regulators and AI developers now agree that to safely roll out AI agents we need to be able to monitor these systems real-time. Once agents are deployed, we should be able to observe them, detect failures, and intervene before things go wrong. But the infrastructure to actually monitor agents at scale isn't being built fast enough. It doesn't exist yet in any mature form for the volume and complexity at which agents are being deployed.

As more enterprises, from banking to healthcare, 

 in critical functions, they don’t have the monitoring infrastructure that regulators and safety experts are assuming exists. Take the EU AI Act, which requires that high-risk AI systems be “effectively overseen” by humans. Or the 2026 edition of International AI Safety Report, which calls for monitoring not just the inputs and outputs of agents but also its “chain-of-thought” for signs of unsafe plans well before any actions are undertaken. Google DeepMind published a 

 last month to monitor its own internal agents that write code and act on internal systems. Even with vast resources, Google noted that “it will take increasingly costly measures to contain and oversee them.” The risks are real—in March, an internal Meta agent 

 sensitive user and company data for two hours before it was caught.

We have seen a version of this problem before. In cybersecurity, as enterprises moved more of their operations online, they had to monitor a growing volume of activity and detect intrusions in real time. Only they had decades to build shared tools, standards and expertise around what to log and threats to monitor for. With agents, we don’t have that long to get it right.

We’re working on a short timeline. The share of “action” tools that agents use to directly edit files, run code, or execute financial transactions rose from 

 between November 2024 to February 2026. And agents are already failing in ways both predictable and unpredictable. Last year, a “prompt injection” hidden in a webpage 

 Perplexity's Comet agent to navigate to a user's banking site to extract their credentials. Less predictably, a zero-click vulnerability in Microsoft 365 Copilot 

 an attacker to instruct the AI agent via email to exfiltrate a user’s sensitive files. Even at major companies shipping these systems that went through safety testing, the failures slipped past.

The monitoring that could catch these failures isn't there. Building it at scale comes with real challenges.

There is the cost of “logging” what the agents are doing. Consider a bank deploying agents to handle 

—verifying identities, pulling credit histories, flagging compliance risks. Log every LLM call, every reasoning step, every tool invocation, every interaction with other agents, and the costs of retaining and processing what’s happening becomes impractical. Even if the bank is willing to pay, all of this data doesn't currently reside in one server and it is distributed across actors who don’t share it.

Once you have captured what the agents are doing, companies will still need to detect failures in real time. Using AI itself to flag problems and intervene in real time will be more effective at scale, but it adds significant compute and engineering overhead. For instance, Anthropic found that real-time “constitutional classifiers” used to detect policy violations in model outputs, can increase inference costs 

. For the bank using agents to handle thousands of clients onboarding each day, that overhead quickly compounds.

When automated systems flag potential failures, someone still has to review them. In cybersecurity, analysts are overwhelmed by thousands of alerts each day, with 71% of Security Operations Center (SOC) professionals 

. Many of the alerts are false positives, and critical signals are routinely missed. When the system flags a potential compliance violation during client onboarding, a human reviewer still has to determine whether the agent made a defensible judgment call or a discriminatory one. Making that call requires domain expertise, not just the ability to triage alerts.

These problems won't get solved at once. But there's a near-term move enterprises can make on their own. They can start by tiering monitoring based on the risk each agent deployment carries.

Right now, decisions on what to monitor are driven by engineering budgets rather than risk profile. That's the equivalent of a hospital deciding which patients to monitor based on the price of sensors rather than the severity of their condition. Enterprises deploying agents in critical functions need to make two choices. The first is to capture meaningful logs at the right level of detail, not every reasoning step and tool call. The second is where to monitor heavily, applying more monitoring to high-stakes deployments and not the same level everywhere. An agent summarizing meeting notes is not the same as one running compliance checks on a loan application.

Cybersecurity learned both these moves. The industry had to pivot to sampling (e.g., log every 1 in 1000 packets) through protocols like NetFlow and tiering monitoring by where the risk lives. Anthropic recently brought the compute overhead of its constitutional classifiers down 

. It did this by running a lightweight check on all traffic and reserving the expensive analysis for the small share that gets flagged as risky. Building this kind of infrastructure requires engineering investment that most enterprises haven't made yet.

But tiering at the enterprise level isn't the whole answer. A shared format for what agent traces should contain, including what model was called, what tools were invoked, and what data was accessed, would let monitoring work consistently across the industry. Privacy protections must be designed into the schema from the start, not added later. That includes what gets logged, who can access it, and how long it's retained. Still, industry-wide coordination takes time, and companies don't have to wait for it to start tiering on their own.

A reasonable objection is that monitoring is the wrong frame entirely. The argument goes that we should constrain agents architecturally, through approaches like "sandboxing" and capability limits, rather than monitor them while they act. Those approaches matter. But constraints alone assume you can predict failures. Monitoring exists for the ones you can't.

The cybersecurity industry learned that monitoring built after the breach is always more expensive than monitoring built in advance. AI agents are compressing that timeline. They're already executing transactions, accessing sensitive data, and making decisions that affect real people. The infrastructure to monitor them will get built. The only question is whether enterprises start now or wait for the high-profile public failure that forces them to.

There’s a bullet point on page 38 of the UN’s 

Preliminary Report of the Independent International Scientific Panel on AI

 (IISPAI), released last week, that says something sensible: "The primary drivers of harm are design and deployment decisions—– underlying system architectures of influence, including targeting, amplification and behavioral design—not individual outputs." Yet much of the rest of the report undermines this statement, since its contents are in debt to a definition of AI that studiously avoids reference to the humans who develop and train these models.

In an apparent bid to be neutral, the report engages in precisely what its mandate aimed to avoid: politics. This matters, because the UN's assessment is a reference point for AI governance worldwide. A document that obscures human accountability will carry that blind spot into the policies it informs.

Definitional drift on AI from the OECD to the UN

The problem stems from the report’s definition of AI. Representing the consensus of 40 experts, it reflects previous language from the 

, but its revision subtly pushes human designers and developers away from the systems they build — a trend that we can track across many major policy definitions of AI from 2019 to 2026.

For the IISPAI, AI is defined as systems that "perceive, learn and act," "infer," "learn from experience," and "acquire experience." The case studies presented emphasize LLMs for harms and other architectures to name benefits. The authors defend the imprecision, suggesting that "what unifies AI more than any single architecture is that modern systems learn from experience represented by data.”

Some of this imprecision could be tolerated as a way to future-proof policy against rapidly changing technologies. But this does not justify omitting the actors that create technology — definitions should presume humans won’t become obsolete. Equating "experience" and "data" is an ideological twist toward anthropomorphism that steers policy toward a specific position.

The Panel has real credentials from beyond industry. It is co-chaired by Yoshua Bengio, a Turing Award winner whose recent existential risk advocacy has emphasized AI systems that act in ways designers don’t anticipate. But Bengio’s x-risk work relies upon an anthropomorphic definition common to both boosters and doomers that is helpful to the industry. Existential risk warnings function as 

: it serves industry well to have an opponent constantly warning the world about the terrifying transformational power of your latest product line.

Seemingly in opposition, x-risk and boosterism typically share a belief that the system is powerful enough to be the 

. This focus on what the system is capable of doing crowds out the question of who structures it to do those things. Bengio’s x-risk orientation is present in this definition, and it displaces human accountability. The panel’s other co-chair is Maria Ressa, whose Nobel-winning journalism documented how platforms amplify disinformation and enable authoritarian control of the information environment. Their co-signed 

 acknowledges the problem even as it removes the capacity to act on it: on the one hand, it acknowledges power concentrated into “a handful of companies and a handful of governments.” On the other hand, it announces that humans “do not control these systems.” The living people responsible for the harms detailed by Ressa’s work are effectively absent from the report.

Definitions structure policy. The Panel's "shared evidence base" could be the sole frame of reference for nations without independent AI research capacity. But the Panel’s framing will circulate through policy, 

 other agencies’ ability to locate complex and diffused forms of human accountability. First, it erases the origins of models: 

 Second, it misrepresents the mechanisms that move data into outputs: 

The report erases the origins of AI models using a myth I call 

. Following Donna Haraway’s critique of the “

,” the system-from-nowhere frame treats LLMs not as products but as independent operators, cut off from the humans who build and deploy them. The IISPAI authors draw that boundary whenever they describe automated actions without naming the humans who set the goals — and determine 

 such decisions are automated in their pursuit.

For example, the report describes AI models "lying and cheating to avoid being shut down," it explains that "AI chatbots have developed sycophancy... to prolong interactions and create emotional attachment." Chatbots did not 

 sycophancy. Reinforcement learning from human feedback (RLHF) strengthened this tendency and executives released it, a byproduct of a business model based on user engagement. Models did not 

; developers prompted it with language that allowed for lying as one path toward its objectives. The report mistakes 

The claim that systems have “experience represented by data” ignores the court battles over where that data comes from — “human cultural traces,” as the report calls the text and images in the training corpus. 

 have criticized both the sourcing of this data and the impact of models on their livelihoods. When copyright claims are mentioned in the report, they are framed only as an obstacle to developing global AI capacity.

Humans also select and organize the architectures that turn that data into patterns, and those choices reflect goals set, typically, by the market. The report contains no significant mention of the labor of data annotation, RLHF, content moderation, or the 

 where this work takes place, except to call human-labelled data “a bottleneck” for developers. The Panel presents the system as simply existing, as an inevitable product of accumulated “learning.”

AI systems do not think in human terms. Anthropomorphism is often harmless — people say their car turns its headlights on in a tunnel because it 'thinks' it's nighttime—but in policy contexts it’s a 

 that redefines human terms, roles, and responsibilities to fit the capacities of machines.

While the text, code and images a model produces can be indistinguishable from human language, it emerges from a wholly distinct process. Our presence makes it seem otherwise: language closes the gap between forms of experience, even when that experience is fully contrived. But recasting the model's data — a collective resource—as its individual “experience” implies a machine that labors its way to conclusions rather than associating its way to them (sometimes to astonishing effect). Defer to this shorthand often enough, and we can (and often do) imagine more beneath the surface of an LLM than is actually there. Such personification sets us up to forget who builds these systems, how, and why.

Consider the IISPAI’s claim that LLMs "systematically mislead" by "lying and cheating." Models are erudite zombies: they produce text influenced by training data, optimization during pre- and post-training, and the content of the prompt. The model sometimes 

 into fictions we read as lies; it does not require an intent to lie. When we worry about deceptive machines, we do not ask critical questions about why we should trust them in the first place.

 to infer. They are built, piece by piece, from neural networks that pull data toward a human-assigned target. Models do not 

 to incorporate RLHF; companies do. Models do not 

 to expand their answers through the vocabulary of reasoning; companies massage those patterns through post-training. When we ask what machines learn rather than what data is provided to them and what structures are deployed to find patterns in that data, we sidestep important conversations about design and deployment decisions.

When text, code, or images arrive on our screen, it’s compelling, and easy to forget the machinery that makes it happen. We should ask instead: who 

 of reply over any other? In state-of-the-art models, engineers and designers optimize models to minimize friction, to sound more compelling, to present more accurate information. The emphasis on “AI” hides the machinery from view as well as the humans who decide what it does.

The report’s chapter “What does the evidence show?” leans heavily on benchmarks without acknowledging the limits of what they 

. It uses vague terms such as "expert-level reasoning,” but at what level do experts reason? It treats scores as claims about “general capability,” itself a 

 argue, these tests assess intelligence as the capacity to take the test, but "it's not like a lawyer's job is to answer bar exam questions all day."

By orienting the Report to a consensus under a mandate of scientific neutrality, the panel could not be expected to name the real antagonisms between the AI industry and those it exploits or harms. Instead, it invented a fantasy: the system from nowhere. In convenient, anthropomorphic terms, an entire tech ecosystem becomes a mythological entity called "AI," and policymakers are asked to condemn or praise it. It shapes a compromise through extremes — a balance between "enormous benefits" and "considerable risks.”

Consider what the definition does to the report's own summary of harms. It warns of risks that AI "behavior diverges from human goals and values, with prominent risks including bias, AI-initiated deception, sycophancy and loss of control." Rejecting the system-from-nowhere frame reveals meaningful leverage points: the training data that encodes discriminatory histories, reward functions that select for engagement over accuracy, and interface design decisions that promote user passivity or expose vulnerable users.

Scientific neutrality doesn’t mean resisting accountability. The International Panel on Climate Change’s (IPCC) source and impact attribution work is a template: year after year, we are reminded that global warming is a product of human activity. We can say the same thing about AI: it is not a force of nature, and its origins are clear. The harms the report lays out are real. Its authors should not be afraid to say that the companies developing language models are, in fact, developing language models. What, precisely, do they imagine these companies are doing otherwise?

If we name the annotators, moderators, developers, and decision-makers the existing definition erases, we center the power of the people who make AI what it is, rather than treating humankind as passive observers of its inevitable arrival. This would shift the question from what 

 to what companies, engineers, communities and governments 

 technology to do, opening space for democratic participation in setting goals and weighing trade-offs.

The next iteration must abandon this neutrality of extremes and acknowledge that these systems are human at their core. I suggest starting with that fleeting bullet point from page 38: "The primary drivers of harm are design and deployment decisions." This applies not only to the platforms that use these systems, but to the industry that, yes, 

Fifty of the 92 organizations that asked to join the UN's new cybersecurity process this year were 

 from taking part, and in most cases no reason was given. When the body holds its first substantive session next week, many of the incident responders, security companies, digital rights organizations and researchers who run and defend cyberspace will be absent, leaving the global cyber rules at risk of losing touch with the expertise they depend on.

Cyberattacks now shape both war and everyday life

. Ransomware gangs shut down hospitals. State-linked hackers hit retailers, pipelines and banks, disrupting daily life and creating risks for national security and economic stability. Cyber operations have also become a standard feature of armed conflict, knocking out power grids, disabling communications and disrupting military and civilian systems alike alongside troops on the ground. As these capabilities grow more sophisticated, and are used across the spectrum between peace and conflict, developing and implementing the rules for how they should be used responsibly and legally matters more than ever. 

 on ICT security, a process where states negotiate the cyber rules and how they should work in practice. Cyberspace is not built, run or defended by governments alone, and the rules of this process reflect that: while states lead the negotiations, they explicitly allow non-state actors, often referred to collectively as the multistakeholder community, including researchers, technical community and civil society groups, to take part too. However, more than half of the organizations that applied to take part in the process were blocked. The result is a process that risks becoming progressively detached from the operational and technical expertise on which effective cyber governance depends.

The numbers tell part of the story. Of the 92 non-governmental organizations that applied for accreditation this year, 50 were 

. Russia alone objected to 43, a list spanning think tanks, universities, cybersecurity companies and civil society organizations across Europe, the Americas, Asia and Oceania. A handful of other states lodged further objections. The scale suggests a blanket approach that has the effect of keeping many stakeholders out. In almost every case, no reason was given. 

: an objecting state is only asked to make known to the Chair of the process "on a voluntary basis," the general basis of its objection. A single state objecting is enough to block an organization outright, with no threshold of support needed from other states and no formal appeal available to the organization itself.

The Chair has been consulting objecting states in the hope of reversals, and deserves credit for trying, but that process is capped at three months of informal consultations, with no mechanism to force a decision and little prospect these states will withdraw before the session opens. A procedure meant to be used judiciously, as an exception, has become a general bar.

The cost of leaving expertise at the door

It is tempting to file this under diplomatic procedure, the kind of skirmish UN processes generate routinely. That would be a mistake. Objections of this kind have historically 

reflected a mix of geopolitical disagreement and competing views

 over whether governments alone should govern these issues. But the organizations excluded are not observers of the domain being regulated; most of them operate it. They are the incident responders working through the night when a hospital's systems go down, and the network operators and standards bodies keeping the internet functioning. They are the companies sharing threat intelligence the moment an attack is detected, often before governments are aware of the scale or nature of the incident. And they are the researchers and civil society groups who have spent years studying how these capabilities harm ordinary people, and who hold both governments and companies accountable.

The exclusion also lands unevenly. Wealthy governments can convene their own companies and fund their own research; less-resourced states rely on the multilateral table to access that expertise. Each blocked organization risks widening the gap between governments that can access this expertise and those that rely more heavily on multilateral processes to provide it, further deepening inequalities between states.

None of this means stakeholders have no voice left. 

, an informal part of the process, with two working groups, one looking at challenges in cyberspace and one on capacity-building are meant to include experts in their discussions. As such, they remain a genuine and valuable channel for technical input. In addition, stakeholders can continue to submit written contributions and organize events around the process regardless of accreditation. These channels matter and should be protected. But they operate alongside or after the formal negotiations, not within them, and cannot substitute for a seat at the table where the text itself is negotiated and agreed. Input offered once a text has already taken shape is generally less able to influence it than input offered while it is still being drafted. By the time stakeholders can react, positions have often already hardened around whatever the most cautious states are willing to accept.

Which brings us to governments that support the multistakeholder model. That commitment is written into the mechanism's foundations: states agreed, by consensus, to engage stakeholders in a "systematic, sustained and substantive manner." Living up to that will take more than reaffirming it.

Supportive governments should arrive at the first negotiating session, ready to act. With reversal unlikely before the session opens, the immediate task is less about undoing this round of blocks than preventing it becoming the template for every round after it. Governments should make strong statements on the record, and use the exchange of views on accreditation to ask what the rules already invite: that objecting states explain their reasoning, so it can be tested. We expect some states will already do this — but there is strength in numbers, and the more countries that make these points, particularly from the Global Majority, the stronger the case. 

Creating that record, getting these objections and any reasons for them written down, is important even if it doesn't change the outcome this time. They should press for transparency in how decisions are made and unmade, particularly ahead of the next accreditation window, so this year's blocked organizations are not blocked by default again. They should protect the avenues that remain, above all the Dedicated Thematic Groups, agreed by consensus to include stakeholders, and say plainly that objections raised in the plenary should not bleed into those spaces. And they should nominate diverse, independent experts to the pool of outside speakers invited to brief the meetings and open their own delegations to independent participation where stakeholders have been shut out.

Beyond the negotiating room, they should fund stakeholder coordination and participation, particularly from the Global Majority, rather than assume engagement in a process this demanding sustains itself. They should also strategically engage their own domestic companies and civil society directly, telling them where their expertise and evidence could be most useful, rather than assuming they already know.

The multistakeholder community has work of its own to do. It should organize across regions, pooling evidence and aligning messages so their contributions carry maximum weight. It should keep contributing to the framework states have already agreed to. Accredited or not, the work of securing cyberspace does not pause for a procedural bar.

The deeper point is one that should be familiar to states committed to pluralistic governance. They govern diverse societies on the basis that the most consequential decisions should not rest with the state alone. Many of the countries that have built their digital economies and security on an open, secure and inclusive cyberspace have spoken up against a model where governments alone call every shot at the UN, but voicing objection has not been enough to stop it. These states have an obligation to make the multistakeholder model work, not as a favor to these actors, but because rules written by governments alone will fail on their own terms. Fifty organizations being blocked this year may not be reversible. Whether it becomes the norm still is. That is the real question this session begins to answer.

Disclosure: Global Partners Digital and Oxford Information Labs, where the authors are affiliated, were among the organizations blocked from accreditation this year.

Surveillance technology, heavily regulated in Europe, is freely deployed across India. Spanish firm Herta Security is among the top suppliers behind facial recognition technology solutions for Indian systems, according to Investigate Europe.

At Howrah station in Kolkata, the trains never really stop, and neither do the cameras. One of India's busiest railway terminals, around a million people pour through it every day, with crowds so dense that following a single face can be near impossible, at least with the human eye.

Yet mounted high above the entrance and exit gates, platforms, food courts and waiting rooms, around 100 live facial recognition cameras silently keep track. The system, installed in the past year, lifts faces from the live feed and cross-checks them against a database of photos: wanted offenders, criminal suspects, missing people. A match alerts the railway police to the exact spot where a person was seen and authorizes them to approach the person.

Some commuters seem unaware that they are being surveilled in this way. Barnali Biswas, a private sector employee who passes through six days a week, was unbothered when questioned by a reporter, saying she thought it was probably good for safety. "People with nothing to do with crime had nothing to fear," she said. This kind of camera, enabled with facial recognition software, is increasingly familiar across India. And in eastern India, one supplier of that software is the Spanish firm Herta Security.

According to local partners and company documents, Herta supplies hundreds of railway stations across the region, Delhi's largest prison complex, a pilgrimage site in Ayodhya, and the city control rooms of Ahmedabad.

Herta confirmed the use of its technology in India but in a written statement said that it would "not disclose confidential customer information". Its software may also be deployed at Howrah station itself, though local railway officials would not confirm this when asked.

A source at one of Herta's Indian partners told Investigate Europe that they estimated that more than 4,000 cameras across the world’s largest democracy are now powered by the Spanish firm’s technology.

At least part of that rollout is linked to the Nirbhaya Fund, a pot of public money created to tackle sexual violence after a shocking 2012 Delhi bus gang rape. Women’s rights groups in India argue that the millions of euros earmarked for the fund have been spent on mass surveillance rather than on more direct victim protection and support.

According to four leading legal scholars who specialize in EU artificial intelligence and biometrics law, two of these deployments would be deemed unlawful if they operated inside the European Union: the system on Indian Railways' Eastern Region and a city-wide surveillance program in Ahmedabad.

Herta said it took such concerns seriously, adding that its products are “developed in line with European data-protection principles, regardless of the market in which they are deployed.” However, the company said that it could not "control how public authorities or system integrators implement the technology in specific environments".

Italian Member of the European Parliament Brando Benifei, however, said the findings raised troubling questions for the European lawmakers. "The fact that surveillance technologies banned in Europe are being exported and deployed elsewhere, such as in India's railway stations, exposes a dangerous double standard."

While the European Union has heavily restricted similar surveillance on its own citizens, Investigate Europe has found that it funded the Spanish company to develop its technology. Since 2020, Herta has received more than €3.3 million in EU research funding for projects involving "crowd behavior analysis" and facial recognition.

Herta Security was founded in Barcelona in 2009 by former Bosch employee Javier Rodríguez Saeta, a specialist in biometrics – automated identification based on people’s physical features or behavior. "Back in 2005," Rodríguez would later recall 

, "I already thought that in the future it would be necessary to identify people who did not want to be identified."

Today, the company's flagship product, BioSurveillance NEXT, is designed for what its own 

marketing materials describe as crowd-scale deployment

. Rodríguez has spoken of using it for "demonstrations, sports races, religious gatherings and airports.” The data sheet advertises real-time searches against databases of up to 100 million subjects.

The technology is underpinned by graphics-processing units, the same chips that power most current artificial intelligence and accelerate facial recognition so it can be performed in real time. In 2015, AI chipmaker Nvidia recognized Herta as 

. By 2019, Herta claimed to have amassed more than 200 clients across 50 countries.

The market Herta serves has expanded rapidly in recent years. Such tech is now used at sports stadiums in the United States, by police forces in the United Kingdom and has also been trialed by police forces across the EU.

However, the technology is heavily restricted in Europe. Since 2025, the EU AI Act has 

largely prohibited the use of real-time remote biometric identification systems

, above all facial recognition, in public spaces for law enforcement purposes. Those drafting the law identified this as one of the few uses of AI they considered unacceptable.

The ban allows three narrow exceptions: targeted searches for victims of trafficking, sexual exploitation or kidnapping, or missing persons; the prevention of a specific, imminent terrorist threat; and the identification of suspects of a closed list of serious crimes including terrorism, murder and rape. Even then, each individual use must be authorized by a judge in advance and logged in an EU database.

Herta has a track record of working with clients beyond EU borders. In a marketing presentation from 2021 and seen by Investigate Europe, Herta lists several projects.

These include city-wide surveillance schemes known as "safe-city projects" in Jamaica, Thailand and Mumbai in India. Other uses of its surveillance software include police forces in Colombia and Indonesia; football stadiums in Belarus and Russia and airports in Mexico, Nicaragua and Nigeria. It is not clear whether these partnerships are still active today.

Herta’s software has also been used in Europe. Notably, the company’s technology was trialled by the German Federal Police in 2017 and 2018 

asked about the risk that Herta’s products might be misused

. "In Europe," he said, "we are very well protected, we have very clear legislation that sets out the uses of this technology. There's nothing to fear." At that time, the EU-wide ban on real-time facial recognition in public spaces was not in place. The AI Act’s prohibition only took effect in February 2025. He did not address the protections available to citizens in his other markets such as India, where similar strict legislation is absent.

The Indian portfolio: Railways, prisons and temples

Herta's expansion into India has unfolded over a decade. Around 2014, the company supplied four or five cameras for the Mumbai Safe City project, according to a local business partner.

A former senior researcher at Herta, who asked to remain anonymous, said that the data Herta acquired through its early Indian deployments was central to overcoming problems with the algorithm's poor performance on non-white faces.

Herta told Investigate Europe that it "does not use customer operational data from deployments as a default mechanism to train or improve its algorithms". Any use of personal or biometric data for algorithmic training "would require a clear legal basis.”

The real breakthrough came eight years later. In 2022, a Delhi-based company 

won an €11.5 million tender contract for a video surveillance system

 covering hundreds of stations in eastern India. The facial recognition layer of the system runs on Herta software.

 that 540 facial recognition systems were operational at 143 stations. The full plan covers 392 stations, among them some of the largest stations in the region. Herta’s local partners said this could be fitted to more than 1,500 cameras in the near future.

Authorities regularly cross-reference faces against a watchlist of people of interest. It is not clear what criteria are used to list people, but one of Herta’s local business partners said there were around one million subjects on the watchlist.

The scale of the system is unlike anything available in Europe, they added. "On the busiest station, if I just implement one camera, you run 10,000 people in five minutes." If the system returns a match, an alert is sent to an armed police force authorized to stop them, they claimed.

treated facial recognition matches at 80 percent similarity as positive identifications

. With millions of travelers, even a small error rate is not insignificant. Significantly, the watchlist data is not public, nor are the criteria for inclusion, and there is no route to contest a false match.

Herta’s presence on the Indian rail network is not confined to the Eastern Region. Under a program run by the Railway Land Development Authority, which modernizes stations nationwide, Herta’s software has reached terminals as far as Jaipur and Secunderabad, according to Herta’s local partner.

Its software also runs facial recognition systems across three of Delhi's prison complexes, Tihar, Mandoli and Rohini, the local partner claimed. The government contract for the deployment is reportedly worth 352 million rupees (€3.2 million).

The system is not confined to public or state infrastructure. It also runs at the Ram Mandir temple in Ayodhya, where the local partner claims that the surveillance system alerts local police when individuals on a watchlist are identified. Similar deployments at two other temples are in progress, they added.

Herta also operates in the context of safe-city programs: in a marketing presentation obtained by Investigate Europe, the company claimed to have deployed 140 facial recognition cameras in Ahmedabad, and lists further deployments in several other cities.

Investigate Europe sought comments regarding the deployments by Eastern Railway, the authority implementing the Safe City Ahmedabad project, and the authorities responsible for managing Delhi’s prisons. All inquiries went unanswered.

Herta’s software is running on around 4,000 cameras across the country, according to one of its local partners.

Herta’s expansion in India is apparently linked to a policy shift triggered by an infamous rape case. In December 2012, a young physiotherapy student boarded a private bus in Delhi with a male friend after watching a film in the local cinema. Unknown to them, the bus was off-duty. The driver and five other men had stopped to pick the two of them up under the pretense of carrying passengers.

Over an hour, the six men beat her friend, gang-raped her, and threw both of them, seriously injured, from the bus. She died of her injuries two weeks later. The case triggered widespread protests across India and forced a national debate about gender violence, public safety and the state’s failure to protect women.

. Civil society groups that campaigned in the wake of the case had asked for victim support, legal aid and shelter networks, recalled a human rights lawyer involved in the discussion at the time.

The initial commitment of 10 billion rupees (€92 million) was for "the empowerment, safety and security of women and girl children". The fund came to be known by the same unofficial name the Indian press had given the young woman: Nirbhaya, meaning fearless.

Over the years, the Indian government allocated more money to the fund. By March 2025, around 58 billion rupees (€532 million) had been disbursed. Indian government data shared in February 2024 showed that roughly 50 percent went to surveillance and policing functions, versus 31 percent for direct victim support services and emergency helplines.

Some argue these new surveillance systems have not made women significantly safer. In 2014, the year after the Nirbhaya Fund was established, 340,000 crimes against Indian women were reported by 

. By 2023, the most recent year for which data is available, that figure had risen by roughly a third to 450,000.

Flavia Agnes, a Mumbai-based lawyer who estimates she has worked on around 100,000 cases of violence against women over a 40-year career, said the design of the response to the national outcry was wrong from the start.

"I have not come across cases where this kind of surveillance has helped to identify the accused," she said. Most violence against women in India does not happen in the places the cameras can see like train stations, she explains, but in the home. "About 95 percent of the rape cases take place by known people."

Audrey D'Mello, who directs a center providing legal aid to survivors of violence in Mumbai, has not seen evidence of the surveillance working in her field. The infrastructure built in the name of women's safety, she said, has rarely been about women's safety. She is more cynical about the government’s project. "They also realized," she said, "that it's easier to push in the name of women."

There is no direct evidence suggesting that the use of surveillance cameras in public spaces has not increased women’s safety in the country.

The Indian Ministry of Women and Child Development, which oversees the Nirbhaya Fund, did not respond to requests for comment.

Four scholars who work on EU artificial intelligence and biometrics law said the deployments in the Indian Railways Eastern Region and the Safe City Ahmedabad program would be unlawful in the EU.

"Such a deployment in an EU member state would quite clearly violate" the EU’s AI Act, said Rita Matulionyte from Macquarie Law School in Sydney.

Catherine Jasserand, of KU Leuven in Belgium, reached the same conclusion. "I don't think these two cases constitute good examples of the live use of facial recognition technology by police in public spaces."

"The two deployments are too broad to fall under one of the exceptions," she said, referring to the carve-outs for searches for vulnerable people, terror threats and suspects of serious crimes. Even under those exceptions, EU law requires it first to pass a national statute authorizing the practice, with detailed rules on supervision, reporting and judicial control. Spain, where Herta is based, has passed no such law, meaning the company could not legally run the system at home as in India.

India's own legal framework offers no comparable protection. India has no comprehensive data-protection law in force, and what it has passed 

exempts policing and law enforcement so broadly

Latest

Podcast

Recent

Our content delivered to your inbox

Thank you!

Around the globe