Five Real-World Failures Expose Need for Effective Detection of AI-Generated Media

shirin anlen is the AI Research Technologist and Impact Manager for WITNESS.

At the start of 2024, fears about AI-driven disinformation—deepfakes, synthetic audio, and manipulated images—dominated headlines once again. But by year’s end, a new narrative emerged: the sky hadn’t fallen—AI hadn’t swung elections. Crisis averted, or so it seemed.

But this framing misses the point.

Behind the headlines, communities on the frontlines of information integrity—journalists and human rights defenders—especially in the Global Majority, continued to face high-stakes threats. And what they encountered was alarming: AI detection tools, even the most advanced, are failing in the places and moments where they’re needed most. These failures aren’t just technical—they’re systemic and inequitable.

By 2025, AI-generated video had become a central weapon in global conflicts. During the Israel-Iran escalation, a torrent of synthetic media flooded social platforms: fake videos of missile strikes and manipulated clips of downed planes were rapidly disseminated, often with little attempt at realism. This was AI slop in action: low-quality, fast-produced synthetic content that thrives not on credibility, but on speed and repetition. Similar tactics emerged in the India-Pakistan context, where a viral deepfake showed a Pakistani general appearing to concede defeat. In Los Angeles, protests over immigration raids were hijacked by waves of false content, including AI-generated videos of fabricated National Guard deployments and recycled clips miscaptioned by chatbots.

These are not isolated incidents—they’re signals of a broader shift. In the age of AI slop, synthetic media doesn’t necessarily need to be undetectable or even convincing to inflict harm. It needs to land first, overwhelm moderation systems, and exploit the attention economy before verification can catch up. When detection tools lag, fail, or contradict one another, the result is not just confusion—it’s the collapse of public trust in any shared reality.

The "detection equity gap"

At WITNESS, we’ve seen these failures unfold in real time. Since launching our Prepare, Don’t Panic initiative in 2018, we’ve heard a consistent message from our global partners: they lack timely, reliable, and trustworthy tools to identify deceptive AI media. These gaps are particularly urgent in contexts where generative AI affects democracy, social resilience, and conflict. That’s why in 2023, we launched the Deepfakes Rapid Response Force (DRRF)—a pioneering support initiative that connects frontline journalists, fact-checkers, and human rights groups with deepfake and media forensic experts for real-time, evidence-based analysis of suspicious, deceptive AI content.

The DRRF’s investigations—from Cambodia to Mexico, Sudan to Ukraine and Ghana—expose the real-world limitations of even the most advanced AI detection tools. These aren’t abstract failures—they’re failures that widen what we call the “detection equity gap,” where tools designed for high-bandwidth, English-language, lab-based environments constantly break down in the noisy, multilingual, under-resourced contexts where they’re most urgently needed.

Case 1: Language barriers and technological blind spots (Cambodia, Ghana)

In Cambodia, a leaked audio clip allegedly featuring a former prime minister ordering an assassination spread quickly. But some AI tools couldn’t analyze it—because they didn’t support Khmer. The clip’s background noise and heavy compression further confused the models. Technical challenges that standard detection tools often struggle with, demonstrating their inadequacy when confronted with the messy reality of how information circulates online. A similar issue arose in Ghana, where a suspected deepfake video of a political figure returned uncertain results, partly due to compression artifacts and the subject’s darker skin tone. These aren’t edge cases—they expose how current tools are built for sanitized data, dominant languages, and lighter-skinned subjects. 

Case 2: Untrained media format in conflict zones (Sudan)

In Sudan, a leaked walkie-talkie transmission allegedly captured military leadership ordering attacks on civilians. Yet detection tools failed—not because the content was too advanced, but because walkie-talkie audio wasn’t in their training data. Analysts flagged low confidence, and no answer was better than a bad one. This raises a critical question: How can detection tools possibly anticipate the infinite and evolving types of media they might need to analyze, particularly in dynamic conflict environments? In crisis zones and humanitarian contexts, media comes in all forms—low-res, compressed, chaotic. Tools must be trained accordingly. And when they fail, they must explain why. Without transparency and explainability, users can’t understand why a tool failed—or even that it failed. Then a tool becomes not a source of clarity, but another point of confusion and potential distrust.

Case 3: False certainty and misleading tools (Mexico)

In Mexico, a viral image claimed to show President López Obrador posing with drug kingpin El Chapo. Two versions—one suspected to be the original and one suspected to be manipulated—were fed into public AI detection tools. The result? Conflicting, inconclusive outputs. Worse than no answer at all, this inconsistency amplified public confusion and mistrust. While advanced AI tools resolved the issue with high confidence, such expertise and access are rarely available to journalists and civil society groups working on the frontlines. This disparity highlights a significant barrier: a thoughtfully designed access to detection tools that serve the needs of targeted, at-risk communities, such as journalists and human rights defenders, without introducing new security or privacy risks. Without equitable access, detection becomes a privilege for the few.

Case 4: Detection tools are not enough (Ukraine)

In Ukraine, pro-Russian accounts shared an image of a Ukrainian soldier who had lost several limbs, using it to falsely link the Ukrainian army to Nazism and to generate viral memes. Analysis teams used a range of AI detection tools, yet encountered conflicting results regarding potential AI manipulation. Resolving this required integrating manual verification methods: tracking related accounts, identifying similar image styles, and leveraging additional image analysis tools beyond AI detection. This case reveals an essential truth: no single tool is sufficient. Real-world verification demands a multi-modal, human-centered, context-aware approach. AI detection must be integrated into broader investigative workflows, not treated as a silver bullet.

Case 5: Weaponizing doubt (Ghana)

In Ghana, a video of the Vice Presidential Candidate making controversial promises was immediately dismissed by him as a deepfake. Yet the video clearly showed a real event: multiple camera angles, a large crowd, a sign language interpreter—almost virtually impossible to fake with the current state of AI. This wasn’t a case of manipulated media, but of manipulated narrative. A powerful figure exploited public confusion about deepfakes and the absence of AI literacy to cast doubt on genuine evidence. In a world where detection is uneven and AI literacy is low, bad actors don’t need to fake media—they just need to claim it’s fake.

A dangerous truth

These five cases—spanning different regions, media formats, and political stakes— expose a dangerous truth: we are measuring the success of AI detection tools in the wrong places, under the wrong conditions, for the wrong audiences. The global information ecosystem is increasingly vulnerable, yet we continue to treat detection as a purely technical problem that can be solved by better algorithms. We must fundamentally rethink how we define success in AI detection—and who it serves.

At WITNESS, we believe it's time for a new standard—one grounded not just in technical accuracy, but in social relevance and real-world usability. That’s why we developed the TRIED Benchmark: Truly Innovative and Effective AI Detection—a new framework for evaluating detection tools based on their practical impact and capacity to meet the needs of diverse users in high-stakes environments. Officially launching at the AI for Good Global Summit in Geneva on July 10, TRIED is grounded in global consultations, direct casework on deceptive AI, and feedback from affected communities. It emphasizes adaptability to compressed, noisy, and fast-moving content; fairness across languages, formats, and cultural contexts; transparency in how tools work and where they fail; targeted accessibility for the communities most affected; and integration with existing verification practices and investigative workflows.

The TRIED Benchmark also provides actionable guidance across the ecosystem. Developers must design tools that are adaptable, continually updated, and transparent. Standards bodies should set minimum requirements for effective detection solutions, embed explainability and resilience into their guidelines. Regulatory efforts, both international and domestic, must embed oversight and accountability mechanisms that reflect the principles of trustworthy, human-centered AI. Governments and market leaders have a responsibility to redefine innovation and invest in sociotechnical evaluations and media literacy programs that empower users to assess and interpret AI-generated content.

The costs of AI failure won’t always make global headlines. They will be felt in the collapse of trust, in the manipulation of public discourse, and in the deepening of already-existing inequalities. If we want a credible information ecosystem, we must stop treating AI detection as a narrow technical challenge and start building systems that reflect and respect the world’s complexity.