Artwork including a giant bust of Elon Musk sits along the route the billionaire's SpaceX Starbase south Texas launch site. (Photo by Laura Brett/Sipa via AP Images)

What Policymakers and the Public Need to Know About the Cult of AI

A bald eagle, signifying the US Department of Justice, next to the Google logo.

The Google Remedies Turn Landmark Verdicts Into a Whimper

CEO of Anthropic Dario Amodei, addresses the gathering at the AI Impact Summit, in New Delhi, India, February 19, 2026. REUTERS/Bhawika Chhabra

Who Should Pace the Frontier? Not Dario Amodei

A traditional Chinese arch bridge over green water surrounded by trees sits in front of large skyscraper buildings.

A traditional Chinese arch bridge sits in front of the 'Huijin International Building' — the building that hosts DeepSeek's headquarters, in Hangzhou, Zhejiang Province, China, on March 18, 2026. (Photo by STR/NurPhoto via AP)

China Is Running a Different AI Race

iPhones are displayed during an announcement of new products on Wednesday, Sept. 9, 2026, on the Apple campus in Cupertino, Calif. (AP Photo/Jeff Chiu)

Will Apple’s ‘Reference Image’ Feature Help Defend Against AI Manipulation?

Hundreds attend a Town of Tonawanda Planning Board meeting in Kenmore, New York on Wednesday, June 3, 2026, as they expressed vocal and at times boisterous opposition about a proposed data center on the site of the now closed Tonawanda Coke facility. (Photo by Craig Ruttle/Sipa via AP Images)

Americans Shrugged Off Surveillance. Then Came Flock and Data Centers.

Meta founder and CEO Mark Zuckerberg attends the Allen & Company Sun Valley Conference at the Sun Valley Lodge on July 9, 2026 in Sun Valley, Idaho. (Photo by Kevin Dietsch/Getty Images)

Two Reasons Meta’s Settlement with State Attorneys General is a Win for Mark Zuckerberg

A protester holds a 'Don't let AI take our future' placard during the demonstration by the group Pause AI outside Downing Street. (Photo by Vuk Valcic / SOPA Images via AP)

The Like Button Helped Social Media Put Profit Over People. We Can’t Let AI Do the Same.

Anthropic CEO Dario Amodei at Salesforce's Dreamforce conference at the Moscone Center on September 15, 2026 in San Francisco, California. (Photo by Benjamin Fanjoy/Getty Images)

Making Sense of AI Dread? Don't Get Distracted by Doomer CEOs or the China Canard

European Commission President Ursula von der Leyen delivers a speech at the annual State of the European Union at the European Parliament in Strasbourg, eastern France, Wednesday, Sept 16, 2026. (AP Photo/Pascal Bastien)

What Von der Leyen’s State of the Union Means for Europe’s Tech Ambitions

OpenAI CEO Sam Altman speaks during a two-day meeting of Group of 20 ministers in charge of innovation policy in Chapel Hill, North Carolina, on Sept. 2, 2026. (Kyodo via AP Images)

Detecting AI Agent Failures Is Not Enough to Govern Them

Protesters rally on the steps of State Capitol in Salem, Ore., on September 8, 2026, calling for a moratorium on new data center development in the state for three years. (Photo by Alex Milan Tracy/Sipa via AP Images)

All Smokescreen, No Fire in Data Center Foreign Influence Claims

California Assemblymembers meet during an Assembly session at the state capitol, Monday, Aug. 31, 2026, in Sacramento, Calif. (AP Photo/Jeff Chiu)

California Leads the Way With New Slate of AI Bills

BRUSSELS, BELGIUM - JUNE 9, 2023: The symbol of the European Parliament on the glass façade of the European Parliament. Close up

Europe’s Digital Rules Need an Independent Enforcer

CEO of Anthropic Dario Amodei attends a working lunch with G7 leaders, G7 outreach partners, and global tech CEOs on innovation and AI, during the G7 Summit on June 17, 2026 in Evian-les-Bains, France. (Photo by Anna Moneymaker/Getty Images)

To Make AI Rules, Policymakers Must Resist the Influence of ‘Radical Intentionalists’

Elementary school students arrive on the first day of school in Munich on September 2025. (Frank Hoermann/Sven Simon/picture-alliance/dpa/AP Images)

Let’s Not Wait for Court Cases to Address Cognitive Harms in Education

Ursula von der Leyen, President of the European Commission. Photo by: Malin Wunderlich/picture-alliance/dpa/AP Images

Von der Leyen Moves to Lock Down Social Media for Under-15s Across Europe

A computer monitor with a parody of a tech company logo floats on the waves while human hands reach up from the depths.

A Rising Tide Lifts All Bots by Rose Willis & Kathryn Conrad / 

Millions of Volunteers Are Keeping the Internet Human. For Now.

Olivia Konar holds a banner with names of young people who advocates say died as a result of social media outside the Ronald V. Dellums Federal Building and US Courthouse on Tuesday, Aug. 18, 2026, in Oakland, Calif. Her sister Coco Konar died in 2022 when she was 17. (AP Photo/Noah Berger)

Are the Meta Settlement Terms a Win for Kids or a Digital Rights Risk?

Processor, CPU or Chip With the Flag of European Union

How Europe Can Escape a Captured AI Ecosystem

 the iPhone 18 Pro. Among the device’s new photography features is “Apple Reference Image,” and it represents the Silicon Valley behemoth’s entry into content provenance. "Now you can prove the authenticity of an image taken on iPhone. Secure, pixel-level signing of photo data happens at the time of capture, making it easy to verify exactly what the camera sensor saw," the company 

 expands on how the company will do this.

For those of us who have spent more than a decade arguing that cameras should include such features, this is welcome news. Starting fifteen years ago, in reaction to a profusion of claims that real footage from contexts like Syria was falsified and a growing sense that the trustworthiness of critical images across society was under threat, myself and colleagues at WITNESS and our counterparts at the 

 began building our own tools for trustworthy image and video capture (now 

 or Eyewitness Mode on every camera to show what was real in the face of denial.

Original pic from a pitch deck circa 2013 from WITNESS and Guardian Project imagining an Eye Witness Mode

We talked to many of the major tech players (including Apple circa 2016-7) but we were too early. Now generative AI has made the problem of confirming what is real and what is not dramatically more urgent.

Over the past years we've had the maturation of the Coalition for Content Provenance and Authenticity (C2PA) standard for showing the recipe of reality and AI in what we see and hear, and pioneering work from Google implementing C2PA (and complementary tools like SynthID that watermark AI content) in the 

Apple has now introduced its own approach. Apple’s Reference Image is a serious piece of engineering, and a genuine response to the epistemic crisis of AI and ambient uncertainty. It addresses some serious challenges we've all had in this area, while it also presents some unanswered questions and indicates some crucial next steps.

So what to make of it? My lens on the subject is informed by my experience collaborating on independent capture apps used in specialist contexts in human rights and journalism, co-authoring 

 that evaluated these types of trusted capture tools for images and videos against fourteen key dilemmas, and subsequently working on the guiding principles and ‘

’ for the C2PA standards now in use by Google, OpenAI, TikTok and others. I was also briefed by Apple in advance of the launch, and some of my analysis draws on that conversation.

In a nutshell, what does Apple Reference Image do? Basically, the camera sensor itself signs each image it produces. Each camera sensor generates its own cryptographic key when it is first initialized on the factory line, and never releases the private half. Apple only certifies the public key. When you shoot in the new Reference mode, which is an opt-in choice, you reboot the camera into a strict state where the sensor doesn’t alter pixels at all: no de-mosaicing, no tone mapping, no compression. The raw sensor data, together with a lower bound on the capture time, is signed by the sensor before it leaves the hardware. The phone's security chip then adds its own signature on the metadata that exists alongside the pixels, and an upper bound timestamp is added shortly afterwards. That device-specific signed bundle sits on your phone as a secure “digital negative.” The concept is a deliberate echo of film and the old film-lab idea that, in case of doubt as to an image’s authenticity, you looked at the negative.

Later, and only if you choose to, you can “develop” that negative, so to speak. The phone sends it to Apple’s Private Cloud Compute servers, which check the signature, ensure that the sensor and device are genuinely paired, apply the minimum processing needed to make it viewable — the de-mosaicing, tone mapping and compression the camera skipped before — and then sign the result with Apple’s own signature, built to stay secure for decades. The same Apple signature appears on all Reference Images. You end up with two images: your normal photo, which you can edit or process however you like, and the Reference Image, which shows what the sensor actually captured. You can toggle between versions on the device (and Apple promises with third party viewers in the future). If you’ve erased, added or changed something in the main photo, whatever you’ve removed or edited, for whatever reason, is still there in the Reference Image. Once developed, the negative itself is moved to deleted photos and purged after 30 days unless you rescue it.

Apple deserves credit on some key elements which have been longstanding concerns for those of us working in this area: A capture approach that is the most reliable yet seen for ensuring a photo shows reality, an approach that does not link image trust to identity, and a robust capacity to respond to a failure in the authentication approach. At all three of these: ‘semantic authenticity,’ ‘resilience to compromise’ and ‘privacy preservation’ are articulated as core requirements for Apple in their whitepaper.

1. Strong, generally non-probabilistic defense of the photons that crossed the sensor, grounded in the sensor.

The verification is cryptographic, not a probability, and it's at the sensor level, not the software level. The camera signs the pixels as it captures them, the phone's security chip countersigns that along with the zoom and exposure (which come from software not the sensor), and a factory record proves both parts came from the same handset. Notably, Reference Image doesn't rely on gathering inaccurate and potentially privacy-compromising data such as location for verification. It works offline with a rolling timestamp that bounds a time in which it was shot, and is collected as soon as available. Then the image either checks out or it doesn’t, in the same binary way that C2PA manifests either validate or don't.

Apple runs additional fraud analysis on the raw sensor data as a second line of defense, scoring whether it has the physical characteristics expected of their sensors. This check is an AI model whose workings Apple keeps private. This is a defensible position, since in the world of manipulation detection, as in fraud more broadly, making this public would open it to attack. If a series of photos from one camera drift far enough from what Apple's sensors should physically produce, all images from that sensor can be revoked. Both the single image assessment and the running score are probabilistic layers, a confidence score, sitting alongside an otherwise binary claim.

Their technical whitepaper does not specify if their fraud analysis also includes looking for the common attack of screen recapture images (when a photo is taken of an existing image or screen). This has been a part of fraud defense in the capture app ecosystem for a while.

Strong sensor level work is a real win here and an addition to the ‘defending-reality’ ecosystem (more on that below).

2. Strong privacy and a clear design goal around avoiding identity as the root of trust.

The Reference Image doesn’t publicly identify the user, their phone, or their sensor. Every Image is signed with the same Apple signature, rather than anything specific to you, your device or your camera when it emerges from the Private Cloud Compute. I appreciate how Apple’s technical paper makes this an explicit design goal. It has been a longstanding concern of activists working with vulnerable defenders and civilian journalists that it should not be necessary to give up anonymity in order to prove an image is authentic. Identity disclosure should not be a prerequisite for trust.

Apple commits to a core design requirement that given two Reference Images taken with the same iPhone, nothing in them lets you work out that the same device took both. This addresses a kind of risk my former WITNESS colleague Jacobo Castellanos pointed to (manifesting in slightly different ways) in a 

recent report on C2PA implementation risks

 where usage of the same device or of a dedicated software traces back to a vulnerable individual. However, Apple can still make that link, because it needs to in order to revoke a compromised sensor (on the risks of that, read on). Third parties can’t, and Apple told me it doesn’t connect sensors to individuals.

One caveat — there is the option outlined in the whitepaper to share the negative directly in ‘undeveloped’ state. While presented as an option for professional photographers, this also raises a vulnerability since this negative hasn’t been through the process that strips out device-specific material. It carries certificates unique to that sensor and that phone, with the consequence that anyone holding two negatives can tell they came from the same device, with no help from Apple. With a Reference Image, that information sits only in Apple’s own records.

This privacy approach extends to the viewer of images. One concern, cited by Jacobo Castellanos, in provenance approaches such as C2PA is “viewer surveillance via the validation infrastructure” where viewers of sensitive material may implicate themselves by looking up verification data via an external server query, creating “a behavioral record… which content was verified, by whom, from where, and when.” Apple has taken steps to minimize the risks of looking at a Reference Image by using on-device lists to confirm an image is still a trustworthy record rather than by using a look-up online.

Apple can revoke an individual Reference Image, or every image from a specific sensor, if something goes wrong. Just recently, there was an 

incident showing a hack on the Google Pixel approach to verification

 that the tech giant couldn't fix. Building confidence in knowing what is real and what is synthetic is going to be an uphill education battle if the tools for trust keep getting broken without recourse. Apple has a clear recourse built into their approach; the question is who controls and governs this option, to which I return below.

Apple can retroactively revoke a single image, or every image a given sensor ever produced. If it does, photographs already published and already in other people’s hands stop showing as verified. That capability exists for good reasons, and something like it is necessary if a sensor is ever compromised (as with the case noted above 

However, it also means the durability of evidence collected by iPhone users depends on the company’s ability to resist pressure to withdraw an image or a set of images, and to provide redress for mistakes. There are big questions around revocation — 

Do we want that power with Apple? Could they be compelled to do this? And who knows when this happens, and can they challenge it?

 The manufacturing and revocation records that make revocation possible seem to imply that Apple’s “we don’t link sensors to individuals” assurance could be more a policy commitment than a technical impossibility if the rubber hits the road. For a journalist protecting a source or a human rights documenter working under an authoritarian government whose images are being challenged, that distinction of policy versus technical does matter. From experience of how data and transparency matter in such contexts with what amounts to 

, we need to be able to see the numbers, decision-making and appeals process, and we need public transparency reporting for decisions.

As I note above, Apple has a powerful design goal that I have argued in favor of for over a decade: you should not have to give up anonymity to prove an image is real. And to a third party, a Reference Image (but not a shared negative) is genuinely anonymous. There is no photographer credential attached, and no way to tell that two images came from the same phone.

But this anonymity is not absolute: a Reference Image creator’s identity is not known to consumers and viewers, but Apple has options. Apple’s own paper describes a revocation service holding a record of every developed image against the sensor that made it, and the device manifest binds that sensor to a particular handset. What stands between a sensor and a named person is Apple’s business records and Apple’s choice not to connect the two, not mathematics. For most people that distinction is academic and probably irrelevant. Not so if you’re documenting a security force repressing a protest, and worry that they may come looking for you or might have an interest in discrediting your work. Two potential risks appear to exist here. Apple doesn’t see the photos you develop from negatives in the Private Cloud Compute, but developing a Reference Image does create a record tying it back to a sensor, so if Apple later was handed a public Reference Image it could confirm the device it came from. More dramatically revoking a specific sensor-device discredits everything that phone ever shot in Reference Image mode all at once. In both cases, Apple would use the same identifier that lets your phone check whether an image has been revoked to find the sensor.

3. Apple has to be in it for the long haul if all roads go through it.

The negative provided by Apple Reference Image is raw sensor data and isn’t viewable on its own, so every image you actually intend to use has to be ‘developed’ through Apple’s servers. 

 is seriously engineered privacy infrastructure and I don’t doubt the confidentiality guarantees. Apple has also looked ahead to the upcoming post-quantum cryptography cliff. They’re using a signature that is designed for a world where current signatures are broken by quantum computing, on the premise that the evidence of authenticity must last beyond the next 5-10 years.

But it’s still a huge dependency at the most fundamental level, and it’s unclear what happens to undeveloped negatives or even validated Reference Images if the service is discontinued or is blocked in a country. This may matter more than it first looks, because the negative is deleted by default once developed. What is left is the Apple-signed image, not the sensor-signed original. Evidence should not have a possible expiry date set by a product roadmap, even from a company as dominant as Apple and with a team that seems committed to the need.

Apple points to the capacity of experts to verify the actions of the Private Cloud Compute; the stage that turns a negative into an Apple-signed Reference Image. We will need good answers on how the system is suitable and usable in key situations like evidence — if this becomes the norm, 

how does the black box become coherently transparent and legible

 I co-authored at the human rights organization WITNESS.

Sharing Reference Images is per-image and opt-in. Capture the proof when you want to, disclose it only when you decide to. This differs from the always-on version of ‘AI or real’ based on C2PA on the Google Pixel.

Apple doesn’t explain in the technical paper why capture has to be opt-in, but being optional is good from a user-control and privacy perspective, since there are real trade-offs in an always-on service, particularly the risks of retaining many ‘negatives’ that contain device-identifiable data.

However, experience with Guardian Project and WITNESS' ProofMode as well as other crisis documentation tools has found that if people, such as civilian witnesses in a crisis, have to specifically switch into a new mode to take a critical photo, they often miss that step. This is not an unsolvable problem, but Apple will have some educating to do.

5. Selective and unevenly distributed ‘this is real’ tools bring their own challenges.

Reference Image launches on the main cameras of two premium phones as an opt-in mode you have to remember to use. Meanwhile, we're in a strange moment of transition. Long term, I worry about real images being dismissed as fakes, but I also worry about real photographs being dismissed because they aren't credentialed as real. This applies as a concern to both images with Content Credentials and C2PA metadata, and going forward also to Apple Reference Images.

It is good that Apple intends to open this feature up to 3rd party apps for verification and capture in the future, but in the near term, most of the world’s cameras won’t have this technology, and many of the people doing the most important documentation, witnessing and journalism are shooting on cheap Android phones. Apple’s user education over the coming year needs to lead, perhaps perversely, with what a photo with a Reference Image marker means, but also what the absence of one does not. Otherwise we may see the cultivation of a 

perverse ratchet effect of expectations of photographic proof 

that the majority of image-takers in the world cannot meet.

6. ‘Real or AI’ is a journey, not a one-off, and is evolving in a bigger world than Apple — where is the C2PA collaboration?

The final question is the most significant one. Apple’s answer to the fact that real photos get edited is elegant: keep the untouched negative, and let people compare. When you toggle between the two you see exactly what changed (Google's Pixel phone does something similar via C2PA, though with a less robust confirmation of the original if the exploits noted above are not fixed).

That works well for the case where someone hands you both files. It’s strong for a professional photographer or first-hand documenter who gets to hold onto their original, and challenge a fake that circulates. It’s valuable for some specific business cases - for example, an insurance claim. It’s weaker as an ecosystem answer. It needs a human to look and judge. It says nothing technically transparent about which changes were made and how.

And perhaps more importantly, we live in a world where edits and evolutions of real images in the moment, and over time, via material and non-material use of AI and by multiple entities, across diverse platforms, and with a range of intentions, is the norm. This listing of the recipe of our complex digital creation, manipulation and consumption is a gap that C2PA and Content Credentials fills: a signed, machine-readable record of what each tool and entity did along the way. C2PA often talks about three legs of a stool — metadata to show the origins and the edits combined with watermarking and fingerprinting/hashes to reconnect the metadata if lost.

But C2PA has struggled with the challenges of showing ground-truth reality from when photons hit sensors. This is where Apple comes in. What I’d like to see is Apple’s sensor signature, which is the strongest capture attestation anyone has built, feeding into an open and portable edit history. We need a system in which this information can be handed off interoperably into other tools and other ‘real-AI’ recipes and concoctions, C2PA being one strong option now used by many others. Apple has already joined one ecosystem bet by

 to incorporate Google's SynthID for watermarking AI images and edits later this year (a critical step to industry-wide adoption). Let's see what other collaborations are possible to defend reality. Putting watermarking, robust metadata recipes readable by all, and a best-in-field privacy-protecting sensor-based attestation together would be a powerful combination.

Some thoughtful places where Apple could take this forward include:

Collaborate and interoperate with C2PA so negatives and Reference Images can be easily developed, verified and then imported into C2PA-compliant apps, so that edits after capture documented in a form many others’ tools can read, and a powerful sensor record can be part of a recipe we can all read showing the mix of reality and AI in what we see.

Make verification of a developed Reference Image (and additionally, confirmation if it has been revoked) possible outside the Apple ecosystem, as soon as possible — as they've indicated is the intention.

Third-party capture APIs asap would let developers building for high risk human rights and journalism work reach the same hardware guarantees.

Revocation governance matters: publish what’s retained, for how long, who can compel it, and whether targeted revocations are disclosed.

Be loud, in the rollout, that a photo without a Reference Image is not a suspicious photo (people won't all have iPhones and not everyone remembers or chooses to use these modes)

Human rights activists and citizen journalists first called out the need for a robust method to conclusively prove that a photo showed what it said it did. We saw how easy it was to dismiss reality even before AI forced every single one of us to doubt the evidence of our eyes.

We now have an option to do this on one of the most powerful cameras, with some of the best privacy-preserving technology, and from one of the leading hardware companies in the world. But this technology cannot be an Apple-only island. We need to be able to use this ground-truth proof in a broader world of reality-determination, and whimsical, communicative and malicious AI edits that spread across messaging, social and search. Accessibility, interoperability, transparent governance and a recognition of the realities of a world where absence of technical proof can and will be leveraged against the most vulnerable accounts are critical next steps to build on this key advance.

Why are so many people across the political spectrum resisting AI-powering data centers and Flock’s network of AI-infused surveillance cameras? The national movement—lawsuits, legislative lobbying, protests, and sabotage—is striking because Americans have already ceded so much privacy and autonomy to tech companies. We largely complain and shrug from one techlash to the next, while increasingly invasive technologies are 

. Data centers have been around for decades, and privacy scholars and activists have long warned about the dangers of automated license plate readers (ALPRs) and community surveillance cameras. But until now, neither had the heightened level of visibility that makes people less willing to tolerate their harms. Salience is the driving force behind data center construction projects getting 

; and municipal and agency contracts with Flock being 

The most common explanation for why the harms associated with Flock cameras and data centers have become so salient is 

. Much of the writing about Flock and data center activism portrays them as parallel protests motivated by festering resentment against the perceived injustices of an increasingly powerful, consolidated tech industry that receives too much support from an overreaching government. In this spirit, The Verge editor Nilay Patel 

, “data center construction has become a proxy for the public’s feelings about AI,” and former Federal Communications Commission chair Tom Wheeler calls data centers “

the symbol of citizen frustration about AI

.” Relatedly, journalist Amanda Marcotte 

, “Flock cameras have become a symbol of growing anger over the efforts by technology oligarchs to impose their dystopian fantasies on the country.” Likewise, writer Jacob Richards 

 “the resistance against Flock cameras is just part of a larger anti-tech populism.”

The populist angle makes sense. Across the country, people are worried that the White House and Silicon Valley are moving too quickly to develop and deploy artificial intelligence. A recent Gallup and Bentley University 

 found that in 2026, 39% of Americans “believe AI does more harm than good,” up from 31% in 2025. Even Anthropic CEO Dario Amodei 

, “Ordinary people don’t trust companies, governments, or the tech industry and always suspect that we are cooking up some new way to screw them over.”

The populist framing also helps explain what resisters have in common. They might be worried about the surveillance state or their family’s utility bills. They might want modest policy changes or a sweeping ban. Whatever their motivation or goal, they share a sense that AI is being imposed on them—something they didn’t ask for, weren’t consulted about, and associate with people who have far too much sway over their lives. As psychologists have pointed out, direct threats to freedom or autonomy make people more likely to 

. And with each new public act of resistance, people who 

 that others accepted the status quo become more emboldened to make their voices heard.

DeFlock privacy advocacy sticker warns of a nearby Flock Safety AI-enabled automated license plate reader tracking vehicles, San Francisco, California, September 9, 2026. (Photo by Smith Collection/Sipa via AP Images)

The populist framing does a good job of capturing the pervasive frustration with a lack of democratic consultation and accountability, but it doesn’t fully explain why the harms associated with Flock cameras and data centers have become so salient. To understand this better, we need to consider three aspects of normalization. First, people 

 by determining that immediate or personal benefits are a fair tradeoff for abstract or distributed harms. Second, once acclimated, people have a new baseline for assessing tradeoffs, and are more likely to weigh new immediate, concrete costs more heavily than 

any distant, abstract benefits they may be presented

 with. And third, each new tradeoff where people are willing to give up a “small” loss of privacy or autonomy for some benefit—something scholars Woodrow Hartzog, Johanna Gunawan, and Evan Selinger call “

”—desensitizes them to salience of cumulative negative effects. 

ALPRs and surveillance cameras have existed for years, but recently, Flock’s presence has grown rapidly across local communities. That expansion can be viscerally jarring. There are new cameras along roads, near frequented businesses, and sometimes in public spaces close to private residential property. The increased presence of the hardware makes visible the larger network of ongoing, concentrated surveillance, largely for law enforcement. Furthermore, the promised benefit of improved public safety pales in comparison to the numerous documented instances of law enforcement 

acting on unverified or incorrect information

 serious consequences for innocent people

. People, therefore, are not only feeling a loss of control over their autonomy but also over their ability to understand how and when they are being watched, by whom, and for what purposes. As law professor Andrew Ferguson stresses in 

, today’s mass surveillance systems are flipping the default from law enforcement only investigating you if you do something suspicious or illegal to everyone is under suspicion and worth tracking.

Let’s compare this situation to doorbell cameras. People are so acclimated that most ignore them or barely register their presence as background noise. A doorbell camera promises a simple bargain: give up some of your privacy, but in return you’ll receive more security. The Nancy Guthrie kidnapping case perfectly illustrates the acceptance of this tradeoff. Guthrie, mother of NBC News journalist and Today co-anchor Savannah Guthrie, did not pay for her Google Nest doorbell to be linked to the company’s cloud service, which allows video footage to be retained for 10 days beyond local storage. Nevertheless, footage from over a week earlier 

was recovered from Google’s backend system

, and there was little public concern that its retrieval violated the terms of the product’s policy. This lack of concern may reflect how normalized people are to pervasive data collection, where the ongoing retention of information is accepted, even when it goes beyond the expected scope and scale. And of course, the fact that this extended data retention provided footage of the attackers makes the tradeoff for this additional “nick” easier to accept.

 is another important example. Here, too, people accept the tradeoff as they primarily see the cameras as tools for protecting private property and family, as well as their immediate neighborhood. The full power of surveillance network effects wasn’t staring them in the face until they saw Amazon’s Super Bowl commercial, “

.” The Search Party feature prompts nearby Ring cameras to scan their respective areas for AI-powered matches of lost pets, and it was released in 

 with little to no public pushback. Fast forward five months to a commercial that made it easy to see what a coordinated network of AI-infused Ring cameras looks like, and there was massive outcry. As Senator Ed Markey (D-Mass.) 

, “Amazon inadvertently revealed the serious privacy and civil liberties risks attendant to these types of Artificial Intelligence-enabled image recognition technologies.” In other words, the commercial helped people visualize what privacy scholars and advocates have long pointed out and which Jamie Siminoff, Ring’s founder, 

Now, did the backlash have any impact? In the immediate fallout, Amazon announced it would

 not continue with its proposed partnership

 with Flock Safety to share data with law enforcement, which was unrelated to the Search Party feature. The larger connected network of Ring cameras still exists, and the Search Party feature remains active for customers. One might wonder why the outcry didn’t spark a mass movement to cancel Ring subscriptions in protest. The answer could be as simple as people did not want to spend time researching whether there are more privacy-friendly alternatives on the market. But normalization probably also played a role. People were already willing to give up some privacy for more security, and perhaps the canceled partnership with Flock convinced them that features like Search Party only slightly diminish privacy in comparison. With that reassurance, they didn’t feel compelled to give up a technology they were already acclimated to and reliant on.

As with Flock cameras, data centers create a similar visibility problem that affects how people perceive this tradeoff. Until recently, many people have relied on cloud storage, streaming services, social media, and AI chatbots while remaining largely unaware of the material infrastructure that supports them and the harms that come with it. But that salience is rapidly changing, and the once invisible, weightless digital economy has finally come back to earth. That’s because the push to build sprawling data centers across the country has imposed many locally experienced burdens, from high electricity and water demands to significant increases in light and noise pollution. And for those worried about surveillance technologies like Flock cameras, data centers have taken on new meaning and urgency because they are now easier to see as surveillance-enhancing infrastructure. 

The opposition to new data centers and Flock cameras tells us something important about why normalization isn’t inevitable. Normalization seems more likely to occur when the costs and power relationships associated with a technology are abstract, easy to ignore, and dispersed. But once the harms become concrete, locally observable, and a shared loss of control feels too strong to ignore, the spark for change gets lit. Given these dynamics, whether resistance spreads to other things will depend on whether people recognize that with each new “nick” that is normalized, surveillance will continue its structural shift from being the exception to becoming the default condition of everyday life.

The landmark antitrust trials against Google appear to have ended with a whimper. Over the past two years, 

 have ruled in separate government cases that Google maintained its monopolies in search and digital advertising tools (or “ad tech”) through improper and illegal methods. (The Open Markets Institute, where I work, 

 in support of the government in both matters.) While Google has been a relentlessly competitive firm, its competition, as two judges concluded, has not always been fair.

The judicial remedies, however, have not matched the severity of the offenses. In an 

 released to the public yesterday, Judge Leonie Brinkema declined to break up Google in the ad tech case by ordering the sale of its digital ad subsidiaries. And so, two jurists have now ruled that Google broke the law and yet refused to take away the company’s dangerous competitive weapons.

Google revolutionized how people search for information on the internet. In the late 1990s, there was an abundance of search engines: remember AltaVista, HotBot, and Lycos? Google joined the scene and quickly distinguished itself. It offered a clean interface and excellent results. Enter a keyword or string into Google and see links to the ten most useful results on page one. It eclipsed its rivals and eventually pushed them to effective irrelevance. Google competed and captured the search market by making a superior tool, becoming an everyday verb in the process.

For a time, it appeared to represent the best of American capitalism. On top of its socially beneficial innovation in search, Google, at first, refused to embrace the 

 business model that was taking off at the turn of the 21st century. Instead, it made money through contextual ads tied to search queries. Google seemed to be practicing its corporate slogan, “

Its ensuing history, however, shows that competition is not always good. As media scholar Matthew Crain writes in his book 

, Google adopted surveillance advertising in the wake of its 2004 initial public offering and in response to pressure from Wall Street to develop new revenue streams. It tracked users across its properties to serve them targeted content. According to the US House Judiciary Committee, the company undertook 

 between 2001 and 2020, in part, to enable it to expand and perfect its surveillance machine from search to the rest of the web and eventually all facets of life. Google bought businesses like YouTube, DoubleClick (subsequently renamed DFP), Android, and AdMob. The company that originally won by developing a better search engine was increasingly competing through surveillance, cross-platform data harvesting, and acquisitions. The “don’t be evil” motto was ringing hollow two decades ago.

The US government’s antitrust victories against Google over the past two years illustrate how the corporation has maintained dominant positions through unsavory competition.

In the first Google case, which the first 

 and 11 state attorneys general initiated in October 2020, Judge Amit Mehta ruled in August 2024 that the world’s leading search provider didn’t compete by continuing to improve its tools. Instead, it battled rivals by collectively paying Apple, Samsung, Verizon, and other partners billions of dollars every year to exclusively pre-install Google search products on the devices they made or marketed. For instance, in exchange for default placement of Google search in the Safari browser, Apple received an estimated $20 billion in 2022 alone. Rather than concentrate on increasing its functional superiority over search alternatives, Google effectively paid handset makers and wireless carriers huge sums of money 

 to do business with competitors. Thereby, it ensured Bing, DuckDuckGo, and others would remain marginal players.

While the company used carrots to shut out search rivals, the ad tech case, filed by the 

 and eight states in January 2023, showed Google employing sticks as a competitive weapon. Publishers, such as much of the news media, make money by selling space for ads on their sites. Among other unfair practices, Google compelled them to use DFP to access its must-have ad exchange AdX, which connected them with advertisers like Coca-Cola and Nike.

, Judge Brinkema repeatedly called Google’s bundling of AdX and DFP “coercive.” Google did not compete and attract business to DFP by offering better terms or improving its quality. It bullied publishers to accept DFP as part of a package and to forgo the use of rival tools. In the judge’s words, Google “compelled publishers to use DFP, not because they viewed it as a superior product, but rather due to Google’s exploitation of its control over AdX’s preeminent position in the open-web display ad exchange market.” By unfairly excluding rivals, Google maintained a lucrative chokepoint over online advertising and extracted monopolistic tolls on display ads across much of the internet. This market exploitation robbed publishers and advertisers alike and left even less money on the table for ad-supported journalism.

The two judges, however, refused to take away Google’s illicit economic weapons when crafting remedies for the corporation’s lawbreaking. Last September, 

 rejected the government’s request to prohibit Google from paying firms like Apple for default installation of its search tools. He cited the harm to distribution partners—Apple, Samsung, and Verizon are well-heeled businesses and would surely manage just fine without large payments from Google—and the purported precariousness of Google’s search monopoly in the face of AI competition. And in her written opinion yesterday, 

 stated she would not strip the company of its coercive power in ad tech by ordering the divestiture of AdX or DFP, even though the 

 called divestitures while“drastic” also the “most effective, of antitrust remedies” and stated that hardship to defendants cannot be the basis for denying the government effective structural solutions.

Both judges displayed a dual quality of clarity and confusion in their rulings. They recognized Google’s use of improper competition in their liability decisions but declined to excise its instruments of lawbreaking in their remedial orders. Despite being found to have broken federal antitrust law in two cases, Google generally gets to keep its unfair advantages in the market. As a result, in the words of a 1947 

, in both cases, “the Government has won a lawsuit and lost a cause.”

Over the weekend, Anthropic CEO Dario Amodei published a new essay, titled “

” Amodei argues for slowing down the pace of AI development, and offers a three-part plan for industry regulation. OpenAI CEO Sam Altman and SpaceX CEO Elon Musk have both responded favorably to his proposal.

For years, I’ve made a habit of dissecting manifestos and missives from tech CEOs. I shared my early reactions to Amodei’s 

. There are a few points that deserve further elaboration. Chief among them: 

should pace the frontier? Should the AI industry be permitted to police and regulate itself?

Let me start positively. Amodei writes that “A race to the bottom, spurred by commercial incentives, can make [the risks of AI doing serious harm] more acute.” Yes, absolutely.

Nothing about the AI future is inevitable. As 

, the arc of any emerging technology bends toward money. Regulations shape market incentives and social behaviors, rewarding certain technological uses and punishing others. The current status quo, featuring a handful of unencumbered tech companies in an outright race to win market share and attract investor cash, is a recipe for the worst possible outcomes.

But it’s also worth noting that Amodei’s company is currently winning that AI race. Anthropic is preparing for its own IPO, boasting about a 

 of profitability, all while beating OpenAI at its own game. Even if we grant that Amodei’s concern is genuine (and I think it is), it clearly takes a back seat to his commercial ambitions. He is not worried enough to delay going public.

read this essay. If you believe Amodei is 

, then you will read it as an act of responsible thought-leadership. If you believe he is another huckster, you will read it as more hucksterdom.

For the moment, let’s grant Amodei temporary status as the 

. I believe he has a conscience. I believe 

 the things he is saying. And the same simply 

or Elon Musk. But just as Effective Altruism begins from good intentions, and then eventually arrives at ‘the most effective form of altruism is 

,’ those good intentions are easily warped in utterly predictable ways.

At a high enough level of abstraction, Amodei’s suggestions all sound appealing. He proposes (1) instituting mandatory third-party evaluation and monitoring at all frontier labs, (2) developing a (self-) regulatory regime that all AI labs in democratic countries commit to, and (3) bilateral coordination with China on AI model development. These are all potentially good ideas. But the details matter quite a bit. Amodei’s scheme for pacing the frontier amounts to a series of proposals that all benefit Anthropic. These proposals are all subject to revision if-and-when they get in Anthropic’s way. We should read him accordingly.

And the fundamental problem is this: Dario Amodei believes we ought to trust the AI industry to effectively regulate itself. He has consistently called for regulation, but only of the sort that gives Anthropic a comparative advantage within the industry. He believes the government should mandate the same testing and auditing practices across the industry that Anthropic has already instituted. But, as 

, the Federal Trade Commission could be much more aggressive in enforcing regulations and holding the industry liable for the harms caused by its products. It is a mistake to let the AI industry shape the contours of its own regulatory system, even if we grant the good intentions of its leaders. We know from history exactly how that will turn out.

Take his proposal for embedded evaluators. Amodei writes:

 Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as 

), whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes. This is the key step for 

of any pacing commitments, and has precedent in the banking industry, which sometimes involves regulatory “supervisors” embedded along with employees. Anthropic is unilaterally committing to this step now. We intend this to be part of a broader push to redouble efforts on our safety and alignment work.

Amodei writes that this “has precedent in the banking industry, which sometimes involves regulatory “supervisors” embedded along with employees.”

But the thing about the banking industry is that these regulatory “supervisors” are only there because the government requires them. Without the force of government oversight (and the looming threat of draconian fines), this sort of monitoring doesn’t work.

Imagine how this would actually work in practice: these third-party evaluators are meant to have desks and office space at Anthropic. They’ll examine the same models, write their own reports, and offer a “second opinion free of commercial incentives.” Who pays their salary, and how competitive is that salary? I keep seeing advertisements from Anthropic for entry-level employees in the range of $400K. Top-level engineers are attracting eight- or nine-figure salaries from the company’s competitors. Either these third-party monitors are going to make similar salaries — provided by the very same funders who are deeply invested in Anthropic — or they are going to be little more than glorified apprentices. If you have a desk at Anthropic, and you are analyzing the same data as Anthropic employees, and you have the same skillset as Anthropic employees, and you are paid 1/5th what Anthropic employees are paid, then you will eventually apply for a job at Anthropic. Third-party, independent evaluators will either become a revolving door to the industry, or else they’ll be utterly outmatched.

The scenario that Amodei is describing is less like the banking industry post-2008 and more like 

Arthur Andersen’s relationship to Enron

. In the absence of government enforcement, independent monitoring gets captured by industry, particularly when big money is involved.

Latest

Podcast

Recent

Our content delivered to your inbox

Thank you!

Around the globe