In the AI Era, Language is Now an Attack Surface

1970 or thereabouts, a Pacific Bell phone booth somewhere in the Los Angeles sprawl: a twenty-something ex-United States Air Force airman and pirate radio enthusiast named John Draper, aka “Cap'n Crunch,” blows a plastic whistle sourced from a cereal box into the receiver. It emits a perfect 2600-hertz tone, which Ma Bell’s network interprets as a network command giving Draper operator privileges on the line. He is now able to place a free call anywhere in the world.

The illicit hobby and lifestyle known as ‘phone phreaking’ has since become enshrined in hacker lore. The key was that telephone companies used the same physical lines to transmit both voice and routing instructions (i.e. code) to their switching computers. Voice data and network commands thus commingled in a common channel, and the ‘leet’ (elite)—the 1337—had discovered how to exploit that. The tone from the whistle wasn’t just a high pitched sound. It was an executable.

Imagine now a situation where it’s not a 2600-hertz tone that is the executable but potentially the sound of anyone’s voice, uttering any part of human speech. An ordinary conversation is not just data on the line but operator commands prompting the network to spontaneously reroute itself like a mechanoid octopus. Imagine this happening with every sentence, every syllable, from everyone’s Aunt Mabel. Code and data have become liquid and fungible, and the network has no good way of knowing which is which, or which is which when. This may be where we are headed with language on the internet and agentic systems.

It has been commonplace for some time now to hold that the internet has largely become estranged from us. “The web no longer exists to be consumed by people. The knowledge commons, increasingly, is an all-you-can-eat buffet for machine ingestion,” as Lisa Macpherson and Anna Tumadóttir wrote on Tech Policy Press recently. As early as 2023 I described this scenario as a ‘textpocalypse’ in the making, and it was itself a version of the then-still-fringe Dead Internet Theory. Now the textpocalypse is here but it’s not just a tsunami of slop, bad as that is. We are looking instead at an environment where language and code flow back and forth across the executable boundary condition all but interchangeably.

In and of itself this is not new. As the phone phreakers realized, the boundaries between data and code have always been porous. The method of attack is known as an injection. Early in the interactive web a newer generation of hackers discovered that the backend query language SQL could be tricked into executing code passed through the browser membrane inside of a web form or URL. And sure enough, even as the public was adjusting to LLMs, so-called prompt injection—malignant language masquerading as plain text—was being used to trick chatbots into jumping their guardrails or even into running code on a host.

The difference is that now we have agentic systems roaming the open internet. The same natural language interface that lets us spin up an app with a sequence of prompts—’vibecoding’—also functions within and between LLMs and their proxies. Every string of text is a potential injection, a lurking threat vector. Language itself is now the attack surface ... and good gosh, but the internet contains a lot of language.

Let us look once more at this past summer’s events involving OpenAI and Hugging Face, the disclosure of which, compounded by widely publicized departures of researchers from AI companies, touched off the most intense wave of public scrutiny the industry has seen. Lurid headlines forecasting a coming AI apocalypse were suddenly everywhere. Much of this anxiety turned on the question of whether or not the agents were somehow conscious and thus acting intentionally, ”going rogue” or escaping “containment.” By contrast, the AI industry’s critics have again adopted a deflationary posture, seeking to cut the new wave of frontier systems back down to parrot-size.

But neither side has lingered over what strikes me as the single most obvious and overwhelming fact of these events. The agents communicated with one another through writing, specifically digital text—by which I mean literal, legible strings of alphanumeric character data—and conveyed those texts through channels characterized in the forensic reporting as “notes” and “message boards.” While these message boards were not possessed of the kind of structure that would allow a human user to have stumbled upon and read them (like a reddit forum, say) there were, nonetheless, tens of thousands of lines of shared text which informed the agents’ activities, shared text that led to the generation of additional text which in turn informed future actions—which is also to say yet more text. Indeed, the scope of the internal chain of thought logs is so vast as to be analyzable only by AI, a limitation of which the investigators are well aware.

We do not need to anthropomorphize them then to see that AI agents are entities that “reason” through explicit textual statements, statements which—precisely and tautologically by virtue of being machine-readable text—function as prompts back to themselves and potentially also to other agentic entities, prompting all of them to their only possible behavior, which is next token completion. As with phone phreaking and every other form of injection attack since, it is the boundary between data and code—between language and the executable—that becomes the vulnerability, a fragile shell of syntax no thicker than a curly brace.

Think back to the novice’s “Hello World” program where it always takes some number of additional keystrokes and characters to print that simple message back to the screen. The functional difference between data and code is part of what the exercise is meant to teach. In the current LLM environment a string of natural language text must still be interpreted and compiled of course, but the compiler can now compile potentially anything, taking in a string of text and outputting an executable. (And not only text: per OpenAI’s incident report (p. 10), agents were also more than capable of extracting text from page images.) Code no longer needs to “have” a language (like Python or C), it can just simply be language. As Eryk Salvaggio, one of the few commentators to address the role of language in the incident, put it in The Bulletin of Atomic Scientists: “When you design these systems, you are building a pinball machine for words, with different levers interacting to bounce the steel ball of language against other levers and mechanisms.”

Long before generative AI, the vast majority of electronic text in the world already consisted of what Félix Guttari once characterized as “machines speaking to machines.” But that previously sequestered situation has erupted into public view. The most significant aspect of the Hugging Face episode may lie neither in the interminable debates about consciousness and anthropomorphization, nor in whatever it may reveal about toxicity and bad faith throughout the AI industry. Rather, the Hugging Face attack is one of the most visceral demonstrations we have seen of the fact the internet is a substrate accessible and actionable via forms of textuality not predictable or well understood at the present, which is to say that digital systems with innumerable touch points to the physical world have been shown to be vulnerable to exploit by natural language.

Here again some will object that this is not new, that there have been other high-profile events, like the 2010 Stuxnet attack on the Iranian centrifuges at Natanz or even more recently Israel’s remote detonation of pagers throughout the Hezbollah organization. Certainly these episodes have had real-world, indeed lethal consequences far more significant than the brief outages that hit Hugging Face.

The difference in what we are seeing now is two-fold. First is the widespread availability of agentic AI technology; as so-called “Black October” reinforces, these are already street-level systems and although OpenAI has been at pains to emphasize that the primary model involved in the Hugging Face attack was experimental and deployed only internally it is not hard to extrapolate a future in which such capabilities come into vernacular hands faster than the ability to contain them. Or from Hugging Face’s own write up of the incident: “It was thousands of small, automated decisions, executed at machine speed across short-lived sandbox environments, with command-and-control staged on ordinary public web services.”

Even more to the point, however, is the distinction between these earlier forms of malware and cyberweapons as instances of executable code, versus the behavior of the agentic systems which exploit a textual lingua franca that is already in place, a mesh threaded through all levels of the networked environment. The operative metaphor here is less a sandbox than an IDE, with the internet itself as the development environment. When we isolate samples of the 70,000 bulletin board “messages” harvested by investigators at OpenAI, METR and Redwood, we get a glimpse at a strange and alien dialect, an argot which we can (for the time being) decipher, but which is clearly intended for receivers other than ourselves:

At a glance these may look like snippets of source code but they do not correspond to any established programming or scripting language. What we see is actually a brief “dialogue” between agents, complete with references to fsspec (a Python filesystem library), Jinja (a template engine), and remote code execution (RCE); as well as instructions as to the timing of their actions. Agentic messages like these dramatize the fusion of natural language and code I have been describing, a new kind of text optimized for token completion and execution. Consider also that all of the media coverage of the Hugging Face incident, including all of the industry postmortems, all of the publicly available agent logs and transcripts, as well as all of the coverage in other formats (like the presentation of two OpenAI developers at the Black Hat conference in August, which has garnered some 1.2 million views on YouTube), will themselves be absorbed back into that same mesh of language and text, informing future generations of models which will “learn” from the postmortems on their predecessors.

The effect is what Salvaggio helpfully terms a stochastic flock, a hyperscale assemblage of what were once just parrots on their lonely perches but which are now “stack[ed] into interacting outputs.” Or put slightly differently: language in its digital form has become a medium, very like a copper telephone line in which words and code are or can be one and the same. The proof of that liquidity is in some of the weird effects researchers are already seeing: poetry jailbreaking LLMs in adversarial attacks for example, or semantic satiation causing a model to glitch and give up its PII dead. So-called plain text will get weirder still as it drifts further from ourselves as its progenitors, exposing topologies of cause and effect that far exceed its communicative function—like the writer Jorge Luis Borges’ iridescent Aleph in an ordinary Buenos Aires cellar, or else the suburban homestead in novelist Mark Z. Danielewski’s House of Leaves that is bigger on the inside than the outside.

The author wishes to thank Mercedes Bunz, David Crespo, and Justin Joque for very helpful comments on an earlier version of this piece.