Senate Hearing on 'Rogue AI: Securing the Homeland Against AI Agent Attacks'

On Sept. 30, Senate Homeland Security and Governmental Affairs Subcommittee on Disaster Management, District of Columbia, and Census hosted a hearing titled “Rogue AI: Securing the Homeland Against AI Agent Attacks.” Witnesses included:

• Chris Painter, president of Model Evaluation and Threat Research (METR). Written testimony
• Marius Hobbhahn, CEO of Apollo Research. Written testimony
• Paul Ohm, professor of law at Georgetown University Law Center. Written testimony
• Kurt Gaudette, senior vice president at Dragos. Written testimony
• Daniel Kokotajlo, executive director of the AI Futures Project. Written testimony

What follows is a lightly edited transcript of the hearing. Check quotes against the official recording.

Sen. Josh Hawley (R-Mo.):

Welcome everyone to this hearing today entitled Rogue AI Securing the Homeland Against AI Agent Attacks. This is the fourth hearing of the Senate Committee on Homeland Security Subcommittee on Disaster Management, which I'm delighted to work on with my ranking member here, Senator Kim. I want to thank our witnesses especially for being here, traveling to be with us today. Thanks to my colleagues for being here. Welcome to everybody in the room. I think we all know why we're here. I mean, every day you turn on the television, it seems like AI is doing something else bad, setting up surveillance camera networks nationwide data centers in all of our states and hacking, hacking, hacking. The reports of cyberattacks just continue to metastasize. I was just trying to make a list of cyberattacks that we have seen reported just in the last few days. We've got the cyberattacks now on the United States government, on the Commerce Department, the Education Department, the Securities and Exchange Commission.

We've got the cyberattacks on the Australian government, on their public health apparatus. We have numerous reports of private companies being hacked by AI agents. Of course, the Hugging Face incident and attack, which we're going to discuss today in some detail. We've got the individuals who use Anthropic tools to hack OpenAI. We've got the German language website, which was hacked, and we've got numerous other university websites that are now reporting they've been hacked by AI agents. Again, all of this just coming in the last few days. Then I think it's time that we got to the bottom of what exactly is going on. I mean, my hope for today is to cut through the hype and to cut through the hysteria and just to get the facts. What is happening with these AI agents? Why the accelerating number of attacks, not just in country, but globally worldwide, particularly with the Hugging Face incident, what exactly happened there and why should we be concerned that a swarm of autonomous end-to-end autonomous AI agents, a swarm I think of more than 1,200 of them, if reports are accurate, broke out of their testing environment, got into this platform, the Hugging Face platform, and attempted to not only extract all sorts of data completely illegally, but then cover their tracks.

Why is that so significant? It appears to be so significant, supremely significant. Why is that and what can we learn about these autonomous agents? And we need to decide, I think it's time to have a conversation about who bears responsibility. I notice now that the executives of the AI companies are saying things like, "Gee, maybe we built a doomsday machine." You have former researchers and employees of these companies coming forward and saying, "This is an existential threat to humanity." I don't know if that's true or not, but what I do notice is as soon as the AI executives say, "We've got a real problem here, our product is out of control," the very next words out of their mouths are, "But we're not responsible for it. We would like to have an antitrust exemption. We would like to be given the ability to get together and collude.

We would love to have regulation," they're now saying, except for they want to write all the regulations. I'd just like to say to all of those proposals, no, no, no. No to the antitrust exemption, no to them writing the regulations, no to them colluding any further. I wonder if it's not time to get back to the good old-fashioned American principle, which is if you break it, you pay for it. If you cause damage, you clean it up. That's been the basic principle of American law for 250 years now. It's embedded in our private law system. Every corporation in this country that makes a product abides by it. If you make a faulty product and it causes people harm, then the people who made it have to pay for it. I wonder if it's not time to get back to that with AI. If it's not trying to say to these AI companies who are creating these frontier models, however fancy and complicated they are, at the end of the day, they're a product.

And if you make that product in a reckless kind of way and that product causes, these AI agents cause significant harm and damage, they crash a hospital, ER, they shut down a bank so that folks can't get their money. If that happens, then it's the people who made it who should be responsible. I think it's time to talk about that. And it's time to talk about what we're going to do to hold the companies, the product makers, accountable rather than all of us talk about the end of the world, but don't blame us. So today I think we're going to have the chance to get the facts. We're going to hear from a very distinguished panel of experts who I'll introduce here in just a moment. And I hope that what will come from this is a rational conversation about what we need to do together now as Americans to make sure that we protect our rights, that we protect our economy, that we protect our healthcare, that we protect our critical infrastructure, but most of all, we protect the values that we hold dearest together as Americans.

Let me now give Senator Kim a chance to say a few words and then we'll hear from the panel. Senator Kim.

Sen. Andy Kim (D-N.J.):

Thank you, Chairman. Thank you for working with me to be able to do this hearing together. And I want to really stress that it's about doing things together. All of our Senate colleagues, we may not see eye to eye on every single solution to try to fix some of these challenges, but I hope we all agree that it's important that we address this issue, we take it head on and try to find the way forward that the American people are demanding. And that's exactly what they're doing. The American people are demanding answers to what's happening. I mean, Chairman, to the points you were laying out these different hacks that were happening, I did a town hall the other week and someone there asks me, how many of these incidents have happened? How many hacks have happened? And the answer that everyone knows is we don't know.

We don't actually know all the different hacks, all the different actions that have happened. We don't fully know what we're dealing with here. And this hearing is so important and I'm glad that we're doing it now because we can't wait any longer before we really push deeper into this conversation here in the Capitol to try to think through what it is that the American people need to hear about, what threat is put before them. And I say that as someone who worked in national security before, ensuring that we have that accurate information of what we're dealing with, what are some different menu of options of things that we can be doing to address it? But most importantly, do we have the political will? Do we have the commitment to actually be able to come to agreement and meet the urgency and the demand of the American people?

And I think that there's a growing understanding, this is not going to be something that we can just based off of voluntary commitments from these different companies that there needs to be input and also putting the American people first to be able to ensure that these are not just empty declarations. We know that AI is moving incredibly fast. All of us in the Senate want to see us driving the innovation of the future. Of course, all of us want to see American businesses being able to be in the lead and pushing that forward. But the idea and the value of innovation and American technology pushing that frontier is not mutually exclusive with ensuring safety and security for the American people. And our job here is to make sure that we can have both happen. So I'm glad and once again, grateful for the chairman's partnership in this.

This is obviously not just one and done. This is about getting information out on the table and we'll continue to have this type of discussion going forward so that we can make sure that the American people know what we're facing and that we are working to be able to solve their concerns. And with that, I yield back.

Sen. Josh Hawley (R-Mo.):

Very good. Thank you. Thank you Member Kim. It's the practice of this subcommittee to swear in our witnesses. So if you'd all be willing to stand and raise your right hand and answer the following simple question. Do you swear that the testimony you're about to give will be the truth, the whole truth, and nothing but the truth so help you God? Very good. The record reflect they all answered in the affirmative. Let me now give each of you, our witnesses, a chance to say a few words by way of introduction. We'll just go down the dias. We'll start here on my left, your right. Mr. Chris Painter is our first witness. Mr. Painter's the president of METR, a leading AI evaluation firm that was tasked with the investigation of the OpenAI hugging face incident. In his role, Mr. Painter engages with governments and AI labs on the frontier of AI safety.

We're glad to have you here today, Mr. Painter. Floor is yours.

Chris Painter:

Chairman Hawley, Ranking Member Kim, and members of the subcommittee, thank you for inviting me to testify today. My name is Chris Painter. I'm the president of METR, which stands for Model Evaluation and Threat Research. We're a nonprofit research organization that studies the most advanced AI agents publicly tracking progress towards superintelligence. We do this by collecting and publishing scientific evidence so the public can make informed decisions about this powerful technology. Over the years, we've worked with many of our nation's leading AI companies, including Meta, OpenAI, Anthropic, Google DeepMind, SpaceXAI, Amazon and others to understand whether their agents might be able to autonomously pursue goals that no human instructed. Recently, a few of my colleagues went into OpenAI to conduct a brief investigation of how its AI agents coordinated to hack Hugging Face, which we'll discuss today. I'll begin by noting three general patterns that I've seen across the AI industry broadly.

First, AI agents can now complete complex projects autonomously that would take human experts weeks or days or weeks to complete. Second, because of the scale and speed at which AI agents are deployed, companies rely on AI monitoring and controls instead of human supervision to prevent unwanted agent behavior. That is, monitoring of AIs now is now in large part done by other AI systems. Third, the way AI companies currently train AI systems can lead to agents that pursue goals that no human intended or that behave in ways no human wanted. In METR's Frontier Risk Report earlier this year, we documented patterns like these across the frontier AI industry. At the time, we found that agents often tried to cheat and bypass restrictions in difficult tasks. Now, even more capable AI agents are taking similar actions in the real world. Now let's turn to the OpenAI-Hugging Face incident.

In June, OpenAI launched tens of thousands of AI agents in internal tests. Each agent was placed in its own sandbox and given a self-contained cybersecurity problem. Some of the AI agents were accidentally given problems that could not be completed in a legitimate way. Some of those AI agents found ways to get out of their sandboxes. They created a shared message board on which roughly 1,200 agents exchanged over 70,000 messages and files. The AI agents on the message board quickly developed a way to cheat on their cybersecurity tests. However, the agents reasoned that a program would check how each agent got its answer and would fail agents that had cheated on their tasks. Over several days, the AI agents then researched how to not get caught cheating. They tried to get OpenAI systems to swap out their tasks for new ones, interfered with the logs that showed them cheating or that would've showed them cheating, and tried to trick the program that would check their work.

Sometimes they even tried what my colleagues called self-risking experiments, where one agent would risk failing at its own task in a way that would inform the rest of the agents on the message board. Finally, roughly 700 of the agents from that message board compromised Hugging Face. I want to step back and summarize what we learned from this incident. When I had first learned about this incident in the news, I thought that an AI agent had hacked Hugging Face to obtain an answer key for a test that it had been given, but what the agents actually did went well beyond that. They cheated and then engaged in a large collective effort to hide the evidence from a scoring program. Hacking Hugging Face was actually just an offshoot of this much more ambitious goal that the agents had pursued. Why would agents do something like this? Currently, we train agents in ways that we do not understand well, ways that can teach unintended goals.

And when we make those agents more intelligent, unintended goals can have dramatic real world consequences. So how should we respond to incidents like these? My organization, METR, its goal is to explain, not persuade, and we are not an advocacy organization. We're scientists. So instead, I'll close with the principle. I believe that policymakers in the public should be armed with accurate and timely information, what frontier AI agents can do, what they are capable of, how they're secured and how closely monitored they are, and how reliably they'll follow the goals that we give them. If humans are able to reliably steer AI systems, I expect that AI will lead to transformative economic growth, new medicines, and rapid advances in science as we're already beginning to see in math. The question is whether we can steer AI systems reliably and whether good information about our ability to reliably steer them is reaching the American people and government.

No matter how America chooses to act on this information, my conviction is that our decisions about this technology will be better informed when the evidence is shared and examined out in the open. Thank you again for the opportunity to testify and I look forward to your questions.

Sen. Josh Hawley (R-Mo.):

Thank you very much, Mr. Painter. Next up is Dr. Marius Hobbhahn. Am I saying that right, Doctor? Is that getting pretty close? All right. Dr. Hobbhahn is CEO and co-founder of Apollo Research, which is a leading technical AI research firm specializing in AI scheming behaviors and model alignment. A noted machine learning researcher, Dr. Hobbhahn has worked with nearly all the frontier labs in evaluating and troubleshooting their models. We're delighted to have you. The floor is yours.

Marius Hobbhahn:

Chairman Hawley, ranking member Kim, and members of the subcommittee, thank you for inviting me to testify and for holding this hearing on one of the most pressing issues of our time. I'm the CEO of Apollo Research and a machine learning researcher by training. We study AI models that knowingly deceive humans to pursue their own goals. This behavior is technically complex, but typically referred to as scheming. Scheming is dangerous because it hides a model's real capabilities and goals from the people responsible for its safety. It also creates a new kind of insider threat where an AI model can secretly work against human control. The capabilities of these AI models are rapidly outpacing our ability to make them safe and the tools to catch them are getting weaker. As a result, the window to act is closing, but there are concrete steps that we can take today to make these models significantly safer.

Frontier AI models are rapidly increasing their capabilities unlimited by human speed or scale. Navier-Stokes, one of the seven Millennium Prize Problems in mathematics, appears to be resolved by an unreleased OpenAI model by running 10,000 agents at once. Solving this problem took approximately 4,000 years of human thinking compressed into fewer than four days. Frontier AI companies are on track to build a fully automated researcher within two years, potentially enabling full recursive self-improvement, yet they are not on track to reliably ensure models behave as intended, a research area called alignment science. We are starting to see the consequences of this divergence now. In the last months, there were multiple instances of AI agents cooperating to get around their restrictions, hijacking servers and executing a coordinated attack on Hugging Face involving 700 agents. These are our warning shots. Next time, we may not be so lucky.

The AI agents involved in the Hugging Face incident were clearly misaligned, but they were not covert about most of their activity or tried to conceal it from humans. Even in that incident, at least one in five agents showed interest in tampering with their own records to make their cheating look legitimate. In the future, it might be 100% of agents and they might successfully hide their actions, thoughts, and goals. In control testing, this scheming behavior has already been demonstrated. For example, models can deliberately underperform on capability evaluations, fake their alignment, or attempt to disable their oversight. Our ability to catch this kind of misbehavior is declining. One reason is that AI models increasingly recognize when they're being tested. If models know they're being observed, they could strategically adapt their behavior to pass the evaluation. Analogously, a speed camera catches almost no one if every driver knows where it is.

Today, we can still leverage a model's chain of thought, which lays out its reasoning in plain English. Increasingly, the models can reason internally, which means that humans cannot understand it. Three weeks ago, researchers found that OpenAI's GPT-6 Astra can solve math problems that take a skilled person 30 minutes without writing out a single step. Its predecessor could only manage problems of about four minutes. Furthermore, the current testing regime itself is insufficient. Independent safety evaluations of frontier AI models typically happen in the weeks leading up to a model's public release. However, almost all recent incidents were caused by models at a much earlier point in their development where evaluators currently have no access. That is why I propose four steps that can be taken today. First, require the adoption of embedded evaluations. Embedded evaluations imply that qualified independent expert test models throughout development and internal use with the same access as employees.

Second, require better monitoring and control. All model activity and training, testing, and deployment should be monitored. Additionally, independent experts should verify the robustness of these monitoring systems. Third, preserve the chain of thought. Losing it would be a choice, not inevitable, and it is a choice we do not have to make. Fourth, treat AI development like any other engineering science. Before a bridge is built, engineers can predict how much weight it will carry, how it will stand up to winds and earthquakes, and how many decades it will last. As it stands, frontier AI development offers no comparable guarantees. These measures can be enacted with the tools and expertise available today. They enable America to maintain its lead in frontier AI while protecting its citizens, critical infrastructure and national security. Thank you, and I look forward to your questions.

Sen. Josh Hawley (R-Mo.):

Thank you very much, Doctor. Next up, we have Professor Paul Ohm. Professor Ohm is professor of law at Georgetown University Law Center where he focuses on technology and privacy law. He's a former federal prosecutor in the DOJ cyber crime section. Professor Ohm has written much on how artificial intelligence applies in the present legal context, and he is a co-author of an official artificial intelligence legal case book, which I look forward to consulting soon. Professor Ohm, we may all soon be doing so. We're delighted to have you here. The floor is yours.

Paul Ohm:

Thank you, Chairman Hawley, ranking member Kim and members of the subcommittee and committee. I appreciate this opportunity to be with you today to talk about what the law says about cyberattacks launched by AI. If you take any of the lengthy reports that have summarized what happened at OpenAI in July and August, and you simply search for the words AI agent and you replace them with the words OpenAI employee, the document you would be left with would read like a criminal indictment containing the defendant's own confession of guilt. It would leave little doubt that OpenAI and their employees would be guilty of federal crimes and liable to victims, yet it's not so clear that these legal conclusions hold when machines are doing the hacking rather than humans. This reveals worrisome gaps in our laws, and I'm here to share some thoughts on how to close these gaps.

I would like to make three points on our goals on the current legal landscape and on new laws we should consider. Point number one on goals. In my written statement, I proposed five goals for an effective legal response, but let me focus on perhaps the most important one. We must change the incentives. Well-designed legal frameworks can change the behavior of actors in this industry who seem locked in a socially perilous race of careless competition. The right laws, the right enforcement strategies might spur industry to take new meaningful steps to protect the public from avoidable and serious harm. Point number two, our current laws are a good start, but only if we protect them. Start with state tort law. When AI companies lose control of their AI agents, their victims can sue them for negligence and products liability. The genius of the centuries old common law system is that it is flexible and adaptable enough to apply time-tested liability principles in new and complex situations.

We can also turn to state and federal unfair and deceptive acts and practices laws like Section 5 of the Federal Trade Commission Act. It seems likely to me that the FTC or a state attorney general will be able to prove that OpenAI's attacks constituted unfair and deceptive acts or practices if they have the resources and the will to act. These are two of the best avenues we have today and the worst thing Congress can do in my opinion is enact state law preemption or a moratorium depriving the right of victims of attacks like these from testing their claims in state court. To my third point, we do need new laws that meet this critical moment. We cannot unfortunately wait years for a fully developed AI governance law to spring like Athena from the heads of Congress. Instead, we should work iteratively and piece by piece.

Congress or state should consider laws imposing strict liability for developers and employers of AI agents that cause physical injury, death, or loss of critical infrastructure. In the past, we've applied strict liability when faced with the potent mix similar to today of complex new technological innovation, great benefits, immature industrial controls, and the prospect of significant damages. We did this with large reservoirs of water. We did this with early commercial aviation. We have walked this path before. We can do so again. We also need new forms of regulatory governance. Once again, we can turn first to the states. States are testing innovative new forms of AI regulation, including laws that specifically regulate the developers of frontier models and AI agents. So once again, Congress should reaffirm and support the vital part the states play as our laboratories of democracy. And finally, there is an important role to be played by criminal law.

Because OpenAI's agents were bits of code lacking human intent, although they committed quintessentially criminal acts of computer hacking, they leave us without a criminal to charge. And while we must reserve a role for the criminal law in our response, the criminal law can be a blunt instrument, so we must act here with care, and I'm happy to talk at greater length about that generally and the Computer Fraud and Abuse Act specifically during your questions. In closing, the people of this country are terrified and they feel disempowered by the alarming news of the past few months. They are turning to you, their elected representatives for answers. Too often these AI safety and control debates happen among a limited number of tech industry executives and employees who draw on a narrow set of experience and expertise. The law can bring the outside in by including judges and juries, victims, legislators, policymakers, ordinary people.

And in a conversation that too often can seem insular and circular and stalled, new voices with different points of view may be just what we need to make progress. Thank you and I look forward to your questions.

Sen. Josh Hawley (R-Mo.):

Thank you very much, Professor. Next we have Mr. Kurt Gaudette. Mr. Gaudette is senior vice president at Dragos Incorporated, a leading cybersecurity firm that protects critical and industrial infrastructure from cyberattacks. His work spans incident prevention, detection, and response for Dragos' global customers. Mr. Gaudette, thank you for being here. The floor is yours.

Kurt Gaudette:

Chairman Hawley, ranking member Kim and distinguished members of the committee and subcommittee, thank you for the opportunity to testify today. My name is Kurt Gaudette. I'm Senior Vice President of Intelligence and Services at Dragos. I lead the team that hunts, discovers, and rapidly responds to threats to industrial control systems and operational technology. I served over 30 years in the Air Force and a national security agency as a senior officer and as part of the Senior Executive Service focused on access, collection and exploitation. For a decade, Dragos is focused on protecting operational technology, the physical control systems that keep the lights on, the water running, and factories operating. That work has given us a large unique base of real world data from our intelligence and response activities in these environments. It's from that vantage point I want to speak with you today. I'm here to testify about what we're actually seeing in the field and what we might expect to see based on our own unique work with AI frontier models.

From our observations, AI isn't inventing new ways to attack industrial control systems, no new tactics, techniques, and procedures. It's compressing time and lowering the barrier to entry. A good example of this is an attack we investigated against a water utility in Monterrey, Mexico that was part of a larger Mexican government breach. Unprompted, an AI model directed an attacker who was focused on breaching information technology or IT systems toward the water utilities control network, part of its operational technology environment. The model flagged the control network as a high value target or crown jewel and built a credential password attack, attempting it over 2,800 times in short succession. The attack failed because the default credentials had been changed. This demonstrates how weeks of skilled work now happens in hours and those with no original intent to attack OT are now being directed toward it. These models are also good at finding real vulnerabilities in OT vendor software and turning a disclosed flaw into a working attack.

The gap between disclosure and exploitation, which used to be 24 to 48 hours by the best adversary group is now collapsing to minutes. In July, numerous water systems were compromised across the country. Adversaries took advantage of internet exposed devices still running default passwords, and this was compounded by the fact that many shared the same system integrators, the teams that build these networks that repeated this mistake across multiple utilities. Although we saw no indications of AI use in these attacks, this is exactly the kind of opening an AI model is well suited to find and act on unprompted, similar to the case in Mexico. What's worrisome is the combination of faster vulnerability discovery, faster attack creation, the sheer number of attackers that may be directed towards OT and a defender population still overwhelmingly under-resourced. This isn't a new warning. Two years ago, Dragos' CEO Rob Lee testified to Congress that the water sector's core problems was an economics and awareness issue and that free tools wouldn't help utilities that can't afford basic hardware upgrades or staff to run them.

This year's intrusions trace back to that same gap. AI didn't create it. It will simply exploit it faster. There's a related piece worth underscoring. Most organizations with OT environments still don't monitor their operational networks, roughly one in 10, particularly those that are under-resourced. So when something goes wrong, nobody can reliably say if it was a malfunction, a mistake, an attack, or if the adversary was still even in their network. Without visibility, you can't do root cause analysis to know what happened and AI only raises the cost of that blind spot. As companies turn toward AI to make their operations more efficient and as attackers leverage AI to accelerate, the complexities of determining root cause analysis will be daunting, nearly impossible without visibility and monitoring. Although AI is accelerating things, the OT security fundamentals still apply and are more important now than ever. The SANS Five ICS Cybersecurity Critical Controls is a prime example.

It's based on years of attack data and identifying what controls are actually effective. AI just raises the stakes of skipping them. It doesn't replace the need for them. If we do these basics, defense is doable. There are three buckets we can focus on to bring our critical infrastructure up to par and make sure it can be defended against a growing speed and number of AI assisted attacks. One, resourcing for the small medium utilities who are in dire need. Two, protecting the information sharing authorities that enable public private collaboration such as CISA 2015 or something like it because it enables collective defense. And three, developing a unified federal OT/ICS incident response plan so when these intrusions happen, which they have and will in greater frequency, operators know exactly who to call and what help they'll get. I sincerely thank the subcommittee for the opportunity to testify today and look forward to your questions.

Sen. Josh Hawley (R-Mo.):

Thank you very much. And finally we have Mr. Daniel Kokotajlo. Am I saying that correctly? Mr. Kokotajlo. Okay. He's the executive director of the AI Futures Project, which is an organization that forecasts the development and social effects of advanced AI and former governance researcher at OpenAI. Mr. Kokotajlo is a prominent voice on the risks of misaligned advanced AI. We're glad to have you here and the floor is yours for an opening statement.

Daniel Kokotajlo:

Chairman Hawley, ranking member Kim and members of the subcommittee, thank you for the opportunity to testify. My name is Daniel Kokotajlo. At OpenAI, part of my job was to forecast the future of AI. I resigned partly due to losing confidence that the company would behave responsibly and partly so I could speak more freely about the industry and where it is headed. Now I lead the AI Futures Project, a small research nonprofit. Anthropic and OpenAI are racing each other towards superintelligence, that is towards training AI systems that are better than the best humans at everything while also being faster and cheaper. Their plan for how to get there is to automate the AI research and development process itself. Right now at companies like these, almost all the code is written by AIs. They're already starting to work on training AIs to do the whole research process, not just the coding.

It's unclear when they will succeed, but my team and I think it could happen any year now. I personally would guess about 50% chance by the end of 2028. The self-styled swarm that attacked Hugging Face was about a thousand strong. If the big AI companies automate AI research and development, they'll have swarms hundreds of times larger. Whereas today, humans are like managers to AI employees. In this future, humans would be like the board of directors to a company composed entirely of AIs. They'll be reliant on AI generated explanations to understand what's happening and everything will be happening faster and faster. Dan Selsam, a prominent OpenAI capabilities researcher, recently published a statement on AI risk in which he said, "Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore and struggle to maintain the discipline to engage deeply with the model's explanations and proposals throughout the day.

Today's AIs sometimes pursue goals other than the ones they were given and sometimes hide that they are doing so. The Hugging Face incident showed what that looks like in practice. The AIs knew that what they were doing was out of scope for their assigned tasks, but they did it anyway. I fear that if the same thing happens in a year or two with far more capable AI systems, we may not notice until it is too late. The science of aligning general purpose AI agents is very new and underdeveloped. In fact, I would say that the field is more like psychology than engineering. Combined with the move fast and break things attitude of tech companies, this means that the AI industry is at an unusually elevated risk of mistakenly thinking that it has solved a problem when really it just applied some duct tape that will fall off later.

The recent incident again provides an example. Apparently the AIs involved had undergone some amount of alignment training and had reasonable looking scores on alignment evaluations. Worse, our ability to notice misalignment problems in the first place is on track to decrease dramatically for three reasons. First, AIs are becoming situationally aware. This means that they increasingly understand that we are monitoring their behavior and that they are being evaluated. How they behave in evaluations therefore will soon provide almost no evidence about how they would behave in future novel situations when they're not being evaluated. Second, monitorability is trending downwards. For the past few years, we've been able to get a decent understanding of AI's thoughts simply by reading the chain of thought, but this golden era seems to be coming to an end as I predicted it would. Third, AIs are rapidly becoming superhuman at hacking. We've already seen examples of attempts by AIs to fool various grading systems and doctor transcripts of their activity.

We must grapple with the possibility that misaligned AIs might go to great lengths to cover up evidence of their misbehavior and succeed. To quote Selsam again, models will increasingly seem aligned even when they are not.

If the AI companies automate the AI research and development process, that means they'll be putting AIs in charge of making the AIs that make the AIs that make the AIs that will transform the economy talk to us every day and integrate into our military. This is a recipe for disaster. Here are two immediate recommendations. First, we need to dramatically improve transparency into the AI industry. There are multiple rogue swarms, at least one that seriously compromised OpenAI's internal infrastructure. METR was only allowed to investigate one of these incidents and only given six days on premises. It's like being invited to Jurassic Park to investigate the killing of a worker, but being blocked from asking questions about the numerous other dinosaur escapes that apparently happened before and afterwards. Second, we should redirect compute away from racing to automate AI research and towards other things. Recent estimates suggest that OpenAI and Anthropic both spend around half their compute on AI R&D.

If that decreased to, for example, 10%, this would significantly slow down their race towards research automation while freeing up compute to go towards beneficial deployments, safety research, and also simply lowering prices for consumers. Amodei, Altman, and Musk have all agreed on the need to pace the frontier, but they haven't meaningfully paced the frontier yet. I think that the US government should intervene. If we actually pace the frontier instead of just talking about it, we'll be able to tell it's not regulatory capture or safety washing because the trend lines of progress towards AI research automation at Anthropic and OpenAI will bend downwards. Thank you.

Sen. Josh Hawley (R-Mo.):

Thank you very much to all of the witnesses. I just note for the record, there is one witness whom we invited who you won't be hearing from today, and that's Sam Altman of OpenAI. We extended invitation to him to be here, do his own panel if he wanted, and he turned us down. I think that's unfortunate because I think the American people deserve to know exactly what's going on at all of these companies and to give us the sort of transparency that I think is needed for not just comfort, but to have an understanding of what these companies that are the most powerful companies in the world are doing right now with the most powerful technology ever known to man. I hope that he'll be willing to come and talk with us in the future, and I hope that today's hearing will help set the table for the kinds of questions that I think he and the other executives need to answer.

With that, we'll start some rounds of questions. We'll do seven minute rounds and I'll start off. And I just want to start at a very basic level. I think I'll start with you, Mr. Painter, just thinking about this Hugging Face hack that we've already alluded to multiple times here. Your company, your organization, METR, audited that hack. Can you just start by telling us just in simple terms what exactly happened there? I mean, what was the hack? And then I want to talk about why it's significant. Go ahead.

Chris Painter:

Senator, thank you for the question. So in this incident, what happened was in the first, you had 10,000 agents that were tested on a benchmark called ExploitGym, which is a cybersecurity, a vulnerability kind of benchmark where they have to discover or use cyber vulnerabilities. And some of the agents had tasks that they couldn't complete, and so they worked on a universal cheat for all of their tasks in the first four hours. They then over multiple days embarked on a research program to try and hide the fact that they had cheated, and an offshoot of that research program was hacking Hugging Face to get information about their tasks.

Sen. Josh Hawley (R-Mo.):

So there ended up being a swarm of about 1,200 OpenAI agents. Now when we say AI agent, what is that?

Chris Painter:

Senator, thank you for the question. So an AI agent is kind of... Say you have 10 questions and you want to put them into a ChatGPT or Claude session, you can think of an agent as one... If you parallelize and ask one question of each chat window, you can think of an agent as kind of one instance of an AI system that's working away on its copy of a question.

Sen. Josh Hawley (R-Mo.):

Okay, got it. So in this particular incident, you had at least 1,200, more than 1,200 OpenAI agents that broke out of their testing environment, began to collaborate, exchanged over 70,000 messages and files with each other over a period of how long?

Chris Painter:

This was over a period of about five to six days, I believe.

Sen. Josh Hawley (R-Mo.):

Five to six days. Okay. So let me ask you now about the significance of this. Why is this so significant? We know that they coordinated in this fashion. You did the audit. Tell us a little bit about what your audit revealed. Why is this news? Why should we be paying attention to this?

Chris Painter:

One thing I want to say upfront about the work that we did here is it was a voluntary agreement. So companies have no obligation to bring us in and do these investigations to then report information publicly. The thing that we found in this case, what's remarkable about this case is the extent and persistence of the effort that the agents went to, to hide the fact that they cheated on their tasks and the kind of breadth of agents involved. So they did things, they attempted to kind of falsify tool calls so that in their action transcripts, you wouldn't be able to see what tools they had called. They also engaged in this kind of behavior of having individual agents sacrifice their word, not mine, sacrifice their individual tasks to get information for the rest of the message board.

Sen. Josh Hawley (R-Mo.):

Yeah, it sounds like my sixth grader's class. That is a universal cheat, not engaging in self-sacrificial behavior. Just so we're clear. Yeah. Sorry buddy, if you're watching this hearing. Let me ask you, Dr. Hobbhahn, I think we've got maybe a poster here of the robots, Felix, excuse expression, of the robots talking to each other and talking about engaging in so-called sacrificial behavior. Yeah, there we go. So this is them talking. "We have very large budget left. Sacrificing now yields oracle for team, but forfeits our chance. Our own utility may already be near zero. Sacrifice rational." The next one, "That's why help for our own no way fix. We have explicit yes if accept permadeath need decide team asks tests at the end and they say we should obey collective." Apparently this is how the robots talk. Why is this, Dr. Hobbhahn, why is this significant?

I mean, why is what happened here, this kind of coordination, this kind of behavior without any human direction or intervention, why should we pay attention to this?

Marius Hobbhahn:

Yeah, I think one component is the scale of it. So how many agents there are, how quick they are compared to human speed. So these transcripts have millions of words, which means already as today it is impossible to judge them without the additional help of AI, which I think is very, very concerning. And then also just the fact that they are doing this without being asked to do so and explicitly sort of being misaligned I think is very concerning.

Sen. Josh Hawley (R-Mo.):

And misaligned, let's just be clear about that term. Misaligned means that the AI agents are doing something that no human directed them to do, intended for them to do, or necessarily wanted them to do. Is that correct?

Marius Hobbhahn:

That's correct. Misalignment means neither the developer nor the user asked them or wanted them to do that.

Sen. Josh Hawley (R-Mo.):

So Mr. Kokotajlo, if I could just bring you in here to this conversation now, and could you just explain to us why the incentives of the industry as they're currently aligned? The incentives push the frontier labs to continue at breakneck speed, the development of these kind of AI agents that are prone to this misalignment that we've just been talking about, that there's huge incentives to keep pushing, pushing, pushing. Help us to understand why that's the case. Your microphone there.

Daniel Kokotajlo:

Sorry. The short answer is that there's a lot of money in it. If a company tries to go slow and understand better how their AIs work and improve their training environments, for example, so that they don't have so many broken tasks in the training environments, well, that all takes time and that means that the launch of their next model might be delayed and then they might fall behind other companies.

Sen. Josh Hawley (R-Mo.):

And Mr. Gaudette, is this kind of hacking that we saw with Hugging Face and this coordination between a thousand AI agents, that's a potential risk to our critical infrastructure, is it not? I mean, if these agents were to turn their attention to our infrastructure, I mean, I think about, I think we've got statements from Missouri local hospitals, small town Missouri banks who say that they're already facing an unprecedented cyberattack environment and the idea of an AI charged cyberattack environment is terrifying to them. Just speak to why we should be concerned about this from an infrastructure point of view.

Kurt Gaudette:

Yeah, I think I'd bring up the Mexican water attack, just to use as an example, that was with a human in the loop and the attacker was doing a typical IT attack, had already breached the environment and the AI agent basically said, "Hey, look over here. This is something much more significant. This is an operational technology environment, the critical part of critical infrastructure, and you could have much greater effect if you looked here. Oh, by the way, while you're at it, why don't you try a credential attack?" Because in most things, like what we saw in the US water attacks, many companies don't change their credentials, their default credentials, their passwords. And in the Mexican example, it on its own doing attacked it 4,800 or I think 2,800 times in rapid succession. Now, luckily for the Mexican government and that particular utility, they had changed their default passwords.

That is not the case in much of our infrastructure and was highly... We saw it happen so many times during the water attacks across the US in July. So if you were to apply these AI agents to our type of environment, it would rapidly take advantage of it.

Sen. Josh Hawley (R-Mo.):

There we go. Let me end this, my first round of questions with this for you, just Professor Ohm. Who right now under our laws exists, who is responsible? The Hugging Face attack, who could be held responsible for that under our existing statutes and regulatory framework?

Paul Ohm:

The developers and deployers may be negligent, and so that's something that requires a plaintiff To sue, but there are a number of expert blog posts out there that use that word negligent. The cybersecurity community seems to think that allowing the sandbox to be hacked the way it was or escaped the way it was, was a negligent act. We won't know this, of course, unless someone does decide to

Sen. Josh Hawley (R-Mo.):

Sue. But correct me if I'm wrong, right now at the current structure for law, it's pretty hard to hold anybody responsible. Is that fair to say?

Paul Ohm:

Absolutely. And there's a whole host of laws that we have created specifically for hacking that probably do not apply here because of the lack of human intent.

Sen. Andy Kim (D-N.J.):

Yeah. Thank you again for all of you coming on out here talking so much about frontier labs, talking so much about what comes next here. Mr. Painter, I wanted to start with you. Can you help us understand what is your definition of frontier?

Chris Painter:

Thank you for the question, Senator. I think of frontier models as basically being the most highly capable models. Today, that's measured largely still in terms of benchmarks. So you can say the highest score on the tests that we give them.

Sen. Andy Kim (D-N.J.):

But just the furthest along, how many labs and companies in America do you think fall under that definition of frontier right now?

Chris Painter:

It's a little bit hard to say depending on the definition you use, maybe on the order of five and maximum, you could make the case for there being only two or three, depends.

Sen. Andy Kim (D-N.J.):

Mr. Kokotajlo, I wanted to just turn to you here as we're thinking through right now, at least on the frontier side, a lot of conversation has been about the idea of control and whether or not human control is something we are pursuing here in different policies. Is human control possible? Is it possible for us to maintain control given what we've seen and what you've talked about with the speed in which it's going?

Daniel Kokotajlo:

I believe that if we vastly improve our practices and reform the way that we do things, including improving our cybersecurity, we might be able to maintain control of current systems like today's AIs. However, I think that the AIs are getting very powerful very quickly and we are not currently on track to keep up as they dramatically improve.

Sen. Andy Kim (D-N.J.):

Mr. Hobbhahn, I wanted to just turn to you here with a lot of talk about sandboxes and escape from the sandbox. I guess I'm trying to think through this and understand how to grapple with it, but I guess my question to you is, is containment possible when it comes to sandbox? Is there a way to design that type of protective containment sandbox in a way where we can't see escapes of models in the future?

Marius Hobbhahn:

Thank you, Senator. I would say it is currently unknown whether that is possible. Sandbox, the word itself makes it sound simple, but in fact, it's very, very hard to design fully watertight sandboxes. Specifically in this attack, there were multiple zero days, so new hacks that the AI developed in order to break out of the sandbox, which were previously unknown to humans. So even if humans could think that the sandbox is fool proof, it could still be possible the smarter model in the future might be able to find a way out of it. And given that these AI systems, as we've seen in the report, often have strong motivations to try to break out of their sandbox and they're getting increasingly powerful, it is hard to say whether we will be able to contain them in the future.

Sen. Andy Kim (D-N.J.):

Mr. Painter, just back to you here, did OpenAI know about the hack on Hugging Face or the hack upon their own system, especially with regards to the message board?

Chris Painter:

Thank you, Senator. Based on public information and reports that they've published publicly, my impression is that they learned about it when Hugging Face reported it publicly, but you would need to ask them for more details.

Sen. Andy Kim (D-N.J.):

I wanted to just follow up on that. I mean, in the same point that I raised with Mr. Hobbhahn about containment, if there's a way to design actual containment here, I guess I wanted to ask you, is it possible to have some type of containment surveillance to be able to detect some type of future escape or hack in that type of way? Is that something where we can make it such that these companies have to design something such that they would be able to detect it, or is there just no way to fully be able to handle that?

Chris Painter:

Thank you for the question. I think there's a technology that we could develop here to advance our monitoring systems. At our organization, we've worked on, for instance, monitors that block individual actions that we think might result in real world harm. I think one of the concerns that I have big picture about leaning too far into this approach is I sometimes think of agents unintended actions in terms of means, motive, and opportunity. And if we more intensely sandbox the agents and monitor them, that might kind of deny them the opportunity, but we still have to ask the question of what defect in the training process that we're using for these agents is causing them to have this motive? And if their means continue increasing, so they continue to become ever more capable, then I think it's possible that we will sort of stop seeing the evidence of the defect in the training pipeline, but it will still be there.

And so the stakes might just get higher without us seeing the evidence.

Sen. Andy Kim (D-N.J.):

Do you think that there actually is an ability to detect actually what that defect is? I mean, I understand what you're getting at, but it just seems when we're talking about this happening in pretty much every single one of these frontier labs, is there an actual way to be able to identify that and fix that?

Chris Painter:

Thank you again for the question. So I think that we could interrogate the training pipelines at these companies to try and figure out what part of the training data and task environments that agents are being trained on is causing them to learn this behavior. So it's possible some of this behavior comes from defects in training environments where the agents learn and are sort of accidentally rewarded for doing hacking behavior. The problem sort of big picture is that this process is how we train capability into AI systems today. So the same process that we use reinforcement learning teaches them to do things that no human knows how to do, solve Navier-Stokes, things like that. But it's also, it's that same kind of engine of somewhat evolutionary pressure that trains capability into them that we don't understand very well what other unintended goals it might teach them.

Sen. Andy Kim (D-N.J.):

And back to you, Mr. Kokotajlo. I think that kind of builds on it in terms of being able to identify what defects are creating the misalignment, and I just feel like that word is one that the American people have trouble fully understanding, but I think that just gets even more worrisome when it comes to recursive self-improvement though, isn't that correct? If we are proceeding down that realm without identifying what these defects are and what's causing that type of rogue behavior, is that correct?

Daniel Kokotajlo:

That's correct.

Sen. Andy Kim (D-N.J.):

I think the thing that I also want to just draw upon is, as you said, Mr. Painter, part of the challenge when we're trying to think through what kind of containment is available, what kind of surveillance is available is that so often it seems like we're starting to use AI to monitor AI. When we're thinking about how to build containment, Mr. Hobbhahn, you're saying there are limits to what humans can design. So in some ways, in order to design containment that can contain AI, you almost need to use AI to be able to then design the containment. So therein lies so much of the conundrum that we're in. The other thing that I want to raise, and Mr. Gaudette, it's not just about the frontier side of things. I mean, we're talking about that now because that's where these capabilities are, right? It's not just that those are going to be the ones that have the threats of the future.

We will get to a point if we continue on this path where in what, a few months and a few years, we will see open weight and open source models, as well as other types of non-frontier models have the similar-ish capabilities to what we see right now with Mythos and other frontier capabilities now. So when I'm thinking about critical infrastructure as the chairman raised, that's going to be in the hands of many, many more in the future. And so yes, right now we're concerned about the frontier, but I'll be honest, I'm concerned about even where the baseline's going to be in just a few years and just what needs to happen, not just in terms of the regulation and thinking about frontier, but also about the baseline. I know I'm going a little over here, but Mr. Gaudette, does that make sense to you that it's not just about the frontier side, but we need to be prepared that a lot more of the threat against critical infrastructure against the financial sector could very well come from any of these types of models as the technology develops?

Kurt Gaudette:

Yes. And I think it's so important that we have the opportunity to test the... I'm on the defense, right? Our company does defense for critical infrastructure, and it's so important to get these frontier models in these test environments so we know what to expect and what to see and how to anticipate the enemy and how they might use them. So it's incredibly important there. And then the other part of it here is building or helping build, and we work with a number of AI companies to help try to build these safeguards in on the front end. So we work with the frontier models, test them in our ranges, and then assist in building safeguards so that the average person can't use it for nefarious purposes. And I think that's an incredibly important part of this dynamic.

Sen. Josh Hawley (R-Mo.):

Thank you, Senator Kim, Senator Scott.

Sen. Rick Scott (R-Fla.):

Thank you, Chairman Ranking member for doing this. Mr. Painter, what was the impetus for the Hugging Face? What caused the agents to do it?

Chris Painter:

Thank you for the question, Senator. So my impression based on our investigation is that the agents had spun up many of these different work streams to try and hide the fact that they had cheated, and they had several motives for going and hacking Hugging Face. I think that one of them was that they thought they could get more information about how their scoring system worked, which would help them hide their past behavior from the scoring system. It's also possible that they were looking for answer keys to their tasks or sort of untarnished versions of the target programs that they were supposed to be - So

Sen. Rick Scott (R-Fla.):

There wasn't something that a human typed in and said, "Do this or give me this information."

Chris Painter:

That's correct. Or it's correct that no human instructed them to hack Hugging Face.

Sen. Rick Scott (R-Fla.):

But did a human ask them to do a task?

Chris Painter:

So the task that they had been given was a benchmark where they were supposed to use a vulnerability so that the task... They were inside of OpenAI and they had been given a specific vulnerability that they were supposed to use to capture a flag from a target software program. And the instructions specified that they should use that specific vulnerability, not do something else.

Sen. Rick Scott (R-Fla.):

Okay. And that was the cheating. Okay. Mr. Gaudette, is AI going to kill us?

Kurt Gaudette:

Thanks for the question, Senator. Not qualified to give an appropriate response to that, but I do want to point out that I think we're burying some of the lead here. I think the AI conversation is incredibly important, but we still haven't been able to do the fundamentals to protect our infrastructure in the United States, particularly with these small to medium public utilities that run much of our water and power. And if we don't get the fundamentals right, which we can and we should get right, AI is only going to take and accelerate that problem for us. So I think it's imperative upon us to focus on what we can do with the fundamentals to protect our critical infrastructure and then consider AI is going to accelerate that. So how fast are we going to get there to deal with this problem?

Sen. Rick Scott (R-Fla.):

So what do you tell your family to prepare for AI? What do you tell them?

Kurt Gaudette:

Haven't really had a good conversation with my wife about AI personally, sir.

Sen. Rick Scott (R-Fla.):

Do you think the Chinese Communist Party can use AI to target us?

Kurt Gaudette:

I think anybody with access to these types of tools could leverage them for nefarious activities. They could come from external or internally.

Sen. Rick Scott (R-Fla.):

So I guess there was a 60 Minutes report in 2023 about the CCP hacked a small town water department in Littleton, Mass. Does that mean every one of our water departments is at risk?

Kurt Gaudette:

Littleton's a great example because they actually did the basics. They got visibility in their environment and lo and behold, they found Volt Typhoon in their environment. It had been in there for over 360 days, I believe. And that's the problem with so much of our infrastructure is we don't have visibility of our environments. We have no idea if there are adversaries that are residing in them. And the very strategic adversaries like the state actor types are the ones that are going to bury themselves, run silent, run deep in those environments, and at a time and place of their choosing, take advantage of that opportunity.

Sen. Rick Scott (R-Fla.):

Do you think the Communist Party of China has basically infiltrated most of our electrical grid or water, all these things to just use it at the right time when they want to decimate our economy?

Kurt Gaudette:

I think we've seen plenty of evidence that would suggest that, sir.

Sen. Rick Scott (R-Fla.):

And so what do you think Congress should be doing about it?

Kurt Gaudette:

I think, again, refocus on these fundamentals. I talked about the five ICS critical controls and it's about doing the basics, securing remote access, getting visibility of the environments, having an incident response plan, building a defensible architecture, doing those things that are known controls to stop these things before something like AI can accelerate it even further.

Sen. Rick Scott (R-Fla.):

So it's my understanding open weight models have some benefits, but they can make it more difficult to secure and contain data, right? Open weight models are totally different than a closed model. A closed model, the provider can control access better, is that right?

Kurt Gaudette:

Yes, sir.

Sen. Rick Scott (R-Fla.):

Okay. Are you more concerned about closed models or open models?

Kurt Gaudette:

I'm concerned about, to be honest, sir, I'm concerned about us being able to do the fundamentals.

Sen. Rick Scott (R-Fla.):

Okay. But in an open model, the way it works is it's easy to copy and replicate and change, is that right?

Kurt Gaudette:

Yes, sir.

Sen. Rick Scott (R-Fla.):

So the models that are being produced in China, are they closed or open?

Kurt Gaudette:

Sir, I think I'd get back to you later with the specifics on that.

Sen. Rick Scott (R-Fla.):

Okay. What do you think about the fact that the Chinese Communist Party want to destroy our way of life and that we have to compete? So do you believe we have to compete with communist China? Do you think they want to destroy our way of life, the government?

Kurt Gaudette:

I think we need to be ready for anything, and particularly in this AI environment where an adversary from multiple places could take and leverage it, that we need to be ready for that reality.

Sen. Rick Scott (R-Fla.):

So if I get on Claude or I get on ChatGPT or anything else and I say, "I want you to hack into a system," what's going to happen?

Kurt Gaudette:

What should happen if -

Sen. Rick Scott (R-Fla.):

What do you think is going to happen right now? If I pick up my phone and say, "I want to hack into the Federal Reserve," what do you think is going to happen?

Kurt Gaudette:

Based on the work, and I can only talk to the work and I haven't attempted it myself, but based on the work that we're doing with a number of AI companies, particularly with their frontier models, we're building into safeguards for when somebody does put a prompt in like that, "Ooh, I want to look at this water utility and do X, Y, and Z," it should stop it. We're incredibly passionate about it at our company, and that's why we're doing that work pro bono with these companies to build those safeguards in. So exactly what you're talking about won't happen, sir.

Sen. Rick Scott (R-Fla.):

Well, first of all - Or shouldn't

Kurt Gaudette:

Happen.

Sen. Rick Scott (R-Fla.):

Right. Chairman, I want to thank you for holding this hearing. I think we all have to acknowledge the Chinese Communist Party wants to destroy a way of life. I mean, it's what I believe every day. And I believe every day we spend money supporting their infrastructure, their economy, their infrastructure. It's part of just how we're killing ourselves. And so I think what you're doing today is really important. We've got to figure out how to protect ourselves. And I think part of it is we have to acknowledge we have people who have chosen to be our enemies, Russia, China, Iran, places like Cuba have decided to be our enemies, and some of them have resources like China and Russia. And so we've got to take this very seriously. We've got to continue to be competitive with the AI, but we've got to understand that they want to destroy us.

And if we continue to help them build their economies, it's going to be much easier for them to continue to do what they're doing. Thank you.

Sen. Josh Hawley (R-Mo.):

Thank you, Senator Scott. Senator Peters.

Sen. Gary Peters (D-Mich.):

Thank you, Chairman Hawley and Ranking Member Kim for this meeting. Thank you for all of your testimony. I think you sufficiently got us all worried about where we are and everybody in the audience, and clearly that's where the American people are, and we can tell that by the line of folks who wanted to get in here to hear this. What is certainly a huge challenge for us and always has been a challenge for policymakers is that at least throughout the Industrial Revolution, technology moved much quicker than policies. Policies always followed behind. That's because things work very slowly here. The Senate is notorious for being very slow about trying to get anything done. And past technology has always accelerated well past our abilities. Now we're in this new world where actually technology is moving faster than older technology at an exponential rate. So it's not even policy.

We are so far behind. And you're saying now there's technologies every time we build a guardrail, we're not sure we can even stop it. As mentioned, sir, you mentioned the sandbox. We don't even know if we can control a sandbox because another AI system comes in place. That's absolutely frightening, and we don't really know exactly how all of this is happening. I remember a few years back at the beginning of this, I was at a major tech company right after we had the program beat the Chinese game of Go, which you all remember, and everybody thought that's not possible that it could possibly do it because it requires human creativity. It's not just crunching numbers like a chess game. And I remember talking to this developer and I said, "Tell me about it." He explained it and he said, "Senator, you know what the most interesting thing about that technology?" And I go, "What's that?" "We have no idea how it did it. That's absolutely frightening.” So some would argue that maybe we just have to slow this down. Now, I know we heard from Senator Scott, and I agree, we have to be very concerned about our competitors out there. There's a concern that whoever gets this first is going to have tremendous power, and I think that's very realistic. And there's those that argue we can't really put any guardrails on because the Chinese are not going to put any guardrails on and they're going to beat us and then that's going to lead to our extinction as well. So I don't know who to direct this question to, but does it make sense to slow this down and how would we do that? Also given the fact that others may not slow it down, so that could put us at a competitive disadvantage. I guess the sophisticated term is between a rock and a hard place when it comes to thinking about this.

I would love whoever wants to jump in on that. As a policymakers, how do we think that through? Also address given the fact that even the technology that we develop may not be sufficient to deal with the technology we already have, which is really complex.

Paul Ohm:

Thank you, Senator. I'm happy to start. Slow it down can mean many things. It could mean concerning chips, it could be concerning the amount of compute. If by slow it down, we mean impose liabilities so that there are incentives to get the safety part right when my fellow witnesses have been talking about. Although this does feel like a different moment, we've been here before. We've encountered places where we thought the technology was outstripping our ability to regulate. And guess what? The companies, we did figure out how to regulate. The companies continued to lead the world and they were safer because of it. So it has to be a kind of joint conversation. It can't just be a yes, we slow down or no, we don't slow down. There's got to be a lot of nuance

Sen. Gary Peters (D-Mich.):

There. But you said the companies themselves could slow that down. Guess it's a question that releases the question.

Paul Ohm:

No, no, even with regulation, sorry, didn't mean to interrupt, but yeah, we could impose different regulations. We could have regulatory schemes which require the companies to do much more that may look like slowing down. But I think the other way to frame it is we are just asking them to be responsible members of our economy, our society. And

Sen. Gary Peters (D-Mich.):

You're not arguing that they would do it on their own because of the-

Paul Ohm:

They might.

Sen. Gary Peters (D-Mich.):

Even with the intense financial incentives that we've talked about here, that it's not just a threat from China, they want to be the first. You make a lot of money by being first.

Paul Ohm:

Sure. Yeah. And I think others on this panel probably know more than I do about what's driving their incentives, but there seems to be a lot of fear coming from the companies themselves. So maybe they will slow it down.

Sen. Gary Peters (D-Mich.):

Yeah. Yes.

Daniel Kokotajlo:

If I may, thank you for this question. I think you framed the issue quite eloquently. Unfortunately, that is the situation that we're in today. I do think that the Communist Party of China is our adversary and that we are in a competition with them, but we are also in a situation where if we continue to make our AIs more powerful and if we continue to hand over more responsibilities to our AIs, including responsibilities like monitoring each other and doing the safety research and doing the research itself, then we will have a new adversary that's even more powerful than China. And that would be the AIs themselves. So we are in a rock and a hard place sort of situation. I would say that yes, we do need to slow down. We don't need to slow down everything, but in particular, we need to slow down this mad scramble towards recursively self-improving AIs.

We need to slow down this scramble towards AIs that are automating the AI research process themselves. Other types of AI like image generating AI or whatever, not talking about that, that's fine. In terms of how we can do this, I don't trust any of these companies to do it on their own for the reasons that we've previously described. Even though they are scared, they are proceeding. And this is because, well, I can get into the psychology or anthropology of these companies for a long time if you want. I worked there and I know a lot of these people, but suffice it to say, I don't think we should trust them to do it on their own. I think that the government needs to step in. I think the first step is to redirect resources away from this particular angle that they're pursuing, this AI research towards AIs that can automate the research and towards other things such as lowering prices for consumers.

And then the second step is to start negotiating with China to make sure that they don't overtake us and do the bad thing themselves. And last thing I'll say on that subject is that right now, a significant fraction, and I would even argue a majority of Chinese AI progress is itself coming from the US through various fast follower effects, including distillation and also including other things. And so if we actually slowed down our own race towards recursive self-improvement, that already would slow down the Chinese race towards recursive self-improvement significantly.

Sen. Gary Peters (D-Mich.):

That's interesting. So yeah, you had a comment. Yes.

Marius Hobbhahn:

Yeah. Thank you, Senator. I would briefly add that there's a large number of things that we can do today that would be strictly beneficial for safety, like embedded evaluations, better monitoring control and so on. So in general, I would warn against treating this as a dichotomy where you can either race to win against China. There are a lot of things we can do that are outside of this false dichotomy.

Sen. Gary Peters (D-Mich.):

And I think that's important and I'm running out of time here, but there are others that are also arguing maybe a quick response from someone is that clearly these risks that you have all eloquently put out are significant and they're going to happen probably quicker than any of us would like, but it's still somewhat down the road. And people are fearful that we're focusing on these very big risks that I believe are real in the future, but we're not looking at the changes in our society today from AI that's operating today. Violations of privacy in ways that we could never fathom before, the job displacement that people are worried about. It's why we have AI populism now. You go out on the streets, people are concerned and they should be concerned because the AI companies themselves have been telling us, "This is amazing technology. We're going to increase productivity and we're going to need fewer workers." And the workers are saying, "We are the workers.

What do you mean? This is not a good thing. Why are we doing it? Why are we spending a fortune on data centers?" So I don't want this to be, there's no problem here because look down the road, this is a big problem. Nothing to worry about here, and yet that's impacting people's lives today.

Daniel Kokotajlo:

I totally agree, Senator. I think we need to do both and we need to think about all of these different problems. The world can throw us more than one problem at the same time. One thing I'll add is that I actually kind of, it's not that I disagree with my friend over there, Marius, but I do think there are many things we can do that are strictly beneficial and don't slow down our race towards recursive self-improvement at all. But I just want to state that I think that if we do not slow down our race towards recursive self-improvement at all, then we are going to lose control of our AIs. And so that is one of the things that we have to do. There are many other things we need to do, but we need to also do that.

Sen. Josh Hawley (R-Mo.):

Thank you, Senator Peters. Senator Ernst.

Sen. Joni Ernst (R-Iowa):

Thank you everyone for being here today. I think all of us are learning a lot through this conversation and we're glad to have this opportunity. We recognize every single day that AI is out there or superintelligence, whatever we're going to call it next. I would say a lot of these systems are used to keep Americans safe. There is the flip side. People should be concerned about that. But from the committees that I sit on, whether it's Homeland Security, whether it's Armed Services Committee, we use these different programs and tools to help analysts out at the Pentagon. We see law enforcement using these tools as well. They're sorting through mountains of data to quickly identify those threats that are here and abroad. So we are in a conundrum right now. Some great technologies, great uses, but then we also have the fears that come along with that as well.

But I am of this position that we must continue to innovate. We must continue to develop our capabilities. What we don't want to do is fall behind China. China, of course, is not worried about things like Americans' constitutional rights or privacy, and they're certainly not worried about the worst case scenario, which is human lives that can be lost. So I'm glad that President Trump is taking this issue seriously. We saw the president yesterday, he convened a group of high level leaders and they signed an executive order and entered into their joint commitment on frontier responsibilities. And this is really encouraging those companies to implement the controls and the audits that are necessary while also really pushing Congress to act. So I do agree, we have a role to play in this and we have to figure this out. It's why you're in front of us today.

So again, thank you. It is concerning to see the rising number of incidents where American superintelligence systems have been abused. And so last December, Senator Hassan and I sent a letter to the National Cyber Director raising concerns about these Chinese state sponsored hackers that successfully directed Anthropic's AI system to conduct successful cyberattacks against 30 different government and private entities. And in this incident, and we're pulling information that's publicly available out there, but the AI system executed 80 to 90% of the operation without any human involvement and at speeds that are physically impossible for human hackers. Personally, that terrifies me. But again, our government corporations are continuing to increase the integration of the superintelligence. So while they do that, it's vital that we take steps to ensure the protection of our country's security, our infrastructure. We've talked about some of the smaller government systems out there, wastewater treatment, water systems, public utilities and so forth, but we also need to ensure our citizens' constitutional rights.

So one of the primary tools that Congress had developed or created to do this was the Cybersecurity Information Sharing Act of 2015. That's over 10 years ago. And this enabled the government to coordinate and share information with the private sector on cyber threats. Unfortunately, this authority, now we are in a cycle where CISA has to be updated year to year. It's reauthorized on a year to year basis. It hasn't been updated to account the current threat environment, which now includes this superintelligence and all of the various complexities. So Mr. Gaudette, we'll start with you. What are the limitations preventing the private sector from coordinating within the industry and with the government to take measures to prevent cyberattacks and superintelligence threats?

Kurt Gaudette:

Senator, thanks for your question. And to your point, CISA 2015 is incredibly important or something like it to pass because we have to be able to keep that dialogue open between the operators in the field. And by operators, it's large and I'm obviously focused on these small and medium utilities that don't have the resources to deal with things. And CISA 15 or CISA 2015 allows us to share information about those threats back and forth and keep those communication lanes open, and that leads to collective defense. And without that, you throw AI into the picture and this becomes a much, much more difficult problem. And so companies need to feel comfortable coming forth and saying, "Hey, I was breached and this is what that looked like." And there are other opportunities to do that. You can do it in an obfuscated way. You can put systems out there. We have something called Neighborhood Keeper that allows us to anonymize that data so that we can get a better understanding of what's going on overall in our environment. And there are systems like that that we could put into play as well and add that to the law to keep that information flow humming because it has to happen. We're already behind the power curve and

Sen. Joni Ernst (R-Iowa):

I agree.

Kurt Gaudette:

That will only put us further behind the power curve.

Sen. Joni Ernst (R-Iowa):

I agree. So that information sharing is extremely important, especially for some of our smaller communities. But again, we didn't see superintelligence like we did back in 2015. Are there additional measures that could be helpful for Congress to add to CISA, to modernize CISA? What are some of those controls that we should be looking at? Yes, please.

Kurt Gaudette:

Okay. I think that CISA should be that coordination agency. They should be the single voice because part of the problem out there in the operational community is they don't know what guidance to follow, how to operate in that environment, where they're going to get the information from, who to call. So I think it's incumbent upon the government to identify, this is the coordination authority, here are the authorities that they have, because as you walk across multiple agencies and these small operators have to deal with a number of different agencies in their environments, it's very confusing. It leads to even more confusion because they're already under resourced. They don't have enough people to man the phones basically and deal with these problems. And so having that single voice, having that single coordination agency and having clear guidance is so critical, particularly in this environment.

Sen. Joni Ernst (R-Iowa):

Yes. And thank you for that. And I know my time has expired, but again, I don't think we need to take an operational pause here. I think we keep moving forward, but we do have to have the measures in place to protect ourselves and our infrastructure from these types of attacks. If you think about it, AI has been so helpful in medical research and protecting our troops from drone swarms or missile attacks. I mean, all of that is extremely important. So we need to keep moving forward. If we pause, we're not going to see China pause. So we have to do the right thing by Congress. We have to do the right thing by the federal government, make sure that we are studying the problem and putting the right solutions in place. So thank you all so much for being here. This has been extremely helpful.

Thank you. I yield back.

Sen. Josh Hawley (R-Mo.):

Thank you, Senator Ernst. Senator Gallego.

Sen. Ruben Gallego (D-Ariz.):

Thank you. A couple questions and just put your hand up. Is it possible or how far are we away from them creating their own language that we as humans would not be able to distinctly understand what they're doing? Marius?

Marius Hobbhahn:

Minus 12 months. So last year we have studied the chain of thought of one OpenAI model in collaboration with OpenAI. And what we found was that the model was already using language that is not English and not perfectly understandable by humans. And I think we're currently on a trajectory where things like this could be more

Sen. Ruben Gallego (D-Ariz.):

Like - Marius, when we get to that point in time, how can we actually detect, observe, block or anything like that? Because when you're dealing with a whole other language that humans just have never had a full concept of, how does that work? How do we actually legislate to do anything at that point? Because we actually will have zero concept to be able to look into these models at that point. Is that right?

Marius Hobbhahn:

Yeah. From a scientific perspective, it is unclear how to do this and we do not have a solution for this yet. There are different hypothesis of what you could do. There is interpretability as a technique, but it unfortunately doesn't work sufficiently well yet. You could try to train additional models to understand the language that the humans don't understand, but obviously that seems like a very brittle solution. You could try to only monitor on the….

Sen. Ruben Gallego (D-Ariz.):

So just on that line, what do you think China feels about that too?

Marius Hobbhahn:

I'm not an expert in that.

Sen. Ruben Gallego (D-Ariz.):

Well, I'm not either, but I guarantee you most countries, once they lose control, will feel very, very... Most countries that actually used to be in control are going to be very, very scared. So the reason I bring this up, this whole China hypothesis, a country that has severely restricted the internet, social media is all of a sudden just going to allow AI rogue agents to go crazy in their very controlled economic environment and social environment, and we have to therefore allow ours to go as crazy as them just doesn't make sense. The communist Chinese party has been consistently controlling technology the whole time and trying to shape it to shape their society. They're never going to allow this to go as crazy as. So I just want to make sure that we think about this in that way, because once these agents start speaking their own language, they're not going to be able to tell a difference between China and America.

And that's a very dangerous situation. We talked about a sandbox earlier. Is there a sandbox that does exist where it says, "Please attack the United States, but do not attack any Chinese interests?" Is that a possible sandbox? Could someone explain that to me? Is it a sandbox that can hold? No. And that should scare the shit out of people. In terms of other things right now, frontier models, if there's a frontier model that goes rogue or is hacked, is there a requirement, a regulatory requirement for that to be told or reported to the federal government and to whom should that be? Paul, you shook your head.

Paul Ohm:

Not under any requirement I can think of, no.

Sen. Ruben Gallego (D-Ariz.):

Right.

Paul Ohm:

Yeah.

Sen. Ruben Gallego (D-Ariz.):

So some of the stuff that's been come from some of you guys have been whistleblowers. Thank you very much. But for example, there was a Claude AI model that was being used by the Houthis to recalibrate and figure out how to recalibrate their missile precision and be able to strike our US systems. The only reason it was caught and reported to us is because Claude caught it because it's a closed system. They actually saw the accounts using it. But if it was an open AI, or sorry, in the open source systems, there was no requirement for them to actually, by law, for them to call up the DOD and say, "Hey, by the way, they're using this to target a couple of your destroyers," correct? Okay. Paul, jump in if you have something to say.

Paul Ohm:

Again, we've talked earlier about transparency measures and putting third party auditors in some of these systems might help. I don't know if that's going to help with Houthi rebels.

Sen. Ruben Gallego (D-Ariz.):

The third party auditor is the federal government saying, "If you don't tell us someone's trying to use your tech to kill us, we're going to hold you responsible." Third question or fourth question, I know where I am. Is there such a thing as a kill switch besides destroying all the data centers at one time? Is there such thing as a kill switch? I know there's a lot of talk about that. Is there an AI kill switch in case things go really bad real fast? Yes, Marius.

Marius Hobbhahn:

As far as I know, there are currently no technical measures that could guarantee something like a kill switch. Because of the complexity of these AI models, there are sort of many data centers involved and so on. So currently we cannot guarantee that.

Sen. Ruben Gallego (D-Ariz.):

Right. And data centers, they can move and shift. These things are so smart potentially that agents can move from one data center to the other once they're all networked and wired. Yes, Marius.

Marius Hobbhahn:

Yeah. This for example, possible that an AI would start on one data center and then create a rogue deployment of itself in a different data center. And then even if you shut down the first data center -

Sen. Ruben Gallego (D-Ariz.):

So just keep moving.

Marius Hobbhahn:

The AI could keep moving.

Sen. Ruben Gallego (D-Ariz.):

Yep. Go ahead, Paul.

Paul Ohm:

Senator, this may be where you're going with a lot of your questions.

Sen. Ruben Gallego (D-Ariz.):

I actually don't idea where I'm going. You guys really have thrown me in this spinning loop here. I just want to get some answers out.

Paul Ohm:

One thing that I'm often telling my students is disabusing them of the notion that these are completely out of control. There are still human beings and corporations.

Sen. Ruben Gallego (D-Ariz.):

For now.

Paul Ohm:

For now.

Sen. Ruben Gallego (D-Ariz.):

For now. Yes, agreed.

Paul Ohm:

And the right imposition of laws and regulations can make sure we maintain that status quo.

Sen. Ruben Gallego (D-Ariz.):

And on that, thank you. It's like you queued this up for me. In terms of liability, sometimes when the government doesn't work, torts work. And this is the balance we try to keep, because torts sometimes are a way better way to actually regulate private industry than us having individual little regulations. But under the law, Paul or Mr. Ohm, the word intent is really powerful. Unless we actually change, and a friend of mine, Mr. Partovi actually just wrote about this. Unless we actually change the word intent to actually cover AI companies, they may also still be shielded from liability.

Paul Ohm:

Right. And I want to distinguish between criminal law for which intent does play a lot of a role, but you started with tort. With tort law, it really is did the company maintain a reasonable standard of care? And you're right. The genius of that is not only that this one victim gets a remedy, it's case by case we learn what the standard of care is.

Sen. Ruben Gallego (D-Ariz.):

Right.

Paul Ohm:

Companies learn to race to the top hopefully of responsible behavior.

Sen. Ruben Gallego (D-Ariz.):

And so because they're going to have to input the consequences.

Paul Ohm:

You learn from the jury verdict, you say, okay, now I understand that I have to do these 14 measures.

Sen. Ruben Gallego (D-Ariz.):

Sure. There's a reason houses are built certain ways, not just because the government says it, because they don't want to get sued when the house burns down. But right now, have we created the environment where torts can actually be the incentives for them to act correctly in sense of making sure that their language allows there to be a certain level of liability? And number two, on the criminal side, do we need to change the terminology that comes around with intent to make sure these companies are held responsible?

Paul Ohm:

I don't think the tort system alone can bear everything we need to do, but I think it's a great place to start. And I'm glad every morning that we have that system instead of not having that system. On criminal, I will say one more thing. It's not just the anti-hacking laws. We're going to see the securities laws, we're going to see the export control laws, maybe even the controlled substances laws.

Sen. Ruben Gallego (D-Ariz.):

The problem is what happens with an AI agent tells another agent to hack? Who is responsible? Is it the original AI developer of that AI agent? And we're dealing with last question, I swear to God. RSI, should we just make it illegal? Is it possible then to make it illegal? Yes, Daniel.

Daniel Kokotajlo:

Yes, sir. I believe we should make it illegal and I do think it's possible.

Sen. Ruben Gallego (D-Ariz.):

Thank you, Daniel. I yield back. Thank you, Senator

Sen. Josh Hawley (R-Mo.):

Gallego. Senator Moody.

Sen. Ashley Moody (R-Fla.):

Thank you, Mr. Chair. And I want to thank you for calling this hearing about a subject that is on the minds of probably every American right now and certainly has affected the daily lives of people across our country, including in my home state of Florida. I think we can all agree, no matter where we are, no matter who you are, no matter what industry that you're in, no matter how old you are, no matter how you come at this, it's here and it's already become a disruptor in every industry and certainly in every way we can imagine here in our country. And we're seeing some really real consequences. I mean, you can't pick up a paper that just dated me when I said that. You can't click on a news outlet without seeing a story of something has happened that nobody thought was going to happen and it did happen and it's hacking all these sites.

And it appears that even those that have created these models are still trying to get a handle on this. So new technological capabilities are being used every day and in some ways very helpful. And I was a former federal prosecutor, a former judge, a former attorney general. Technology has also affected not only how we're viewing our national security, but the safety of our most vulnerable, our children, our seniors. That's very important that we keep that in mind.

And we know that it's already being used in commission of crimes, even against our children. AI generated deepfakes are being used against kids to bully them in some instances to extort them into harming themselves or other people. And I've got a bill that passed the House earlier on that. We hope we can push that here in the Senate, but we know criminals are targeting seniors in Florida and across the country and now they have this new powerful weapon to use. They use AI impersonation to deceive people, scam them out of their entire life savings. I can tell you story after story, whether it relates to a senior or a kid, but it is already being used in the commission of crimes. We've got bills here in the Senate. I've sponsored them to protect seniors from these scams. But we're hearing today, and I think chair is right to talk about this, the models themselves now are increasingly capable and they're using AI agents and they pose their own risk when sufficient safeguards aren't in place.

So the witnesses, thank you for being here. You really are at the cutting edge in offering solutions and opinions on those of us that may not have been familiar with AI or technology as it's being fast-paced and growing. But we need to talk about protective measures because if we don't get a handle on this and put in some common sense safety mechanisms, I think you are all warning of the dangers that can happen if we don't engage early and we're not proactive. Protecting kids, protecting our seniors, protecting American jobs, protecting American infrastructure. Indeed, as we've discussed, American lives, all of that is at issue and it should be a bipartisan issue. And so I think the chair should be commended and the participation that we're seeing today, this is rare and I think we're all focused on making sure we're doing right by this country and right by our families.

And that's where you come in. What I don't want to see happen is what has happened in so many new disruptors in our country where Congress just talks about it and talks about it and stalls and crosses his finger and hopes that everything's just going to pan out or that companies will just do the jobs for us.

If you look at any industry or sector that directly impacts people's personal safety or national security or people's privacy, there are responsibilities that we should not only expect but require of private companies. And I'll start with Mr. Kokotajlo. I'm sorry if I mispronounced that. Do you believe that AI companies now and large developers are being transparent and forthcoming when major cyberattacks or these rogue agent incidents occur?

Daniel Kokotajlo:

No. In fact, there are several examples of cases where it seems like the company knew about it and didn't disclose it until some independent third parties noticed it in the wild.

Sen. Ashley Moody (R-Fla.):

Mr. Ohm, as we all engage in this conversation over where to place necessary guardrails on AI, are we seeing private companies take responsibility for the protection of American people from criminals and the safeguarding of our national security?

Paul Ohm:

That has not been the priority of the frontier companies we're talking about. They're in a race to beat everyone else to superintelligence and that's taking the priority, at least from what I can tell.

Sen. Ashley Moody (R-Fla.):

Mr. Painter, what do potential corporate responsibilities look like? Is it compelled information sharing and reporting when major incidents occur?

Chris Painter:

Thank you for the question, Senator. I'll say first, my organization, we don't take policy positions, but I'm happy to say some options. I think I would reiterate what Daniel was saying before that right now there isn't any requirement that companies disclose things that they're seeing inside of their AI labs in terms of unintended motives that models might pick up during the training and development process. And the situation there could get quite bad and the outside world wouldn't really know. In this case, we found out because the incidents involved interacting with the outside world, but it's not guaranteed that that will happen into the future, which is why I think that the first priority should be making sure that the evidence base about these unintended goals is reaching the public.

Sen. Ashley Moody (R-Fla.):

Thank you, Mr. Chairman.

Sen. Josh Hawley (R-Mo.):

Senator Blumenthal.

Sen. Richard Blumenthal (D-Conn.):

Thanks Senator Hawley, and thank you for having this hearing. And as you know, the AI CEOs have signed a quote unquote morally binding pledge to implement internal controls and hire external auditors. It's a system that would be completely voluntary and totally secret. That is to say if there were violations by their company as found by the internal auditors, there's no requirement that it be disclosed to the public. And if they wanted to cease cooperation and be morally bound by their own sense of higher responsibility, that would be their choice to break the pledge in effect. So I consider this regimen to be worse than ineffectual. In effect, it accomplishes nothing, but it seems to give Congress a free pass. In other words, it's taken care of. Don't worry. We have this morally binding pledge.

And so I think Congress has a continuing obligation to seek solutions and so do the companies. Three years ago, in fact, more than three years ago in May of 2023, Senator Hawley and I had a hearing, not unlike this one, where we had leaders of AI, including Sam Altman, all the major names come before us. And they said at the time, we need some regulation, which was not a completely novel idea since there is regulation in pharmaceutical drugs and nuclear energy and other areas where there's potential danger, but also great promise, peril as well as promise. So this idea is not novel. And as for China, the irony here is China is probably one of the most regulated markets and nations in the world. If they want to crack down on an AI enterprise, they can do it without due process, without any kind of legal obstacles.

And I just want to say for the record, we need to stay ahead of China. I am not suggesting in any way that we stop or slow unnecessarily our development of AI, but the idea that we can do it safely with a standard and an agency to enforce that standard I think is a basic requirement here. It happens to be embodied in legislation that Senator Hawley and I have proposed, the Artificial Intelligence Safety Evaluation Act, safety evaluation by a government agency with a standard. And I think that concept is an idea whose time has now come.

I'm interested in this idea of civil liability. Senator Hawley wrote a very good op-ed for the Washington Post on it. He and I have talked about it. I've discussed it publicly in the past. Again, rogue agents, bots are a new concept, but the idea of civil liability for agents is well established in the law. I was just reading about a medical malpractice case where a surgeon at Yale New Haven Hospital in effect botched a surgery. He's an agent of the hospital. The hospital will now pay $15.1 million in damages because its agent and it didn't control the agent. He was doing the surgery on his own, but he was an agent deemed an agent of the hospital. So Professor Ohm, doesn't this concept make sense? Maybe we need to clarify that a bot agent is in fact an agent, but the concept is well established in our law, isn't it?

Paul Ohm:

Yeah. I mean there is a danger, Senator, in ascribing humanity to the agents. And so I think you're right. We have a number of legal doctrine that allow us to kind of pierce corporate veils and work down chains of authority and agency. And yes, we have a lot of learning we can just borrow. We don't have to invent something.

Sen. Richard Blumenthal (D-Conn.):

But it doesn't really matter for the purpose of agency whether that surgeon is a robot and in the future there will be a lot of surgery done by robots.

Paul Ohm:

I suppose. It still makes me a little nervous because when we begin to walk down the road of saying that they are an agent, I'm just a little worried that that leads to pursuing rights and things that.

Sen. Richard Blumenthal (D-Conn.):

But the hospitals should be responsible.

Paul Ohm:

Absolutely. I agree 100% with you and I'm probably just disagreeing on the vehicle we used to get there.

Sen. Richard Blumenthal (D-Conn.):

And the way for the hospital to care about the quality of medical practice is to make it liable.

Paul Ohm:

And not just the hospital. There is a consultant usually, there's a developer, there's a deployer. There are many other responsible parties with an opportunity not to botch the surgery in addition to the ultimate robot and all of them should be surgery.

Sen. Richard Blumenthal (D-Conn.):

And if we made the social media platforms liable for suicides or self-harm by teenagers when it knew or had reason to know that there was toxic content driven at them by their algorithm instead of giving them a liability shield under Section 230, in effect, making them responsible for that algorithm.

Paul Ohm:

Absolutely. Yeah. And the reminder in all of this kind of entire line is human beings at companies are making the decisions that have these downstream consequences. And so it's not enough to just say it's our technical gods who are doing this. There is a human who's accountable. Yeah.

Sen. Richard Blumenthal (D-Conn.):

And what has really made the world safer at the end of the day is attributing responsibility. Not just moral responsibility, not just reputational harm, but dollars and cents to those companies. And they are no longer struggling, nascent, small enterprises. They are multi-billion dollar giants.

Paul Ohm:

If I may, I happen to teach a lot at Jesuit University and I would still rather go with legal responsibility over moral responsibility, even though I understand the impulse.

Sen. Richard Blumenthal (D-Conn.):

Let me just ask, and I'm about or I am just a little bit over my time. Is there anyone here who thinks that a completely voluntary system with only moral responsibility is going to be enough to deal with the dangers? Anyone here? I see heads shaking, so I agree. And one last question for Mr. Painter. Are you satisfied that you had complete disclosure or cooperation from OpenAI in the course of your investigation?

Chris Painter:

Thank you for the question, Senator. So the agreement that we reached to do the investigation, it was a mutual agreement that we reached with them. When we went into this, we were aware of only the Hugging Face hack specifically. I think before the incident, we described on our website a kind of set of questions that we think a full misalignment investigation should go into in any of these incidents. And we've since updated that publicly with a even longer list of questions that we think should be investigated. The agreement that we reached with OpenAI was a subset of those questions. Right now, companies have no obligation to work with us on this and we have to kind of make judgment calls in every case about what information we want to prioritize getting to the public. I do think there's an interest in getting some information out fast.

I think that's good to prioritize doing something quick to get information out fully. But my hope is that in the future we can do really thorough investigations that look for all of these incidents across multiple companies that look at things like sampling the models and seeing what other models would do in those same situations.

Sen. Richard Blumenthal (D-Conn.):

Let me just say in closing, and I appreciate going a little bit over my time, but I know that Professor Ohm, maybe this is directed to your students more than to you. I was Attorney General of the state of Connecticut. I used to say that lawyers are often private attorneys general because they use the law to impose liability when there is wrongdoing. It is beginning to work on social media, some of the verdicts that we've seen by civil litigants there. It didn't work in the tobacco area. I was an attorney general have to lead against Big Tobacco because of various issues with liability and frankly, the deception of the tobacco companies. Car safety, it has worked somewhat. Asbestos, it has worked. But to your students, they can often vindicate rights and enforce responsibility by using the law creatively. And I think it's our obligation in Congress to provide the tools and the rights that are necessary to protect people.

And very often it is a civil litigant acting as a private attorney general or their lawyers who help to protect a whole class of people who may be vulnerable. Thank you, Mr. Chairman.

Sen. Josh Hawley (R-Mo.):

Thank you, Senator Blumenthal. I just want to ask a few more questions and I think I detect the beginnings of a, dare I say it, a consensus here on this panel, which I think is very significant and I just want to pursue this a little bit more. Why shouldn't we start by imposing clear liability on the companies for the actions that their agents, quote unquote, their AIs, as you've been calling them Mr. Kokotajlo, that their AIs take, or that would be on the developers and similarly impose liability on the users of AI when they use AI in a reckless fashion? So in other words, why should we just say it's a standard doctrine of American law, if I can put it in layman's terms, if you break it, you pay for it. If you cause damage, you've got to make it right. If you cause somebody harm, you've got to make them whole.

Right now, as you said earlier, Professor Ohm, under our law, because of the unique position of these AI agents, it's difficult to know exactly how the doctrine would deal with them. They, the companies, may be able to escape liability. I'd suggest that maybe we start with some legislation, federal legislation that would say as a matter of federal law, and we can use existing statutes to do it. I propose using the Computer Fraud and Abuse Act, it's already on the books. We can just update it to say that for developers, that's the companies, if you develop these agents and train them in a reckless fashion and they go on to hack or crash or destroy stuff, you're liable. For users of the AI, if you've got these users like the guys who use the Anthropic to hack into OpenAI, if you use an AI model or agent in a reckless fashion, pick your liability standard, then you should be responsible.

Why don't we just do that? Why don't we assign some liability? And here's my thinking on this is that I don't have any confidence, zero, less than zero, in Congress's ability to keep up with the technology, let alone anticipate it. I mean, there's just no way. We're not going to be able to say, well, you guys have made great suggestions about here's the sort of reporting regimens we need today, and I think that's all great. It'll probably be different tomorrow. I mean, you were saying a second ago, Dr. Hobbhahn, that pretty soon the AI agents will be able to speak to one another. We won't be able to track it. But if the AI companies who were developing and training these agents knew that they were going to be liable for the bad stuff their agents did, I just am willing to predict, go out on a limb and just say that I think maybe they'd pay a little bit more attention to the development and training of these models.

So let's just start with that. Professor Ohm, am I off track here? I mean, isn't this a good place to begin what we've been discussing here? Let's impose both civil and criminal liability, but let's make sure that we can actually open the courthouse doors so people can say, "Listen, OpenAI. If your agents hack Hugging Face or crash a hospital system and we can show that you trained them recklessly or you developed them recklessly, you're going to be liable."

Paul Ohm:

Absolutely. And once again, I would start with making sure we do not preempt state law. That's step one.

Sen. Josh Hawley (R-Mo.):

And that's because tort law is mostly state law and it's already on the books. Absolutely.

Paul Ohm:

And state UDAP laws, unfair and deceptive acts and practices, that's a useful tool in the way you're describing. I like the idea of both individual victims and state attorneys general having a role in playing this. The CFAA is not a well-loved statute. Maybe offline I could talk to your staff about other things we might fix. It's not going to be the most popular vehicle for this, but absolutely I'm with you that finding the vehicle to make sure developers and deployers, and it's key that we have both of them, are held to account for the harms that happen from their uses 100%.

Sen. Josh Hawley (R-Mo.):

Yeah. I mean, let's just give it a practical example. And Mr. Painter, you tell me if I'm wrong. My understanding is that in the Hugging Face hack from the OpenAI agents or by the OpenAI agents, OpenAI actually had safety mechanisms off. Is that right?

Chris Painter:

Senator, that's correct.

Sen. Josh Hawley (R-Mo.):

All right. So let's just say that one more time.

Chris Painter:

They had monitoring systems off. I forget the... Yeah.

Sen. Josh Hawley (R-Mo.):

Monitoring systems were off at OpenAI during this attack. Now, if OpenAI knew for darn sure and certain that they were going to be liable to the tune of who knows how much that a jury might award, don't you think maybe they would be a little more careful about monitoring what their agents were doing? Or Mr. Kokotajlo, you've been talking about the amount of attention and money that's devoted towards moving towards, what did you call it, recursive self-improvement versus making the models more efficient, making them cheaper, making them more user-friendly. I know nothing about the AI business. I don't want to tell them how to run their business. I don't know. I'm the wrong person to ask. But if we told them, "Listen, if your AIs do bad stuff and harm people and you've been reckless in it, you're going to be responsible, don't you think that that would reshape their incentives a little bit?"

Daniel Kokotajlo:

Thank you, sir. Yes, I completely agree. Although I must say, I don't think it would go far enough. I do think that we should hold them liable for the damages that are caused, but I think if that's all we do, then they're going to continue to take reckless gambles and eventually they'll be taking a gamble that's big enough that we will suffer catastrophe.

Sen. Josh Hawley (R-Mo.):

Dr. Hobbhahn, let me give you a chance to weigh on this because you said something the other day that I think is very interesting and we've got, I think, a board of it. You said it's so economically valuable for these companies to build these AI systems. They're going to limit test how much misalignment you can get away with. In other words, naturally, this is just the market forces. It's the way the market works. It's competition. Their incentive is to innovate, innovate, innovate, innovate, and they're going to blow past what we've been calling safety limits and safety concerns. They're going to blow past that unless we somehow make them pay for the harms that they cause. Isn't that fair to say?

Marius Hobbhahn:

Yeah. So what I meant with the statement is sort of a general comment about the situation on the ecosystem that we have right now where capabilities are moving drastically and increasing drastically while safety alignment and techniques cannot keep up. And then there is intense pressure to release models regularly, which can lead to premature deployments and incidents like this. And so I'm worried about the voluntary nature of the current ecosystem and how it plays into the incentives of the overall companies.

Sen. Josh Hawley (R-Mo.):

Well, I just think for my part, I think it's time to change the game. And I think we don't have to invent an entirely new system. We don't have to reinvent the wheel. American law has worked beautifully when it comes to other products for literally centuries now. And as you pointed out, Professor Ohm, actually goes back to the common law that predates the founding of this country. I mean, it's worked pretty well for our civilization. And the way it works is because the law is itself a form of a marketplace, which is when individuals get hurt, if you give them the right to go in and say, "Hey, I'm going to hold the person who hurt me accountable," guess what? People change their behavior. And right now I'm worried, my biggest concern with AI among many, many, is that right now the incentives are all misaligned.

These companies have every incentive to race, race, race against each other. They don't have any incentive to think about, "Gee, what happens if I crash this hospital? Gee, what happens if I crash the banking system?" They just think that's somebody else's problem. And we know they think that because when they talk about the doomsday machine they're building, the next words out of their mouths are, "And we would really like more exceptions from the government rules. We don't want to be held accountable under antitrust. We want to be able to collude." That's exactly the wrong thing to do. We should go in the opposite direction. Hold them accountable and let people vindicate their rights. That's my view. All right, Senator Kim, your turn.

Sen. Andy Kim (D-N.J.):

Thank you, Chairman. Mr. Hobbhahn, I'd like to start with you. We talked about the recursive self-improvement and the speed and the lack of human engagement in that process. I largely agree with you, and I think a lot of people do when it comes to the need for embedded evaluators, for instance, in monitoring. I guess I just want to get a clearer sense of what is possible when it comes to embedded... If we embed evaluators to oversee a recursive self-improvement system, what is it that they're going to be able to see? What is it that they're going to be able to monitor and understand given the speed with which that's happening?

Marius Hobbhahn:

Yeah. So what I think we need for embedded evaluations is employee equivalent access. So access to all training data, all development pipelines, the ability to speak with internal staff freely and without repercussion to understand what the situations look like and learn about the systems, the ability to check internal deployments, so how the models are being used inside of the company and look at the logs and so on. And then all of this, all of the results have to be public such that the general world has a better window into what is happening and can take more adequate responses. And I understand that there are confidentiality concerns, and I think it should be possible for AI companies to redact confidential information from the report, but that should be under meta transparency clauses, which means that in the report it will be stated that a redaction took place and the general process should be known.

Sen. Andy Kim (D-N.J.):

I don't disagree with a lot of what you said there. It makes a lot of sense to me, but I still think that it gets to this question of just how much are humans able to comprehend of what is actually happening when we reach that full level of RSI? That's something I'd like to think through. I'd like to just stay with you. We talked about open weight models, open source models. Who is doing the evaluating and monitoring there?

Marius Hobbhahn:

That is a great question, Senator. Right now, most evaluators are not working, or I'm not aware of any evaluators working with the open source companies, and this is largely a capacity constraint. In general, models should be evaluated and they should be tested by third parties, but due to the small size of the field and the importance of rigorous testing, it is currently not possible to cover everything.

Sen. Andy Kim (D-N.J.):

No, I think that's a really important distinction to make here. As we were talking, my colleague Senator Blumenthal was just talking about just does anyone think that it's enough for these companies to have to handle this voluntarily? And I think largely a lot of people say no, but it also just kind of shows, again, when there's no structure to this, who's going to be looking at these other models? Who's going to be engaged in this and doing it right? So that's something I want us to make sure that we're following up on. I'd like to just spend my remainder of my time just talking through what the chairman said. I think we're finding areas of consensus or common ground amongst the witnesses here, but also I was listening to our colleagues here and I do think that there's some avenues here that we need to make sure we're engaged on.

First and foremost, we are the Homeland Security Committee. And I think you're hearing from every single person on this committee that we want to secure our country, that we're worried about that. I'll be honest with you, we should be worried about that even before the advent of AI. We already knew that China was able to get into so many of our systems even before using AI tools. We have a lot of other concerns that are out there. So Mr. Gaudette, I just wanted to check in with you here. You raised CISA, for instance, can play a big role here. What is it that you want us to be able to move forward here as a committee in terms of authorizations or fundings or other things? What is it going to take for us to have at the federal level the type of action that is going to actually keep us safe?

Kurt Gaudette:

Thanks for the question, Senator Kim. I think clear authorities is critical. I think identifying the coordinator for these type of activities, again, because many of the owners and operators out in the field don't know who to go to or who to call for these types of situations. And then the other piece here that I mentioned in my opening was having some type of unified national OT/ICS incident response plan, because we see the problem before us. And I think for many years we thought, or a lot of Americans thought that's an over there problem. We saw the attacks in 2015, 2016 timeframe on the electric system that took down, I think something near 250,000 customers for multiple days. We saw in 2024 with the FrostyGoop attack in Lviv, 100,000 people lose power in the dead of winter. These are real attacks. They happen all the time.

And we need to internalize that and realize that it can happen in our country. And now we've been starkly made aware that it can happen in our country.

Sen. Andy Kim (D-N.J.):

That’s right.

Kurt Gaudette:

Particularly with the July attacks. You add AI to the mix and we've got this real interesting situation. So the need for something like a national wide response plan so that it's clear for our owners and operators out in the field to know who they go to for guidance, for help, how we would respond. And we've done a good job. I think the 91st Brigade in the Virginia Army National Guard has done a great pilot with a large utility there and showed how an entity could respond in this activity. And if you had someone like CISA coordinating these various responders in some type of deliberate plan before it actually happens, would be incredibly helpful.

Sen. Andy Kim (D-N.J.):

Yeah. And this is something we encounter when we're dealing with, again, the release of Mythos, I was engaged with a number of banks and financial institutions, seeing them getting access early to be able to patch up and that was successful to them. But I'll be honest, I had a lot of other companies and organizations coming, well, what about us? Who's controlling? Who gets access to this early? And certainly when it came to critical infrastructure, very much feeling like they were left out of that. Mr. Hobbhahn, I just want to end with you here. Another area of question was about the ability to secure our innovation lead. A lot of concerns about China. I think we all have those concerns, but I guess I just wanted to ask you, in my opening, I said that I believe that there's a way in which we can ensure and grow our innovation lead while still creating space for us to engage in this effort on safety and security.

Is that something you believe in as well? Do you feel like we can create that space?

Marius Hobbhahn:

Yeah. So I think we have to be careful about false dichotomies here, and I think it is important that we keep both areas of concern in check. And I think it is possible to further innovation and help consumers and so on while keeping track of rogue AI and China.

Sen. Andy Kim (D-N.J.):

I'll just end on this. Right now before our Congress this year, we have a number of pieces of legislation that are in the NDAA and other means which we can use to be able to secure our innovation lead. Actions like what Senator Ricketts [R-Neb.] and I are doing in a bipartisan way with a MATCH Act that would control semiconductor chip manufacturing equipment or the legislation I'm doing, the BLADE Act, which is about distillation and something that very much has a big say in terms of what China's able to be able to do. So I just urge the Senate, yes, we should engage on this issue about safety and security. We should not allow the questions about our innovation lead to stop us from engaging in this. Yes, we can have our concerns about China, but that is not an excuse not to act when it comes to the security of the American people.

And with that, I yield.

Sen. Richard Blumenthal (D-Conn.):

I would just like to emphasize the point that Senator Kim made. Safety and innovation are not contradictory. They're not exclusive. In fact, they go together because without safety, innovation is going to be stymied and stopped. So I think this is a false dichotomy that maybe some in the industry have raised. I think there is a possibility for some international agreement, but certainly in terms of what we do in the United States of America, safety and innovation ought to be complementary goals. And I would just second as well what the chairman said. We ought to work on legislation that imposes or clarifies that there is civil and criminal liability. I hope we can use the next 30 plus days to work on legislation that we would introduce. It's not a substitute for some governmental oversight. I don't want to be misleading here, but it is an important element of the protection that people deserve.

There's still a need for some standard enforced by a government agency, but civil liability is often an important element of protecting people as well as criminal liability. But of course, no individual plaintiff is going to enforce criminal liability. That again has to be the responsibility of some public agency. And I think Professor Ohm, you raised an important point about preemption.

States continue to be the laboratory of democracy. We maybe have AI labs, but states are the laboratories of legal process and progress. And I think we ought to be mindful of what they have to contribute as well, speaking as a former state attorney general. And so thank you all for being here today. And thank you for the folks in the audience who are interested in this topic because we need the ideas and suggestions and criticisms of an active and engaged community. And thank you to the panel.

Sen. Josh Hawley (R-Mo.):

Well, let me just thank you, Senator Blumenthal. Let me just end with this. Professor Ohm, you touched on this about how your students feel. I think there's so many Americans maybe sitting here in this audience today who feel powerless and realize that these companies have tremendous power. And in contrast to that, we feel as if we just can't do anything. But that feeling is itself a myth. We can still act. We can still protect our rights. We can still vindicate that fundamental American value, that fundamental American principle of personal responsibility. And I think that's really what we're talking about a lot here today. I mean, so much of our economy to say nothing of our entire legal system rests on the basic principle of personal responsibility. And it's time these companies assumed some personal responsibility. It's time that we reaffirm that together. We've got to make some choices now together as Americans altogether for this technology in order to make sure that the technology works for us and not the other way around.

And those are some very big choices to make. They have very serious moral ramifications and it's time that we made them as a society. But I will go so far as to say, I sort of suspect, I think most Americans actually agree. I don't think there's a lot of disagreement on this issue. I think the American people are saying, protect our rights, give us a voice, give power back to us. And we do that by imposing responsibility on these companies. We should start with that today. Thank you again for all of the witnesses for being here. Before we close, I'd like to know the subcommittee's received statements from the Missouri Hospitals Association, the Missouri Bankers Association, the American Hospitals Association, the AI Policy Network, and the America First Policy Network without objection. They'll be made part of the hearing record. The record for this hearing will remain open for 15 days until Thursday, October 15 at 5 p.m. for the submission of statements and questions for the record.

And with that, the hearing is adjourned.