AI Governance Reaches Crisis Point Ahead of Trump-Xi Summit

Audio of this conversation is available via your favorite podcast service.

On Friday, September 11, I moderated a discussion at the Institute for America, China, and the Future of Global Affairs at the Johns Hopkins School of Advanced International Studies in Washington, DC as part of an event called “Getting Tech Right US China Policy for a New Era.” Amid a frenzy of headlines about AI safety and just two weeks out from a planned meeting of US President Donald Trump and China’s President Xi Jinping, the discussion was organized around the question, what does AI governance actually demand right now? Joining me were:

• Amba Kak, AI Now Institute
• Kat Duffy, Council on Foreign Relations
• Miro Plueckebaum, Oxford Martin AI Governance Initiative
• Jack Shanahan, SAIS ACF and the Center for a New American Security

What follows is a lightly edited transcript of the discussion.

Justin Hendrix:

My name's Justin Hendrix. I'm CEO and editor of a site called Tech Policy Press, which focuses on issues at the intersection of tech and democracy. I'm normally based in New York City. And when I talk about things happening in the specter of the US Capitol, it's not literally right above my shoulder. So this is a great pleasure to be here and I'm thankful to the organizers for inviting me to help moderate what I think will be a very great session today.

And I'm also planning to use the audio from this conversation for the Tech Policy Press podcast. So I have a kind of habit on the podcast of making sure we do an audio check and let people announce themselves, say their names out loud. That helps me to avoid butchering pronunciations when I do my podcast recording. So I'm going to stick with that and I'm going to just go down the line and ask folks to state their name, their title, and their affiliation so that everyone knows where you're hailing from.

And Miro, I'll start with you.

Miro Plueckebaum:

I'm Miro Plueckebaum. I'm the strategy and research manager at the Oxford Martin AI Governance Initiative. And I also run a Singapore and London based technical AI governance research lab called SASH. And

Justin Hendrix:

Kat.

Kat Duffy:

I'm Kat as in meow, Duffy as in Duff Beer on the Simpsons. And I am a senior fellow for digital and cyberspace policy at the Council on Foreign Relations. And I'm also the director of our LEAD AI initiative.

Justin Hendrix:

Amba.

Amba Kak:

I'm Amba Kak. I'm the co-executive director of the AI Now Institute.

Justin Hendrix:

And Jack.

Jack Shanahan:

Jack Shanahan, retired United States Air Force, but thanks to Afra, I'm not going to say that anymore. I'm an OPC. I'm a one person company. So I'm very proud of that. I am a senior fellow with ACF, but also an adjunct senior fellow with the Center for a New American Security.

Justin Hendrix:

Well, I'm looking forward to this conversation today. We've got a range of perspectives, but I felt maybe I'd start with just reading a couple of headlines and I almost just randomly chose these from my newsfeed. But one of the things that folks have been talking about in the communities I travel in has been just this sort of frenetic pace of news, the frenzy of information, the kind of wild set of events that have taken place over the last couple of weeks, which if you cover tech and you do journalism, at any point in past, any one of these stories would've seemed as if it would occupy days or weeks or months of discourse, and yet they sort of fall away almost as instantly as they're published. But headlines like the Wall Street Journal, “Anthropic says Iran used its American AI model to target US Navy warships.” The Financial Times, “Houthis used Anthropic AI to try to build ballistic missiles”. CNBC, “Extinction warnings ramp up as more OpenAI, Anthropic researchers join calls for an AI slowdown”. And then maybe a little closer to the Capitol here. Politico, “Hawley presses OpenAI over rogue Hugging Face hack.” And finally, Wall Street Journal, “Congress is suddenly waking up to the AI doomsday threat.”

I literally could have drawn any number of headlines like this from the last couple of weeks. But the organizers, if you go back to what they wanted from us for this panel, they wanted us to offer a kind of a possible affirmative vision for AI governance, what we should do, what AI governance demands now. And yet we know that just down the street, senators, Congress, congressional representatives are apparently in a flutter. They're putting out bills rapidly. They're trying to draft legislation in some cases in response to these headlines.

So my first question for you all, and in fact, I'd like to put China out of our minds for just a moment. If you were to walk down there, walk down the street, short clip and tell folks what they should be doing to get AI governance right, right now, what would you emphasize? What would be the thing that you would say should be the kind of framing thought?

And perhaps Kat, I'll put you on the spot to go first.

Kat Duffy:

Yeah. For a year, I've had a series of AI agents I've consistently been refining to help pull in all the news that I need to see. And now even the agents can't keep up with the news cycle. They're like, "You only wanted me to surface 15 things. What do you want me to do?" I mean, I think putting China out of the equation, the most important thing I think we should be asking ourselves is what is America best at? If America is going to lead in this technological transformation, we're arguably going to lead best by relying on our strengths and leaning into those strengths. And historically our strengths have sat in openness, in due process, in accountability, in supporting innovation, in supporting incredible influxes of talent and in recruiting that talent in. And so I would say that a lot of what we needed to be doing with AI governance right now is sort of returning to the fundamentals.

I think American citizens need to have a better understanding of the way that this technology is being developed. I think they should have a much better understanding of the way that it's being financed. We have massive amounts of opacity in our private capital and venture capital flows in private equity. Americans don't know who's funding what, where that money is coming from, nor frankly, do many of the innovators know who is funding them or who is funding their work. There's no transparency on this at all. Americans don't know what their tax dollars are going to and what they may not be going to. I think when you see the concerns right now about data centers and the impacts of those data centers, you're also seeing, for example, the data center in Memphis that was really producing terrible results for the populations close to it, that was also a result of permitting processes that had been shortcut.

So it wasn't that the permitting process didn't exist or that the governance didn't exist that would've made that data center more safe for the community. It's that there was such a rush that the permitting and the governance that was already in place to protect citizens was thrown by the wayside on the altar of supporting this technology. And so that's where I would say let's return to where we're best to begin with.

Justin Hendrix:

Okay. So transparency and embracing some of those principles that have made America America in your view. Amba, what about you? What would you emphasize to the folks down on Capitol Hill right now what AI governance demands?

Amba Kak:

Can I do one second of throat clearing first, which is I think there's at least one person in the audience that agrees with me. I can't see him anymore, but it feels like every time Sam and Jessica, you organize these conversations, they really do feel like a refuge from every other conversation happening outside this room. The most optimistic note I guess I can strike at this moment is that it does feel like the distance between the conversation or the kinds of discourse we're trying to advance in this room feels less further away than it has at many points before. I was thinking this morning that when Ted Cruz comes out in the midst of this news cycle to say that they'd rather it be our killer robots than China's, I do think we've got -

Justin Hendrix:

He needs to think about that one just for a second.

Amba Kak:

It didn't land well. It didn't land well. I think there is a visceral mainstream backlash to just the ridiculousness of the race to the bottom logics that have been perpetuated around this frame. And so that's the good news. Maybe the flip side is you never waste a crisis is maybe the most important policy adage, but equally I think not every policy crisis is equal in the opportunity it creates. And what I'm sitting with in this moment is that the nature of the extinction risk whistleblower moment that we're in right now, it seems like all roads are in fact leading to a policy imagination that looks straight back at the industry for answers to problems that they are creating. And that is what maybe gets us closer to, are we going to waste this crisis or even worse, are we going to undercut all of the public momentum that has come not just from tech whistleblowers, but from data center activism, from parents groups?

Is that going to be undercut towards a policy regime that is essentially functioning as a checkbox certification legitimization of business as usual rather than real accountability and real oversight? And there's many ways in which this is happening. I think the fact that the Bernie Sanders Kassar bill hinges on a definition of super intelligence that is frankly undefinable and would leave most of industry off the hook is a good sign, like the smell test that something is off with that line of that sort of solution being what we see from this or what we make of this crisis. Okay.

Justin Hendrix:

So a lot of threads there, but put super intelligence out of our minds, focus on what the public needs, the public interest rather than the corporate interest. Lots there, but we can come back. Miro, what about you? What does AI governance demand now maybe with your verification hat on?

Miro Plueckebaum:

Well, the examples you gave to kind of set us up already illustrated that the stakes are really rising. AI systems are becoming more powerful. The harm they can do is increasing, but so are competitive pressures between companies. The prize is starting to feel quite real now, regardless of what you think of the valuation of companies being overvalued or not. The valuations are no longer imagined. They're actually generating quite a bit of revenue. And what that means is that in cases where companies do want to behave responsibly, they may get penalized by the market. So you mentioned briefly the Hugging Face incident in the introduction. In that case, OpenAI tried to act responsibly and pause the training of the model for a period of two weeks. Now in this case, they chose to do it unilaterally. No other company paused, but we can't be living in a system where the responsible actor can't behave responsibly because they have to worry about their competitors undercutting them.

So if I were to walk down the street, I would ask folks to start thinking about what type of a regime can we build that allows responsible actors to behave as such when required without being undercut by actors that may not want to behave as responsibly. And that's an open policy question. We're focusing on building tools that can verify whether or not actors have, for example, actually paused, but at what point we ought to pause and for how long and what others should do is a question that we need folks down the road to help figure out.

Justin Hendrix:

Okay. Jack Shanahan, I know you've worked directly on these questions around US and China, but again, keeping China off the table for the moment. If you were to go down the hill, AI governance, what does it demand now?

Jack Shanahan:

Well, first thing I'll do is go back to ground zero, so to speak, on September 11th and redefine what do we mean by governance in the first place? To me, it's rules of the road. I spent 36 years in the military. I can't remember ever using the word governance, but governance ruled everything we did. We called it policy, we called it directives. What are you allowed to do, what you can't do, what you can do under certain circumstances, who is held responsible and accountable when something happened? All those things were what I call rules of the road. We need that in AI in a way that we just haven't had before. This will be a little bit provocative. I'm not sure people would disagree with it, you might take exception with certain ways. I'll say this, but at the end of the Cold War, we entered into a period of unfettered technological libertarianism.

The government was not in the equation with what was going on in Silicon Valley. The government wasn't the customer for these companies. They raced out in front of everybody else. The government didn't understand the technology, had no interest in figuring out how to govern it properly. And the next thing you know, here we are today and people saying, we blinked twice, we look around and this technology's outraced all of us. So now is the time to reset and say, what are the rules of the road? Some of you may be familiar with the term Collingridge dilemma. I am very sympathetic to the Hill in many respects is you move too fast on a technology and try to put out policy, you move too fast, it's the wrong policy and you screwed something up. You wait too long, the technology has taken root. Now what are you going to do about it?

It's really hard to get it in past a certain point. I think people are very serious on the hill about trying to find where is the right time to come in. I think we all sense that that time is now. These incidents are just making it clear to everybody, if you don't act now, something serious is going to happen and we're all going to regret that. So for me, it is a very basic term. It is rules of the road. That's a very easy statement for a lot of different things that have to be done to get it right.

Justin Hendrix:

So I want to pick up on the point of time. It's really the second thing I wanted to get to a little bit. The question has been somewhat, is there going to be some major event that finally gets everybody off the couch and to take action? It does seem like right now maybe there are lots of important things that are happening or frightening things that are happening that together people are beginning out there in the real world. Normally people who don't think about AI policy every day are piecing together in their mind and they're building their own imagination for is this a crisis? I don't know. How long do we think we've got to take advantage of a sort of push towards there needing to do something and how do we avoid doing something when perhaps in some ways we should have done nothing? How do we avoid making mistakes in this moment?

Does anyone have a general view on that? How do we take advantage of the moment without somehow, I don't know, making a mistake in the process?

Amba Kak:

My initial reaction when the first part of your question is when will that crisis arrive or when will we be there? And I think we're there. The question is no longer are we there yet to the crisis that opens up the political window. The policy window is before us. And maybe the best evidence of that is the fact that with, I think no exception today, every single AI firm is standing up to support in theory the prospect of regulation. The terms though are the ones that I think the fight right now is about who is going to sort of set the terms of the debate. Is it, as Jack is saying, going to be this sort of amorphous notion, to me, a softer notion of governance which really is often self-governance in other clothing or is it going to be much more straightforward leverage that the public can actually assert over this industry?

And I think that looks like rules. It looks like liability. It looks like the hard, the concrete enforceable rules. It also looks, to be honest, for one, let's start with making sure we're not wiping off the rules that do exist off the books because that's as a sort of reality check. We're at a moment where in the last few years we have seen AI exceptionalism that I think no industry has had it this good in some ways. So it isn't just no new rules for this industry. There have been several attempts, moratorium in many different shapes and forms, pushes to try to immunize this industry from even adhering to the law as it exists on the books. So the DOJ is stepping up in the New York Times case, also in the Memphis permitting case, and their argument is China, that if the US is to hold national security advantages, then whether it's unpermitted turbines in Memphis or a copyright suit, you cannot touch the AI industry.

And I feel like that the moment to reckon with or challenge that premise is here, but my sort of anxiety and also optimism is like, can we make the best go at it?

Jack Shanahan:

If I could jump in, I want to go back to something you said at the very beginning about this time element where you read these headlines just from the last, what, 48 hours, 72 hours. What makes this technology so different than I think with anything any of us have experienced in the past is the time you would have to actually work on things where now you've got this accelerated rate of development, this breadth and depth and diffusion and this rate of adoption that is so different than anything before. It feels like that yes, the time is now to do something, but the pace is so brutally relentless that human nature says, no, no, give us some time to focus on the headlines you just read. Then the next thing is about to happen. And maybe it's only when that catastrophic event happens that we really do see action.

We wish I could be, I know you want to leave on an optimist note, so maybe we'll get there eventually, but at heart here, the signs are not really great for solving these problems right now with regulation until something else really changed, whether that's the election, whether it's a catastrophic event. But I think this time pressure is different than technologies that we have experienced in the past.

Justin Hendrix:

Kat, Miro, this time pressure, anything to say?

Kat Duffy:

I mean, look, if you're on the hill, your time pressure is the midterms, right? I mean, if you're looking to people to govern over the next few months as opposed to run, I wouldn't hold your breath.

And so it may be less the crises that should be responded to because those aren't necessarily the crises that the electorate is going to hook to as being their kitchen table issue, their thing that they would go to the ballot for. What's really interesting is the polling that's been coming out that is showing that unprecedented amounts of agreement across evangelical Christians and liberal progressives absolutely polling at the same numbers and like 80% use of AI in K-12 education. There are going to be these issues that I think when those of us who are foreign policy thinkers or operate in the NATSEC space and cyber, those other areas, I think are underestimating potentially what Americans are going to be most concerned about. And that type of polling in the lead up to elections between that and the data centers, if I were saying where I think governance might sit or governance proposals might sit, I suspect it will sit in wherever electability sits, not necessarily in where the triaging actual crisis does sit or should sit.

Miro Plueckebaum:

I'm not as policy savvy as my fellow panelists, but just one observation that I'm going to make in a personal capacity in response to the way you framed the question, you said, how do we avoid doing something wrong? And that implies some sort of hesitancy over what to do from the policy point of view. And it's not something I sense a lot from the private sector. The adage is move fast and break things. And on the policy side, it seems to be much more hand-wringy and hesitant. And if policy wants to keep pace with technology, it needs to perhaps move into a much more fast moving, iterative direction where it is understood that you're just not going to get it right every single time, but you need to be able to adjust very quickly on the fly in response to a changing landscape as the companies are doing.

I mean, they're very much playing to win. They're not just. Yeah. So that's just a quick observation there as someone who's, again, not very policy savvy. Well,

Justin Hendrix:

And I think it's a good observation and those of us who've worked on tech policy issues, perhaps one of the things that's working against us is that we are convinced that if a law passes, for instance, we should be prepared to live with it for 40 years because nothing is going to change in the intervening time. That seems to be what we all operate on. But let me now kind of bring the China question back in because Amba mentioned, for instance, the DOJ's effective support for Elon Musk's xAI in Memphis, the argument effectively that we can't really expect to hold xAI to account for poisoning the air in this Black community in Boxtown in South Memphis because we've got to beat Beijing. Because

Amba Kak:

DOD needs Grok.

Justin Hendrix:

That's right. Let's bring China back in. Let's bring the question back in. How does that change anything about your prior suggestion about what AI governance demands now? How does that inform? And Jack, with this one, I'll start with you.

Jack Shanahan:

This is a conversation Paul Triolo and I were having last night based on watching his podcast last week with Kendra Schaefer on this very topic is to have this governance conversation between the United States and China, which I think is absolutely essential. I can't question that question is what does it look like? But to do that, we have to get our own governance house in order and China has to get its own governance house in order. This is not arms control is owned by this one organization in respective countries. It's AI that brings in everything. It brings commerce, it brings treasury, brings in defense, it brings in state. Keep going down the list and the equivalents in China. So first of all, it's a very complicated, messy piece to go down the road of who are we talking to? Why are we talking about it? Exactly what are we going to be talking about?

These are issues that I hope are being resolved. I'm not seeing a lot in the public discussion about on the road to the 24th September meeting, but there is no better time to get going on a conversation. Why would two AI superpowers, do these two AI superpowers want to have this conversation in the first place? I think you start there. We have mutual vulnerabilities and mutual risks when it comes to AI. Without those, I think all you're doing is having trade-based conversations about trade imbalances, tariffs, and all the things that we have been doing vis-a-vis US and China. Now with this technology is showing capacity to do things that are troubling people globally, there is no better opportunity to get going on the conversation, but you can't do that unless you get your own governance house in order. I don't know if we have that in order yet.

So if I were to say worst case, it's actually not the worst case. The worst case is not talking at all. The sort of less worse is we just go in and get started. Okay, that's all right as a starting point, but it can't be a one-off conversation. This is going to be an institutional dial. Let the track twos do all the work that track twos and many people in this room may be familiar with or involved in track twos. They're there to do a lot of the homework before the track ones get going. In this case, I guess the last thing I'll say is track one by itself isn't going to be sufficient because these are technology companies, unlike any other technology that we've worked in the military in the past. The technology companies have to be involved in the conversation exactly how they are, who they are.

Those are big questions that have to be addressed both in China and in the United States. But I don't see how you have a very fruitful conversation on AI governance without the technical expertise to go along with the policy expertise. Those have been two different circles that need to be a Venn diagram before we get into really good conversations on governance between the US and China.

Justin Hendrix:

So maybe before we move on from you, how confident are you that there's a lot of folks who are wondering about what will happen in the next couple of weeks. It's not even clear. Is it clear yet if we're actually having an AI dialogue? We don't really know, do we?

Jack Shanahan:

I don't. I hear rumors. I hear things as they say in New York. I hear things. Something will come out of this. So there is no question. I don't think there's any question. AI will be on the agenda when the two presidents meet. How much will be done in advance of that? I feel not so great about that, although some people are telling me, nah, some work is going on, you just don't see it. Okay, that's fine. Now we'll see what happens. More likely to me is September 24th comes and goes, then there's an agreement for the two sides to get together. Now you start looking at what are the parameters of the dialogue? Is there a separate mil-to-mil talk? I mean, all those things can be worked out. I just see the appetite still is there, but the release of the distillation report followed by China's reaction to that felt like, oh no, are we about to say talks are off?

I don't think that's the case. People are telling me, no, no, they're still on track, but I will be surprised. If I'm wrong, please blast me on LinkedIn or somewhere. I'd love to be wrong. I don't see something really substantive happening before September 24th. Today is the 11th. We have two weeks.

Justin Hendrix:

Kat, you move in similar circles in New York international diplomacy. What are you hearing?

Kat Duffy:

I concur entirely. I think another way that I've been thinking about it is what could come out of the dialogue on September 24th where that announcement or the execution of whatever was promised, the entire world would essentially be in agreement. Xi and Trump deserve to share a Nobel Peace Prize for that. What would win them a joint Nobel Peace Prize? Where the entire world would be like, thank God that they work together on that. We are all safer. We are all better for it. And to Jack's point, there are mutual interests that sit exactly in that category. What is coming out in terms of biohazards is terrifying. All of the work that happens in CBRN, in chemical warfare, what's happening with our lack of ability right now to do truly comprehensive good faith threat sharing among frontier companies. And it can't just be the American frontier companies.

We need true cooperation in key areas. And that is, I truly wish that you could take Trump's top three advisors, Xi's top three advisors, put them in a room and say, think of everything that could win them a Nobel Prize. Start the conversations there and then move backwards.

Justin Hendrix:

Amba, can you see that Nobel Prize coming?

Amba Kak:

No, I can't claim to be an expert on that piece of it.

Kat Duffy:

I'm not saying it's coming. I'm just saying that I think it's a useful framing.

Amba Kak:

A useful thought experiment for sure. What was your prompt though?

Justin Hendrix:

So the original question, bring now China back into it. What needs to happen for AI governance, for us to get it right? And when you think about that component of it, how do we bring that in? And we've been now gone down this path of talking about the dialogue between these two leaders and whether it will produce the thing. Yeah.

Amba Kak:

I guess one thing, and I hate to bring up this example twice because it's such a blood curdling one, but I would rather it be our killer robots rather than theirs. I think, and that's obviously just the tip of a range of, I think, similar logics that we've seen bubbling up for, I would say at least almost a decade now in tech policy in one form or the other.

Justin Hendrix:

Can I just pause just to make sure everybody's on the same page. You don't want the killer robots, because in general, the killer robots, they're killer robots. They're not your killer robots. They're killer robots. They're going to kill everybody.

Amba Kak:

Yeah. And I guess the reason I'm bringing that up is because I think it's become very blatant that public safety, public wellbeing is sort of being reduced to or relegated to a bargaining chip in this great power competition. And it feels to me, and maybe this is again, I'm feeling really optimistic after spending time in this room all day, but it feels to me that that is no longer a politically tenable position to take. And that is not because of the news cycle. I think it's because over the last two years, we've seen a truly extraordinary pushback, which is now evidenced in poll after poll that the American public is not having it and that being sort of irrationally supportive of this industry is actually politically toxic leading into midterms. So I guess my note of optimism is that the world of The US AI, China AI arms race being wielded as a shield, I feel like that's becoming less, that will hopefully, again, let's manifest it, become less potent, particularly in its ability to shut down basic forms of oversight and accountability.

Justin Hendrix:

Okay. I want to talk a little bit about trust as a kind of fundamental, maybe going into this dialogue and whether there is any, but I want to give Miro a chance to take on this question and maybe to explain to the room where you're coming from on verification and why you think there's at least a technical component that may play into this. If we're going to make good AI governance decisions that also seem to sort out partly the China conversation, it will require some technical effort.

Miro Plueckebaum:

Yeah. So with China as part of the equation, the basic problem is the same. You need to set ground rules and you need to be able to determine when someone is diverging from them. The problem is you're now in a low trust environment. So in a domestic context, in theory at least, the government could just walk into a company, inspect everything that they're doing and check whether or not they've diverged from the ground rules. In an international context, especially between geopolitical competitors, you can't do that. You'd reveal a bunch of really sensitive IP, a bunch of sensitive user data. And so the question is what type of tools can we develop that enable the US and China or really any two or multiple actors to verify whether or not the claims that they're making to each other about what they are or aren't doing, for example, in a data center are accurate or not?

And there are about 50 people or so in the world right now working on this question, which is nowhere near enough. There have been some breakthroughs recently that show that building these types of tools that could verify such claims, are you training a new model? Yes or no? What type of model are you serving? Has it been tested? That these things can be proven. It would require retrofitting data centers. It wouldn't require revealing sensitive IP or sensitive user data, but there's still lots of open questions around how scalable are these tools and what type of claims do policymakers actually want to be able to verify. So that's a big open question still.

Justin Hendrix:

Jack, maybe I'll bring you in on this question too. Trust, Miro says maybe we need a technical infrastructure to help get us over the hump there. Have we made success in operating in a trustless or trust-free or trust challenge zone before, especially when it comes to these types of things?

Jack Shanahan:

First to give a nod to Miro's fantastic work on verification. I think I'll turn the whole Reagan trust but verify on his head and say verify, try to build trust over time because the trust isn't there right now. These two countries have this mutual suspicion. It's at a low point. I'm not sure how quickly it gets better, if at all, for the next few years. So if the entering argument is can two countries talk despite the fact that you're entering into it with all the negative parts of lack of trust, mutual suspicion, so forth, of course we can and we must. Soviet Union, United States, lots of examples of that. I had to go back and look. I read a draft paper on verification by five, six, seven authors. It was fantastic because it gave a little bit more history of the United States and China working together, particularly back after the Nixon visit on nuclear things.

CTBT, verification, seismic sensors that were placed and shared data between each other. Then as recently, in 2001, container security initiative, the US and China have actually worked on quite a bit despite the fact that the relationship has always been a little bit fragile. Better back in the days of post-Nixon visit, but still had lots of challenges associated with it. There's no reason whatsoever that you can at least get to the table and acknowledge you're not going to get some grand bargain right off the bat. The trust element is going to be lacking. The reason I enjoy working in the track twos, even though I'm not a Pollyannish person, is you start building relationships person by person. Over time, could that then build up into the track one level where you walk into the door and you know that that person is going to have their national self-interest first and foremost?

Always, we just accept that. But could you still start finding those areas of common agreement? Back to what was said earlier, maybe that's no advanced AI proliferation to non-state actors, terrorist organizations. Is that possible to do it? I think Miro's showing some technologies. Maybe there are ways to do that. There are entry points into this and it will be slow and it will be painful and it will have to build over time. There will be setbacks. That's okay. This is the world we live in. The danger is, back to this time pressure piece, we're used to working at state department timeframes where this technology is playing out in decades. Now we're talking about what happened with the last thing in 48 hours. So yes, it can happen and should happen and you just accept there are some things that aren't going to be resolved for a while between the two countries.

Justin Hendrix:

Well, maybe I'll bring up one other thing that just happened maybe in the last 72 odd hours or so, this WeChat worm, for instance, that was developed essentially as an experiment, as I understand it, by a research firm out in Palo Alto, not sold to the intelligence community or deployed, but rather brought to the attention of Tencent. I'm sure much to the, I'm sure happiness on some level of Chinese officials who would not have wanted to see such a core piece of communications infrastructure in that country somehow left vulnerable. Does an event like that kind of give us some hope maybe that people can make good decisions that we won't always just be, I don't know, operating either in commercial or national interest that maybe we can regard certain things as not worth messing with?

Kat Duffy:

Well, so for those unfamiliar with this story, the New York Times wrote this on Tuesday morning and quoting my dear colleague Vinh, who's right back there. And essentially this is a smaller cybersecurity researching firm that discovered the ability to create an exploit that would've been a zero click worm. So essentially you don't have to do anything except to be called by someone in your contacts who was also infected with this worm, and then your entire WeChat account could be taken over. All of your contacts could then be used to call their contacts. And so essentially had this gotten out into the world, it could have taken down basically the entire WeChat ecosystem in probably a matter of hours.

To me, this is an example of a good crisis. This is the type of thing that should have captured a tremendous amount of attention because we are incredibly lucky that that damage wasn't done particularly two weeks before these talks. But it's imperative to remember that this was an American company that went to Tencent to fix this, that went to the Chinese company to say, "Hey, you have a huge security vulnerability." And it took them a minute to get Tencent to focus and to fix this and to patch this. And so we want to be sure that security researchers all over the world in China, in the United States, and in all sorts of other countries have access to governments and to private sector partners, to nonprofits to essentially say, "I have discovered this huge vulnerability because the degree to which we are discovering vulnerabilities right now is extraordinary.

And the way that AI is also figuring out methods of connecting vulnerabilities in ways that we had never foreseen is also critically important. And so we need people who can see what's happening to be able to flag that really early. And so for me, this is, again, an example of there are the things that we need to protect regardless of what's happening at the sort of highest level relationship of these countries. And one of them is researchers access, not only to doing the research, but also to helping identify these things to build solutions for them before they go wild.

Miro Plueckebaum:

Maybe just to add a personal anecdote here, about nine months ago or so, our group found a universal jailbreak for all reasoning models that would just allow you to circumvent whatever safeguards were being implemented, whether it's a closed or open model and ask it to help you synthesize a new pathogen for instance. And one of the things we really struggled with is, well, how do we get that information to the right people at the companies and in what cadence? And are we allowed to also inform entity listed Chinese companies, for example? I'm very glad that this American company took the step to inform Tencent and it's good that Tencent isn't on the entity list that may have complicated things significantly, but this is where having mechanisms or at least steers from policymakers about what is or isn't welcomed when it comes to sharing these type of incidents is really important because these are typically not questions that technical researchers are great at grappling with and it's also not a good use of their time.

Kat Duffy:

I'm sorry, if I can just add on that quickly, I have spent decades voluntary principles, corporate social responsibility. First it was the voluntary principles and it was business and human rights, then it was corporate social responsibility, then it was ESG. Now it's anybody's game. And if anyone thinks that a company's moral imperative of today will remain their moral imperative of tomorrow when revenue has been endangered, I beg of you to think again. That is simply not the track record that has ever existed or will ever exist, period, full stop. Which means that you have to have structures in place that then give companies both the political space and the legal safety to work together on these issues, but that also gives them the imperative to do so. So that disclosure is not a matter of a calculated political risk because you think that's the right call in that moment.

It is a non-negotiable because that levels the playing field of the risk of all of the companies to say, no, we all have to disclose this type of thing. So I'm not taking on a political risk. I am simply, my hands are tied, I have to do this. And so this is an area that I think is so critically important right now for stronger governance because I think that companies, especially the frontier companies, are in a very difficult position because there is no level playing field that has been established or mandated, nor has there been the creation of any clear structure that would offer them legal protection to be able to work with each other, with partners and allies, with nonprofits, with experts when truly threatening issues arise.

Justin Hendrix:

Amba.

Amba Kak:

I was reflecting on both the example that you brought up, but then it also feels like we've hit that moment where the Hugging Face, the security incidents of the last couple of weeks, the dust seems to be settling. And I think in the wake of that, a few things are feeling sort of more common sense. The first being that these companies don't seem to on their own have a steady hand at the wheel. And I feel like that it's both of question, I think of are these systems safe and aligned? But equally, I think what these incidents turn our attention to is that it might be even more important to ask, are the systems within which they're being integrated resilient to the kinds of harms that they're introducing? And if they aren't, whether that's the banking system or the military or nuclear systems, then let's put back on the table the option that we might wait or not integrate at all.

So I think that feels like a common sense that may be being reestablished as we speak. And the other, and this came out a little bit again, if you've been spending any time on Twitter at all, you would've seen that is anthropomorphizing AI good or not, it's the hot debate of the moment. Is this evidence of sentience or autonomy? Does it matter? Is this a problem for policy and the kinds of narratives that it illuminates? But maybe I think that even there, it feels like mainstream consensus is arriving on the fact that however you fall on that question, there are a series of human decisions that were made that could have been made differently. There is certainly a degree of foresight that several human beings within and outside these labs could have exercised that could have prevented where we ended up. And if anything, I appreciate what Kat is saying, you need to, and Jack, you said this too, that this is a moment to think about bridge building and working together on these problems.

But equally, it's a moment I think to dig our heels in on what it truly means to establish independent oversight and really kick the tires too on what independence means. And it's always a really hard question because I think the sort of asymmetry of information of not just technical understanding, but I guess there are many layers of opacity in this industry and it's going to be very difficult I think to establish independent oversight. But I don't think we can give up this easy. I think we need to start by testing our policy solutions against the bullet of are they just reinforcing power back in these companies to sit as arbiters or decision makers or certainly to be grading their own homework or setting the curriculum? And I feel like the answer at the end of the last few weeks feels like no.

Jack Shanahan:

Can I just close out because all of this gets to what I think is one of the areas, one of the areas that could show some promise in a track one dialogue on AI is this idea and others have said in a previous panel, incident notification, accident databases, sort of aviation like to give that political top cover to say there are ways we can do this. We're not going to jeopardize our own national security. There are certain things the intel community will protect for the intel community's sake. China will do exactly the same thing. But if we go back to national self-interest and if it goes wrong, it goes wrong for everybody. Should we not have ways of sharing some of these catastrophic potential things happening from both countries understanding what that could be and not just both countries. Globally, is there a way to do that?

It's tough. I don't think it's a simple solution, but I think this idea of incident notification, today incident notification mostly done by social media, it's actually pretty effective because they all come out and talk about the same things, but that's different than being an official answer response to something.

Justin Hendrix:

Okay. We're going to move to questions from the audience here in just a moment. So if you've got those, begin to prepare. But this has been a great day in many ways because if you came into this room, and I doubt there are very many of you that did, with a kind of simple idea of the China versus US arms race narrative, there's no way you will leave this room without that having been severely complicated by the types of things that you've heard. But I kind of want to ask a meta question about that narrative.

There are still many people advancing it in very strenuous terms. I just read a think tank paper last week that included in it various kind of imaginations for what we should do if China reaches AGI or super intelligence. Before we do, should we have a bombing campaign ready to take out the data centers? Will we be prepared to steal that technology? What can we do to make sure that we eliminate that technology or at least eliminate the threat of that technology and that might require us to go to war? So there's a powerful narrative at play here in this Capitol and perhaps elsewhere in this country. I don't know, what do we make of the strength of that narrative at the moment? Amba, you've already brought up that it seems not to be flying on Main Street, despite the fact that the data center industry runs ads that literally include imagery of Xi Jinping and Chinese fighter jets and things of that nature, kind of making the argument that we'll need to put this big data center up in your neighborhood to protect us from the Chinese.

What do we make of the strength of the narrative? Where can we imagine it and what its political impact will be over the next six months or a year?

Amba Kak:

Only because we've spent so much time and it also feels like this room is largely aligned on the more perverse uses of the narrative. I think it's worth noting that there are, and I'm sure I missed some of today's conversation, so apologies if I missed it. There are many versions of rightsizing the China threat I think is a really important conversation to have. I think Sam has been the reason that I've heard the best version of those conversations in the past few years. The discomfort though is that that feels like a completely different conversation from the one about what do we do about the very many extremely concerning dangerous manifestations that this industry is already having on our economy and on our society and just making sure that our attention isn't constantly and sort of deliberately turned to the China question as a fig leaf or a sort of proxy for protecting an industry over everything else.

So maybe that's my last note there. I do optimistically, and maybe I'm calling it too early, think that the narrative isn't aging well or certainly it is being pressure tested in this moment from some unlikely corridors, including from the deepest of the existential risk safety community. And I think that is all, to me, eventually good news. But at the end of the day, I don't think it is really about the specifics of the threat or that isn't how it plays out, but I'd love to hear from Jack and others.

Jack Shanahan:

I very, very viscerally react to the arms racing dynamics with AI. I don't like it. I don't like it whatsoever because it sets a very easy narrative of there's a finish line. These are weapons. How many weapons does China, how many weapons does Russia have? Once they get more, we need more. This is AI. It's a continuous competition under uncertainty. I've written about this for an ACF paper. I think we have framed this in very dangerous ways. Of course, there's a military national security component to AI. No doubt about it. It's going to do all sorts of different things. But by framing the whole conversation as a race, that means everybody immediately thinks there's a tape at the end. There's a finish line and somebody at a very high level, a cabinet member says, "We can't afford China to get there first." To get where first, what exactly are they getting to?

What's the difference? What are they achieving? I think it's just made the whole conversation warped. It's not helpful. I think we ought to reset that a little bit in saying, "Okay, this is a continuous competition. We understand there's urgency to it. There's no question about this is a geopolitical, technical, economic race dynamic that's happening, but it's not an arms race." I'll accept the racing piece of it because both countries are going very fast. We see something at the end of this rainbow, there's pot of gold. It has completely corrupted this conversation. And as much as I would like to agree, it's being pressure tested, I think in a five-sided building, it works very well at budget time to go up there and say, "If we lose this race, all is over." I don't like it. I'm a hawk through and through in many ways, but I think when it comes to AI and the kind of technology we're talking about, we strayed off the path in the last decade to suddenly think utopia and dystopia instead of a technology that was going to transform things over time, but just look at it like a very normal kind of technology, which we did, by the way, in Maven and JAIC.

I never got into work thinking, "Oh, the world is going to end tomorrow if we get this wrong." It's like, "This is going to be a 10-year process. This is what we're going to do with it." It was just a different flavor of conversation. I think we've corrupted the whole thing with the arms racing dynamic.

Kat Duffy:

And I would say that corruption started way before we got to AI. So I think one of the challenges that I really see with both the US-China narrative and then also with the AI dangers narrative is that they have both operated in worlds of extremes that have been put in front of decision makers and in front of citizens as being unquestionably true, when in fact they have often just been politically expedient. And so for years, and I think I said this here last year, I'm probably just repeating myself, but for years and years and years as dysfunction increased in Congress, the one thing that you could get people to come to bipartisan agreement on was China. So in DC for years and years and years, everybody could come up with a spin that was like this thing that really doesn't have anything to do with the US and China, China.

China will win if we don't do this. And it was simply because we had lost the ability to broker bipartisan agreement through any other way. And that is something that dates back decades. The problem is that now it is incredibly unclear to, I would argue, generations of policymakers, what is an actual threat, what is an actual concern where China is truly in an adversarial capacity to us and where on the other hand, it's just sort of smoke and mirrors. And I don't think that most of our decision makers actually truly know that because it was a narrative that was used so cheaply and for so long because it was the only narrative that worked. And by the same token, we had such a wave of AI security crisis language for so many years that also no one could figure out, well, is this real or is this the existential risk argument that is trying to get everybody to focus on where we're going to be in five years so they don't focus on where we are now, which is a very legitimate concern.

And so you have a perfect storm of both of these narratives then colliding with each other. And it is, in my opinion, sheer confusion. And that's part of the problem. We don't have the trustworthy actor we can all look to say, this is fire. This is smoke. Let's focus on fire. We simply don't have it. And this is where I get very concerned because I feel like there is a lot of crying wolf and also there may be all sorts of wolves that I'm like, I didn't know that was a wolf. So at least for me, that's my concern. But I also think it's important to own, especially for those of us who have lived in or of DC over the years, to own that part of the problem now is a problem that we created over decades because we couldn't fix our own stuff and we took the cheap path.

And I would add onto that then that the best possible way to bring people together and to centralize control is to tell them that they are them or that we are an us and there is a them. And you shouldn't come to have any division at all around the us. There's no space for division in the us because them.

I have seen that narrative used in so many countries. And I beg of folks that when you have your government telling you there is an us and there is a them, your first question, your first response to that should be, what's in it for you? Why are you saying that? What do you get out of there being an us versus them that doesn't have anything to do actually with us and what will benefit us the most? And that is the other thing that I wish Americans were keeping their eye on in a way that I don't feel that we have been trained to do in the way that so many of my friends and peers in other countries were trained to do to their peril over generations.

Miro Plueckebaum:

I don't really spend any time thinking about the question of whether or not we're in a race because regardless of whether or not we are, we'd still need to build tools to verify claims that we want to make to each other. Whether we're in a race or not, we'll still be in a low trust environment most likely. And so I think we should just get to work on building those tools. But as someone who moved to China in 2018 and lived and worked there for a few years with the goal of figuring out how we could perhaps bridge between China and the international community on questions of AI governance, I've never. For the longest time you could maybe speak to Chinese stakeholders about questions of user safety and content safety, but you'd get blank stares when you spoke about questions related to catastrophic risk. And that has really, really changed over the last 12 months and I've never felt as hopeful about the potential for coordination on some of these questions as I have recently.

Not only have senior members of Chinese leadership made statements that acknowledge some of these catastrophic risks like loss of control risk. We're also increasingly seeing this language reflected in Chinese pre-standards documents. So companies actually have to grapple with the question of how can they mitigate some of these risks? How can they build emergency shutdown mechanisms for AI agents? For example, there's a pre-standard recently released by an important standards body in China, TC260, that explicitly asks companies to think about how they can shut down an AI agent to mitigate Hugging Face-OpenAI style incidents. And what has made me most hopeful is that as Chinese companies are increasingly starting to internationalize, they have to think about how to comply with international rules and regulations, especially the EU AI Act. Unthinkable things are happening where Chinese companies are starting to reach out to international AI safety researchers asking for their help in complying with these standards and making their models more robustly safe.

So I think that's very exciting and there's lots of space now to think about how can we harmonize the way in which we think about managing some of these risks. So

Justin Hendrix:

There's good news

Miro Plueckebaum:

As well.

Justin Hendrix:

You got

Jack Shanahan:

Optimism.

Justin Hendrix:

There we go. On that point, actually, my final question to you all, which I'm going to put to you after our Q&A, I'd give you a bunch of headlines at the beginning that were kind of frightening headlines, things ripped out of the newspapers recently. I'd love for you all to imagine a headline you'd like to read in the next six months or a year that would indicate that things might be going in the right direction. I'll tell you that now, a headline you'd like to read so that you have a little bit of time to think it through in your head while we maybe entertain some questions from the audience. And in particular, listen, this is the last session of the day. Not everyone here was here for every single instant, but it is an opportunity to kind of somewhat synthesize some of the things we've heard.

We can't have a lot of speeches. We don't want those. No one likes that, not least of which are organizers. But if folks want to weave in other points or issues that they've heard during the day, that would be nice to hear. Do I already have a question from the room? I see a hand and I believe a microphone's coming to you. I will remind you, we are recording And if you can, please tell us who you are and where you hail from.

Kuantai Yeh:

Hello. Hi, my name is Kuantai Yeh. I'm a MIPP student here. I just started, but I spent the last 20 years in China. So my question is actually Chinese AI companies are heavily regulated in China. You know the models cannot give a wrong answer, right? And then also the Chinese AI companies, and there is a ministry in China that's governing what the AI companies can do in their answers. And also the Chinese AI companies, recently the ministry told the Chinese AI companies that you cannot offer any service to any apps that allows young people to spend more than two hours in talking to you and thinking that you're somebody real. And then so to me, the 2025 Stanford AI report basically says China has 80% positive attitude towards AI is because the population believe the government is doing the right thing. And now we're talking about here that we are worrying about China and we're worrying about this, therefore we cannot do more policies to govern their behavior, but China is doing all that and they're catching up.

So what is going on here? Thank you.

Justin Hendrix:

Okay. Who would like to take that one?

Jack Shanahan:

I guess I'll start. I think it's one of the things a lot of Americans I run into and talk about AI do not understand is what you just said about regulation on AI in China. They're leading the way in many ways. Now you could say, is it the right regulation? Is it doing all the right things, having the right effects? Matt Sheehan writes all the time about AI regulations in China. They're pretty significant. I think we actually could do more in the United States. This is the whole conversation about governance is where are the regulations that would sort of have the same intent of what we're seeing in China. On the actual capabilities of the systems themselves, I can't speak to that. All I would say is I will agree with you to the point I think you were making here is that this is no longer a conversation about one country here and another country here.

It's two countries that are essentially side by side. I think what China would say is they've sort of given up the battleground on training frontier models. That is the only place that I think you could ever say that China does not either have an advantage or is very close enough for the US to say, look, we're peer competitors here on AI. That's why I think there is no better time to have the conversation because there is so much peer to peer, not just competition, but capability that says if there are any two countries in the world that should be having this conversation, it should begin with the US and China. But I take your point on the regulatory side of it. I think there are things we could actually learn in conversations at either track two or track one level, track 1.5 that brings some of these out.

Well, what kind of regulations are we talking about? Are there things that we do in our free market capitalist way of doing things that would actually be more effective in China? I don't know. I'm just throwing those out. Those are the kind of conversations that we do have at the track two level, could also happen at the track one level. But the regulatory piece is very significant. And I think just most Americans assume, I hear it all the time, we're going to hold ourselves to some moral and ethical standard and China's not going to do that. They're just going to go, go, go. I don't believe that. I've never believed that. I think if a war begins, things are going to be different. We all understand that. But in peace time, I've actually seen quite a bit more on the regulatory side in China than I have seen anywhere close in the United States yet.

Kat Duffy:

I don't think there's anything surprising about the fact that China has exerted centralized control over the emergence of something faster than the United States exerted centralized control over it. The US will never domestically govern AI in the same way that China would and China will never domestically govern AI the way that the US would. That said, one of the things that I worry, and Jack, this is going to your point, one of the things that I worry about with this sort of toxicity of this binary of China bad and we don't want to be China and China is wrong is that it is really impeding the ability of policymakers, decision makers, scholars, interested individuals to look at the decisions that have been made and say, okay, well why did they make that? What does that do? What does that achieve? Where is there a specific type of tooling, for example, that might have been helpful on this?

And take some lessons learned from a country that has been working to govern this much more aggressively than we have and has therefore learned some things. And so this is where I worry that there are lessons that can be learned that we're not pulling because there has been a sort of a narrative that nothing that we might want to emulate or imitate or adapt could possibly come from this other nation or this other jurisdiction. And that I think is a real disservice to us going back to the initial point around the time crunch. We don't have time to waste trying to figure things out where someone else might have already figured out some or part of what would be worth working for us too.

Justin Hendrix:

Another question. I see one at the back. Yeah.

Kevin Klyman:

Hi, Kevin Klyman. I'm a fellow at the Kennedy School. Besides catastrophic risks, one of the risks that lots of the companies focus on is risks to elections and to democracy. We have an election coming up shortly. I'm wondering if you could talk about international AI governance approaches there. I think lots of people want cooperation on cyber, but there is an ongoing cyber war between the two countries, so maybe that's hard. And then also there has been some discussion about how the idea that AI could help citizens in China push back more against authoritarian governance might then lead the Chinese government to be interested in dialogue on that topic.

Justin Hendrix:

And by the way, Kevin's research worth everyone checking out as well. Who'd like to take that one?

Jack Shanahan:

I'll start again. Maybe not surprise me because of my military background. I worry less about killer drones than I worry about everything you just said. The information environment is being corrupted in ways with the most sophisticated AI and we haven't even seen how much better it's going to get. So it's not just a threat to democracy, it's a threat to governments everywhere. The whole idea of what's the relationship between the governed and the governors, it depends on an information environment where people have a certain amount of trust in what they're hearing coming out of official unofficial sources. I think this is one, by the way, in these track two dialogue, China has consistently raised this as a problem area of information operations is, hey, we're worried like you're worried about how this could. Now the effects in China are for a different reason, party stability, is the government going to be under threat from the population?

But this is something that they want to genuinely talk about. The question is what could be some areas of agreement on very sophisticated use of AI for information operations, disinformation? I mean all these things. To me, I think they're difficult to come to an agreement on, but what we're seeing is nobody believes anything anymore. That may be the most corrosive impact of these technologies at all. We don't know what to believe or not to believe. Any government in our world starts feeling their own government is at threat when that happens. How do you actually protect the citizens from that corrosive environment? How do you tell? Is it kindergarten training beginning K through six on what's real, what's not real? I don't know. These are conversations that are genuinely well received by different countries. We all face different outcomes from it, but the technology itself is becoming so unbelievably sophisticated.

How do we even tell anymore what's real and what's not? That to me threatens our democracy, but also threatens other forms of government.

Justin Hendrix:

This is a side conversation, but I've always worried that the answer is create an immutable version of reality by total and complete surveillance always. That will be the answer, right?

Jack Shanahan:

It has become the answer.

Justin Hendrix:

We have a ground truth and therefore we can compare back -

Jack Shanahan:

Whose truth? The AI's truth or the human's truth?

Justin Hendrix:

Good question. I think we have a question up here and the microphone will come to you.

Georgetown Graduate Student:

Hi, thank you. I'm a graduate student at Georgetown, but I'm always very impressed and grateful for the conversations happening here at SAIS and especially ACF. My question is when we think about the meeting between the two presidents later this month and the rumors of the bilateral dialogue on AI that's supposed to happen, having worked in diplomacy, I think I would be pleasantly surprised if they're even able to agree on a definition on AI, let alone AGI. So my question is if we want there to be collaboration on basic standards and norms around trust and safety and catastrophic outcomes, what kind of political message or narrative can both presidents bring back to their domestic audiences that makes sense for them to begin talking about collaborations in 2026 when tensions are so high?

Justin Hendrix:

I feel like this one's teed up for you, Kat. This is kind of the Nobel Prize question. What could the two leaders do in that room that would bring back something?

Kat Duffy:

I mean, I generally think the best thing to do is to start with where there has been traditionally longstanding collaboration, where there are existing norms, where there are existing mechanisms. So I, for example, it's not like the countries haven't cooperated over the years in guarding against bioterrorism, in guarding against global cybersecurity risks, in guarding against nuclear proliferation, in guarding against chemical warfare. Has it been done perfectly? Is it always done by both sides in equal and complete good faith? No, not necessarily. But that is an area where I think citizens and officials on both sides expect their governments to handle that. That isn't something the electorate can handle. We don't look to companies to say, okay, make sure to keep us all safe from a global pandemic.That is something where people look to governments to cooperate together. And so where I would start is by saying first, where do we have any sort of existing or sort of muscle that might have existed for cooperation where we could lean back in and cooperating?

And then from that, we will determine what in the AI mix can be brought into that existing structure mechanism framework. And so for me, I tend to believe you start in something as confusing and fast moving as AI, starting at where you have normative agreement, where you've already litigated the basics is better and you lean into the firmament and then you go from there.

Miro Plueckebaum:

I think not the disagreement, but the only thing to optimize for is that they meet again and talk about AI. Whatever gets you there. If it's mandating DNA synthesis screening to mitigate misuse in bio, do that. If it's cyber, do that. If it's something else, whatever gets them to meet again. Yeah.

Justin Hendrix:

Okay. Another question from the room.

Amba Kak:

Justin, while you're waiting on it - Yes, please. I have to catch a train, but I was thinking about your headline question.

Justin Hendrix:

Okay.

Amba Kak:

I’ll drop mine and leave. My headline I was thinking about, and I think maybe the top one is that “All of Our Candidates for 2028 Refuse AI Money.”

Justin Hendrix:

Wow. Okay. That would be an optimistic one. Yeah, indeed. Thank you. That's a mic drop moment. We'll let you do that. And we have a question back there in the room. Thank you, Amba. Yes. Just wait for the mic if you don't mind. Sorry.

Lisa Sun:

Yeah. My name is Lisa Sun. I'm a private investor. And coming to this conference, I had a lot of expectation. I spent four decades doing US-China work and some of the biggest companies have listed here on our US exchanges. I've been part of it. So my question to this panelist is one that's simply very important to us as investing public, because here in the US we have the most robust system of regulation.

If we were to design the perfect governance, would there be a role for investment companies, for listing companies, for the US stock exchanges, and for the whole community of investing as we think about AI, because it integrally is involved in shaping the economy that we see. So I would be very curious to understand what would be the role of our US stock exchanges and the regulators to really play a pivotal role in designing a more perfect model for us to evaluate risk, benefits, and also all of the other ancillary issues that we are addressing today because they are really profound for us as investors to think about, but often we don't have the tools.

Justin Hendrix:

Anybody here feel they can say what investors ought to be doing?

Kat Duffy:

Well, I don't think it's necessarily what investors ought to be doing so much as they are key stakeholders in this ecosystem. I spend a lot of time talking with corporate executives, with C-suites, with board representatives on AI and on corporate governance. And the uniform feeling at this moment is the function of corporate governance that we have is not at all fit for purpose in this moment. Everyone is flailing. And I think many people would look at the sort of AI boom that is happening right now. And it's really difficult to know if this is going to end up panning out really well or if this is instead essentially mass speculation and we're going to have a huge crash.

I think private sector governance, corporate governance is going to be a fundamental aspect of how America ends up shaping its own governance mechanisms. I also think that those are perspectives that should play a key role in foreign policy. So the program that I lead, we say that AI deployment is foreign policy. The problem is that people just don't view it that way and it has non-traditional stakeholders in it and it has non-traditional incentive structures. And so from my perspective at least, the role of business, the role of investors is critical. And also I think many of the suggestions that I would have, many people in the investor community would not love because it would also potentially require more transparency, more due diligence, more reporting among private capital flows as opposed to just public companies. So I think it could cut both ways as well, but I certainly take your point and think it's a mission critical group.

Jack Shanahan:

And what I'd add on this is I'm on the government advisory board of Insight Partners, one of the biggest venture capital firms in the United States. And I've watched this over the past few years where the money flows. The money was not flowing anywhere close to responsible AI, ESG, anything that would be associated with guardrails on AI. It's different now. You do have companies whose entire business model is on things like verification, is on test and evaluation. California, first state, just passed SB 813, I think it is, mandating IVOs, independent verification organizations. The money will start to flow. The capital will flow only based on incentives. The incentives are changing. They weren't there until recently. I think all the headlines that we started with are beginning to tell people there is money to be had in our wonderfully capitalist system, but they had to have an incentive.

I'd say the incentives exist now that did not exist just a year ago, nevermind two years ago when it was all about speed, speed, speed. Not to say there isn't still emphasis on ‘move fast and break things,’ the standard things you hear out of the valley more and more, but people are recognizing, well, to use this and to make money off it, what do we do to make sure it works correctly? That was my emphasis on test and evaluation in the Pentagon is, look, whatever else anybody tells me, I'm familiar with systems that fail and it ends bad for everybody. So there was an incentive for us to get it right because we wanted to use it operationally. I think as an investor, we're starting to see money flow to something it wasn't flowing to just based on real world events playing out. It's a little hard to say chicken and egg, what comes first here, but I think in this case, you're going to have to have a little bit more come in the form of legislation before companies start saying there's a market here for things like verification regimes, test and evaluation, red teaming.

There's some very successful red teaming companies out there that just didn't exist five years ago on AI.

Justin Hendrix:

I think we've come to the end of our time together here, but I'm going to give you all the last chance, a positive headline that you'd like to read, something that would indicate things have gone in the right direction, six months to a year. And I won't give you a character count. We'll have to keep them short. Miro.

Miro Plueckebaum:

“Trump and XI Agree on AI Summit.”

Kat Duffy:

“Chromebooks Banned Until Further Notice in American Schools.”

Jack Shanahan:

“Turns Out There's Only a 5% Chance the Human Race Will Be Extinct in the Next 10 Years.”

Justin Hendrix:

Well, if any of those headlines come true, you heard them here first. I thank my panel, I thank the organizers… Thank you very much.

Thanks to Samm Sacks and Jessica Chen Weiss at SAIS, and to Pete Peterson and Abby Moser at One District Media.