A Data Science Playbook for Social Justice

Audio of this conversation is available via your favorite podcast service.

How can you tell if an algorithm is any good? For University of Michigan assistant professor of information and public policy Ben Green, the answer lies less in accuracy scores, fairness metrics or any other technical measure than in what the system actually does once it’s out in the world. In his new book, Algorithmic Realism: Data Science Practices to Promote Social Justice (MIT Press), he offers data scientists a method for scoping, building and evaluating algorithms in the service of social justice.

Ben Green:

I am Ben Green. I am an assistant professor of information and public policy at the University of Michigan, and I'm the author of the book Algorithmic Realism: Data Science Practices to Promote Social Justice.

Justin Hendrix:

Ben, I'm looking forward to getting into this book with you today, and maybe just want to start where you start in the book, a little bit of a background around your career and how you got into data science and how you started out on this sort of intellectual journey. You talk about an experience in Memphis just before you set off to pursue your PhD. Maybe we can start there.

Ben Green:

So this book is very personal and biographical for me. It really echoes and reflects my own journey as a data scientist, thinking about the role of myself and other data scientists and algorithms in the world. So when I graduated college, I had studied math and physics and had been learning about data science and was very excited about applying those skills and methods to public policy problems, in particular, local government policy, urban planning type challenges. And so my first opportunity to do that was the summer right after graduation. I worked for the Data Science for Social Good Fellowship. It was at UChicago at the time, so I got to live in downtown Chicago, had a great time. And my team was working with the city of Memphis on a project to help them make sense of data about urban planning and urban revitalization. And I very quickly saw that the challenges of using data for good were not really technical ones in the sense of the types of things that data scientists get trained on.

The first issue that we confronted was just getting access to reasonable data in the first place. It was very difficult to actually get data on the quality of every parcel, every house in the city. That's what we needed to try to understand where the city should be investing money in revitalization efforts, which homes or properties should it be purchasing, rehabilitating, which ones should it demolish? What other actions should it take? So that was one of the big challenges that we confronted. Then after towards the end of the project and as we were doing handoff, we came up with other issues. We had created a web platform for the city to work with, and pretty quickly after the handoff, the system just stopped working. And the real issue was they didn't have the resources to actually maintain this type of system. They didn't have a huge tech team to take on some custom-built website and maintain that for usability for an extended period of time.

And then the biggest issue was just that they actually just didn't have the resources on a policy level to actually take the types of actions that the model that myself and my collaborators had built, which was meant to predict which properties are in need of investment, what actions should the city take for a set of properties within the city? And they just didn't have the funds to actually take these actions of rehabilitation, demolishment, revitalization generally. So this was just one short project, but it quickly woke me up to this gap between the way that technologists often talk about data improving society and the realities of doing this in practice where from a tech perspective, it often feels like, oh, we have all these skills, we can find social problems and develop algorithms to make the optimization or make the prediction or make the classification that provides the correct information.

But on the ground, it looked super different where there are all of these institutional, organizational, and political challenges that are far more difficult to navigate than just figuring out how to optimize a system.

Justin Hendrix:

And as I understand it from the book, that landed you on a journey to think about the practice of data science. And you talk about this idea that you ended up passionate, maybe slightly less about developing algorithms to improve society as studying the methods for how to develop algorithms to improve society. Is that right?

Ben Green:

I had a number of experiences once I started my PhD that echoed this example in Memphis where I worked on building a system and I saw, hey, this is actually so much harder in ways that I have never been trained for. The challenges here are about data sharing, are about politics, are about how is an institution like a police department or a city government going to use the information that they've been given? It felt to me like just focusing my career on trying to build algorithms for a bunch of local governments wasn't the best use of my time or was kind of coming up against all of these questions and challenges that I didn't know how to answer. So I stepped back and really shifted the focus of my research to be about trying to actually make sense of the real world impacts of algorithms and in particular the gap between algorithms that look good on paper and what the impacts of those systems are.

So as just one example of this, a lot of my dissertation focused on human use of algorithmic decision-making tools, looking at how an algorithm can seem like it provides... An algorithm can seem like it's a good system. It satisfies metrics for accuracy, for fairness, the types of things that data scientists tend to look for, but then doesn't work in practice because the intended users don't understand how to use the system. They don't actually want the information that it's providing. They misunderstand the information that it's providing. So that implementation angle was just one aspect of how the typical technical view of algorithms misses what's actually happening on the ground. So my research became really focused on trying to unpack these types of implementation challenges and aspects of how algorithms are actually affecting policy.

Justin Hendrix:

So you say this book is primarily written for data scientists, for researchers who study the impacts of algorithms, but you also hold out it's for legal scholars, policy professionals who are working on AI regulation in particular, who are thinking about how to evaluate algorithms. You say a central concept that the reader has to understand is this notion of algorithmic realism. So maybe give our listeners just the quick one-two on algorithmic realism.

Ben Green:

Yeah. Algorithmic realism is the methodology that I develop and propose in the book, and it is a way of thinking about and doing data science that centers around the real world impacts of algorithms. And algorithmic realism is in contrast to what I describe as algorithmic formalism, which is the typical way that data science is taught and practiced. So within algorithmic formalism, data science methods and evaluation revolves around the technical details. What data are you analyzing? What are the technical metrics of an algorithm? And that is typically how a system would be evaluated. Often there's a discussion of social impact within algorithmic formalism, but it's typically treated either off to the side or dealt with through mathematical formalisms. Algorithmic realism is an approach that guides data scientists in actually taking real world impact much more seriously. The core principle of algorithmic realism is this concept that I call the pragmatist maxim for data science.

And for interested folks who want to look at this later, a lot of this builds on pragmatist philosophy. But this pragmatist maxim for data science says that the ultimate test of an algorithm's attributes is the impacts it creates in practice. So if we want to know if an algorithm is well developed, if it's good for society, if it's equitable, we can't know that by looking at the technical details of the system. We need to actually look at what does this system do out in the real world? And I think this is the type of foundational idea that sounds kind of simple and banal and maybe obvious, and I think most data scientists would probably say like, "Oh yeah, I agree with that. I think I'm already doing that." But if you actually carry it through into the fundamental details of scoping and developing and implementing and evaluating algorithms, you actually can see some pretty transformational shifts.

Justin Hendrix:

You do spend a bit of time still grappling with the extent to which doing data science is doing politics or doing political engagement, but it seems like you feel like there's some education to be done on that front.

Ben Green:

So broadly, this book is an effort to help politicize data scientists and help them feel empowered to be political. One of the through lines of my work dating back at this point about a decade has been trying to help data scientists see themselves as political actors. And one of the biggest pushbacks that I get on that idea is I don't know how to do that. There are certainly data scientists who will say, "I'm neutral. I don't even want any part of this." And there's still a current of that certainly in the field, but a lot of data scientists are open to the idea, but then say, "I don't know what that means for me. You're telling me I'm a political actor. You're telling me I have all this influence. Also, people are telling me to not be too overconfident and to listen to other people. And I'm feeling caught between all of these tensions of the influence I have to make decisions, but also the fear that I'm asserting my privilege and my voice over impacted populations." So this book is trying to both help data scientists see themselves in that light as political actors, but then help them say, "Here's how you do this responsibly.

Here's how you can thread that needle between hubris and fear and act in a way that is oriented towards social justice while also checking yourself throughout the process to make sure that you're being grounded in the perspectives of impacted populations, people who have substantive expertise in the domain you're working in." And so a lot of the book for data scientists is laying out a step-by-step process for how to do this, because I really wanted to get beyond the general calls to action that I have made in my own work, and that is very common in a lot of the responsible AI and critical algorithm studies work that says to data scientists, "You need to be careful, you need to think about power, you need to think about social context," but then doesn't really give data scientists a ton of ideas on steps for how to do that.

And that's one of the places where I think this book can hopefully stand out and be really valuable to data scientists is to provide that sort of step-by-step guide with some theories, some examples, some directions for what it looks like to do this.

Justin Hendrix:

So after you get through the workflows, through the implementation, the scoping, the various mechanisms by which data scientists can go about their work, you stop on algorithmic fairness in particular and this kind of idea. I feel like we spend a lot of time on this podcast talking about issues around fairness, bias, certainly many of the harms people most fear about AI systems and the conversations that we're having right now about AI and the various harms it might produce revolve around questions around fairness and bias and the extent to which these systems may discriminate against people and certainly against marginalized groups. I don't know, where would you pick up there on this question? What does this algorithmic realism approach tell us about how to think about fairness or bias?

Ben Green:

One of the things that excites me about algorithmic realism is that it provides us with new ways of thinking about the challenges that the fields of data science and AI face. So I talked earlier about the sort of step-by-step guide for how to implement a project, and that's a big piece of it. But when you zoom out from that framework, what you get with algorithmic realism is also a new way to think about these challenges and a new way to approach data science projects. So within the book, I talk about that through the lens of algorithmic fairness and explanations, which are these two big challenges that data science has been confronting and trying to figure out. And the broad angle on fairness is that the field has kind of just gotten it wrong in a lot of ways, again, by repeating the mistake of algorithmic formalism.

So on one level, the fact that so many data scientists are aware of these issues of bias and discrimination is great. That's a huge win from where the field was 10 years ago, but they've done that and then they've tried to engage with those questions through the lens of formalism, which means that the primary way that fairness gets dealt with is through various fairness metrics, various ways of statistically testing an algorithm for bias. But these fairness metrics fall really short. They don't really tell us about the actual social impacts of the systems. They often become a sort of Band-Aid solution to systemically unjust algorithms that are used in bad ways. So my solution is not to say, "Hey, I have a new fairness metric or I have a new way to optimize algorithms for fairness." Instead, what I say is actually, if you care about injustice and you're trying to deal with an issue like racial injustice in the criminal justice system, the answer is not to create a statistically fair risk assessment.

There are actually a bunch of other things that you can do that take a more structural lens on this problem. So I talk about things like trying to support using data science to support programs that would more structurally alleviate the reasons why fairness is such a problem in the first place, which is about inequality in people's characteristics. There are significant disparities in the criminal records and social outcomes of people across race and class lines, which is what a lot of these algorithms are picking up on. But then also the impacts of algorithms, the way that an algorithm gets used in practice is often quite unjust. So you have a pretrial risk assessment where if someone is deemed to be high risk, the response in the vast majority of cases is to say, "Okay, great, let's lock that person up so that they don't commit another crime." That's not an inevitable thing that needs to happen.

So what I look for and describe are places where you could use data science to counteract some of these things. I talk about, for instance, community violence interruption programs where you have people embedded within marginalized communities that are working directly to try to prevent violence, particularly gunshot violence. And there are a lot of these programs that are underfunded, short-staffed, and really trying to figure out how can they best support the people in their community. There are a lot of ways that data scientists could come in and help them say, figure out which of our people in our program need the most support, which ones could actually benefit from some additional outreach? These are all things that could actually just reduce the crime risk of people within these communities, making them less susceptible to being deemed high risk in the first place. You can also use data science to help programs that are trying to take different outcomes or different interventions in response to these predictions.

So I point to one program in New York City called Supervised Release that tries to get more people out of pretrial detention and into a release outcome with more social support. Again, that program has a lot of challenges that data scientists could apply themselves to and help that program become more effective for more people, especially some of the highest risk people who are at the center of debates about this program. So broadly, what I would say here is the answer is to really dig in for data scientists and dig into these situations where it's like, "Hey, I'm concerned about racial disparities in pretrial outcomes." And it's like, great, you should be concerned about that. But rather than just jumping to the obvious decision point, take a more zoomed out perspective, try to understand where that problem comes from, and then look for local institutional actors or programs that are dealing with the structural challenge and then see how you can go help those people.

Justin Hendrix:

And presumably this is also where some of your thinking on algorithm in the loop, human in the loop, human algorithm collaboration as you refer to it, comes in. Why is that such a roadblock for people who are trying to benefit society by applying technology?

Ben Green:

This is one of the biggest challenges that has been driving me for many years and actually helped me do a lot of the thinking that pushed me into writing this book in the first place. So there's this common challenge of data scientists creating an algorithm or creating an AI system and saying, "Wow, this should really solve the problem because this provides really good information to make better decisions." But so often that doesn't happen because the intended users don't actually interact with that system the way that the data scientists assumed. And that's not to say that, oh, the users are doing something wrong. It's to say the data scientists didn't really think about the context of implementation. They often work on this very abstract vision of how a decision-making process actually occurs and then build a system based on that, and then the mismatch between those assumptions in reality means that the system isn't that useful.

And even today, we can see this with some of the latest generative AI tools where in theory, it's supposed to help with education, but actually it's reducing student learning, it's reducing lawyers are putting forward a bunch of briefs with made up citations, things like that because people are just relying on these systems. Now, the sort of common approach to deal with this challenge is this framework of explanations saying, "Well, sure, we won't just give the decision maker like a judge or a caseworker, we won't just give them an algorithm, we'll augment that system with an explanation." And so there's all of this work trying to build explanations. And again, it's very formal focused on the statistical details of what a good explanation should look like. But when you actually go and test that empirically, these explanations don't work. They do not improve people's ability to use the algorithm to make better decisions.

So again, this is a situation where algorithmic realism can say, "Okay, well, the issue is not that the explanations are badly developed. We're actually just kind of running down the wrong path. We're chasing the wrong rabbit hole." And the bigger task that algorithmic realism reframes us around is this idea you mentioned of algorithm-in-the-loop decision making, where what we really need to focus on is the idea that we are not developing systems that people are then supposed to oversee. We really need to center the human decision-making process that already exists. And then we are inserting the algorithm into the loop of that, and that's sort of this play on this common idea of human in the loop, which often just means a human overseeing a process. So for developing an algorithm, what we need to do is really dig into what is the workflow of decision makers?

What information do they actually care about? When do they need that information? What types of things are they looking for? And how can I design for that? And so some of the stories I talk about in the book, they're not saying, "Oh, we gave this person a better explanation." It's saying, "Actually, we just rethought what type of system we even need to provide that person in the first place."

Justin Hendrix:

So in the seventh chapter of this book, you get into artificial intelligence, generative AI in particular, start to ask some questions about how the technology that everybody's talking about at the moment may change the practice of data science, certainly change the sorts of approaches and questions that data scientists are asking. And you mentioned the pragmatists, you get onto John Dewey, talk a little bit about his idea of what intelligence is or maybe what intelligence is for. Maybe we could pick up there with a little Dewey.

Ben Green:

I talk a lot about Dewey in the book. He's definitely a big influence on this one, and he talks about intelligence in a way that is just completely foreign compared to how most people in AI talk about intelligence. So broadly, pragmatism is about making knowledge practical. The pragmatists were philosophers who were concerned about how their field was so concerned about dealing with what is the essential truth of the world, what is the foundational unchanging principle of justice? And Dewey and other pragmatists were saying, "This is just completely the wrong way to think about knowledge and philosophy and ideas. We should be thinking about knowledge as practical instruments for problem solving. And these ideas about essential truths are just a fallacy and are actually leading us in the wrong direction." And so he writes about intelligence as something that is instrumental. The purpose of intelligence is not to memorize information, but to actually guide action in service of a better future.

Intelligence has a purpose. It's not something that can be dictated or evaluated just by someone's performance on a test. And so it's very different than how we tend to talk about AI today where there's such an emphasis on these benchmarks. Again, we can see the formalism coming up, or the way that we tend to talk about AI is what is its performance on these benchmark tests? And we have all these metrics for it, but those benchmark tests often have very little to do with practical tasks that people are actually going to be carrying out in their day-to-day work, and even less to do with whether those tasks are socially beneficial. So what I talk about is we need to think about the intelligence side of AI, less about the intelligence of the machines and more about the intelligence of people and rethink our approach away from these ideas of superintelligence or trying to make an AI system that's maximally powerful, and instead think about what are ways that we can actually use this technology to augment people's performance, people's ability to make decisions, take better actions, integrate knowledge, and so on.

And so this builds very neatly off the algorithm-in-the-loop conversation where the idea is we need to build systems that are actually helping people learn and make decisions rather than thinking that we should just be building the most powerful possible tools and then assuming that somehow that will lead to benefit.

Justin Hendrix:

So that changes the job of the data scientist, I assume. You talk about this idea that data scientists may become more like creative strategists.

Ben Green:

Yeah. One of the big questions facing data science today is what does the future of this profession look like given that AI can now do a lot of coding? So I think of data science now as having two different big challenges that it's confronting. So the ethics challenge is one that it's been dealing with now for about a decade, and it's made some progress, but still it's a big question in the field of how do you make these tools responsibly? And nowadays that's become in some ways more pressing with all of these AI safety concerns. But then the other question for data science is not about the building of the tools, but about what it means for them as workers. And I think that algorithmic realism provides a way to deal with both of these things. So I've already talked quite a bit about how it provides a way to integrate ethics into practice more, but it also can help data scientists have a identity that's better suited for a world where a lot of the actual hands-on coding is done by an AI system.

Because a lot of what algorithmic realism provides is a way to think about connecting code with real world challenges. So the types of things that I'm pushing data scientists to do is not, here's a bunch of new technical methods, but to say, here's how you should understand social problems so that you're building a system that's actually worthwhile. Here's how you should evaluate your systems after you've built it so that you can see, has this been beneficial? Do I need to scrap it? Do I need to change it in some way? And all of these tasks are incredibly important for making algorithms useful in practice, but they also give data scientists a clear sense of what their role can be that goes beyond just code. And so I was saying earlier, I'm trying to politicize data scientists and more generally beyond the political side of this, I'm trying to give data scientists a wider sense of what their expertise is and what their professional identity can be.

So I come to this role, this idea of thinking of themselves as data strategists rather than data scientists, where it becomes less and less essential nowadays to be the person who's actually writing the code, but that doesn't mean that data scientists are useless. Actually, it becomes more important than ever to have someone who can help an organization or a team figure out what is the code that we need to write? How do we implement that code? How do we test that code? And again, not just test it in terms of technical details, but test whether it's been useful for us as an organization. So I think that the training and mindset of data scientists needs to change a lot to take on more of that connective tissue role that's less about writing every line of code and more about drawing the connections to organizational strategy and problems.

Justin Hendrix:

Is there a role for regulation and policy in pushing algorithm realism as a practice?

Ben Green:

Yeah, this is definitely not something that can come just on a voluntary basis. So I think of the shifts that are needed in data sciences AI along these two parallel complementary dimensions. One is methodological. You need data scientists to actually have a set of practices that they can employ to build algorithms that are more reliably good for society. That's kind of the core focus of the book, but you can't get there without institutional changes that incentivize those practices more and more because we've all seen how, particularly within the Big Tech companies, they're not trying to promote the public interest. Their problem is not that by and large, they're trying to do good for society and then just lack the methods. They're just operating on a fundamentally different wavelength in terms of trying to maximize profit, maximize shareholder value, market share, all of that. So there is a strong need for educational reform, for regulation that both changes the culture of data science, but also then can mandate government bodies and tech companies to actually take some of the best practices that I am talking about.

Justin Hendrix:

So listen, we've talked very generally here in the abstract. I'm interested in, as you think about the headlines, and I'm talking to you in the midst of this AI safety, AI risk kind of freak out that has extended for now a matter of weeks. When you look at the headlines, how do your ideas reflect back to you when you think about how AI is actually being implemented in the world?

Ben Green:

I think one of the big things that frustrates me about the conversation right now is that it still feels very one-dimensional in the sense that the conversation we're having is about the pace of AI development. So often there's this false binary that the industry will tell us of like, "Oh, well, if you want to regulate us or cut down on any of our practices, you just want to stop innovation." So that's a very extreme version of this frame. But today it's still, even in the conversations on pace, it's still operating on that framework. It's saying, "Okay, there is a predetermined path that we're going down, and the question is just how quickly do we get there?" And sure, slowing down and putting in some guardrails is good, but fundamentally what I'm interested in is laying out a different direction for how AI is developed and what types of AI systems we actually get.

And so slowing down the pace in many ways feels like a path toward legitimizing these companies and allowing them to do more self-regulation than actually getting us to an ecosystem of AI development that actually works for the vast majority of people. So I think there needs to be a lot more conversation about democratic control over AI systems. Pace is still very much an elite-driven process where the average person has still really no say in what that looks like. Meanwhile, on the ground, we're seeing lots of meaningful pushback, not just to the pace of AI, but to the direction that AI development is going. Data center protests, families blocking the implementation of AI in schools, workers pushing back against the implementation of AI in their workplace. These are not people concerned about the pace of AI, they're concerned about the fundamental direction. And so we need to open up pathways and ecosystems where more public interest oriented AI can be developed through the types of practices that I talk about in the book and that other aligned scholars have also put forward in their work.

And so I think that means there's not a sort of single silver bullet answer here, but certainly shifting more towards funding for academic research, particularly academic research in the vein of community-centered work, human-computer interaction, and often as well working on public sector developed systems, putting more resources into the public sector to build technology in-house that works for them rather than needing to rely on contracts with companies like Palantir and Deloitte and OpenAI to have their own systems in place.

Justin Hendrix:

Well, we will see whether maybe more of the world adopts a little bit more of the approach that you're recommending here in a book called "Algorithmic Realism: Data Science Practices to Promote Social Justice." Ben Green, thank you very much.

Ben Green:

Thank you. Always great to chat with you, Justin.