Ethics & Innovation: Inside Microsoft's AI for Good Lab

Susannah de Jager: Welcome to Ethics and Innovation by Oxford+, a special miniseries hosted by me, Susannah de Jager and sponsored by Equinox, Equitable Innovation Oxford. Throughout this ethics and innovation miniseries, we've explored many of the risks created by artificial intelligence. But also an extraordinary counterpoint, what becomes possible when we deliberately point these tools at problems society has struggled to solve? Juan M Lavista Ferres is Microsoft's Corporate Vice President and Chief Data Scientist and leads its AI for Good Lab.

Since Juan co-founded the lab in 2018, his team has worked on more than 350 projects around the world, applying AI and data science to problems spanning health, humanitarian response, sustainability, accessibility, and fundamental human rights. Their collaborators range from the United Nations and American Red Cross to universities, medical researchers, and the Oxford Institute of Technology and Justice. The origins of that work are revealing. Juan became involved in research into Sudden Infant Death, asking whether analysing data at scale, previously impossible, might reveal patterns conventional research had missed.

In this final episode of Ethics and Innovation, we ask Juan what happens when you give some of the world's most powerful analytical tools to scientists, lawyers, and humanitarian organisations tackling some of its hardest problems, and what responsibilities come with that power?

Juan, thank you so much for joining today. So before you founded the AI for Good Lab, you were already a data scientist, and you were looking at societal impact of big data. What made you switch from data to AI enabled data looking at societal problems?

Juan M Lavista Ferres: I actually started my career long time ago working for the InterAmerican Development Bank, and this was similar to the World Bank is an organisation that works helping the region in this particular case, Latin America and the Caribbean with development in general and I saw that the technology could be a great force for good and for a long time, I, even working at Microsoft, I would work on sometimes on the weekends to help some organisations around Seattle. One of my colleagues had lost a child to SIDS, since the Sudden Infant Death Syndrome is the main cause of death on children from between one month and one year old and through that exercise, it was very clear to me that we could use AI and machine learning to help a lot of these researchers learn more about their problems. And that's how basically I started these.

Later on I shared these with some of the leadership here at Microsoft. That basically help us create like, "Hey, can we actually do this as a full-time job?" It really was a dream come true. I'm very thankful to Microsoft for that opportunity and that basically started the AI for Good Lab.

Susannah de Jager: And just go into a bit of detail because you spoke there about Sudden Infant Death Syndrome, and most people will be familiar with it. But what you did and the volume of data that you took in and analysed was completely unprecedented and that's really the point here, is it's not something that could ever have even been approached without AI. And so in that particular case, you analysed 20 million births and 19,000 cases of sudden unexpected infant deaths and I just want to go into it because I found it fascinating. But give us a little bit of the findings from that.

Juan M Lavista Ferres: That was our first research that we need that was related to smoking. So it was very well known and established that maternal smoking would increase the chances of having a child die of SIDS, correct? That was well established.

What wasn't well established at that point were two things that we wanted to understand. First was that what was the impact of any single cigarette, correct? So it was the same smoking 10 cigarettes or 20 or one cigarette a day? There wasn't clarity on that. And not only that, the fact that we could understand that 10 cigarettes was more than five, and we actually, what we found from that research was that there was almost a perfect relationship, like every single cigarette count, and not only that, even one cigarette a day would double the amount of the chances of having a child die of SIDS.

It wasn't something they knew. It wasn't something they knew that even pre-pregnancy smoking in the last trimester before getting pregnant would also increase the chances of SIDS. In order to answer all these questions, you needed that amount of data that we had. It could have been approached, but what was difficult was the researchers that were working on this, these are amazing researchers that have dedicated their lives to help and understand SIDS. They didn't have the knowledge or capacity to deal with this amount of data. That was our contribution to the problem, was helping them ask questions of data, answer questions from the data. That is something that is critical is in order for us, and that's something that we repeat in every project we have, in order for us to have an impact, we always need to partner with an expert, right?

What is not recommended at all is to say, "Hey, I have the data about this particular problem, I can work with this problem without understanding the context, without understanding how the data was collected, without understanding the potential biases of the data." This is why it's so critical that for every project we have, we always partner with an expert in the problem. We're not experts in the problems that we work. We are working from pancreatic cancer to wildfires to detection of plastic in the ocean. We are not expert in those problems. Our expertise is AI and machine learning on computer vision. We partner with experts and we work on those problems together.

Susannah de Jager: So you're not just predicting something, you're actually changing the question that we can ask and the granularity. You already broadly brushed across some of the really kind of wild breadth that you are able to apply yourselves to when working with other experts. So what helps you narrow down what's an AI for Good problem and what you actually apply yourselves to?

Juan M Lavista Ferres: One of the great things about the space that we are is that for majority of the problems out there in the world, behind the scenes, you have data. And as long as there's data and you have the expertise, usually the methods that we use, the AI models that we use, could be used to help solve. And this is something that is particularly interesting because a lot of programmes for us look the same, correct? And I'm gonna give you an example.

So one of the projects that we have is that whenever there's a natural disaster, like the one that just happened in Colombia or in Venezuela, we partner with an organisation that have satellites. We put a satellite on top of a natural disaster, we take pictures, and then we use AI mode to build what is called damage assessment maps. These are maps that show every single house and which ones were affected or not. These are maps that are critical for people on the ground to help save lives.

When you look at that problem, and you look at the very different problem, I mentioned pancreatic cancer, that is basically trying to find early indications of pancreatic cancer from a CT scan, even though these two problems cannot be further apart, for us, those problems are basically the same problem. You have a big image, whether that's a satellite image or a CT scan that looks like an image to us, and what you need to do is find a pattern in that image, whether that's a house that was destroyed or something that is different in your pancreas. For us, from a machine learning perspective, from an AI perspective, these problems are basically the same problem and that happens in majority of the problems that we have.

A lot of these problems are very different problems from a societal perspective. When you work with some of these researchers, some of them which are using some of this technology already, they are not a, like at all in communication with the rest. Like the people that are working on satellite data problems are solving very similar problems are the ones that are solving a medical imaging on CT scans. But these two groups of people don't talk to each other and one of the beauty things for us is to try to come up with things that are solving the satellite data world and apply to the medical imaging world, because again, from a data perspective, they're the same.

So the beauty of what we have is the power to solve problems because these problems are very similar from a data perspective.

Susannah de Jager: So given that similarity and that you're not actually looking for a specific kind of data, given that there's such a broad applicability of the data, how are you narrowing down what you do focus on? Because it must be tyranny of choice.

Juan M Lavista Ferres: Yes, it's exactly the right. I haven't used that word before, but I'm gonna start using it from now on because I actually like it. We have a lot of problems on having a lot of options and part of that challenge is the focus, correct? So every Monday we get together with a significant portion of my team. We review the request. We get requests from multiple places in the world, from other colleagues, from people that knows of our work. Sometimes we do open calls, and we review every one of these requests. In the basis of first, they need to have an impact on society and impact is a bit different from places to places like helping save the giraffes in Tanzania is very different to helping understand a particular case of breast cancer, right?

So we try to look for a portfolio approach and then from these problems, we start asking questions. We go through a funnel. The first question is if we have the data, do we think that AI can solve this problem? And that's something that come from our experience. There's a lot of problems out there. Not all of them can be solved using AI or computer science. That's basically the first question we ask. The second question we ask is there data out there that was collected that could help us solve the problem? A lot of problems out there, I would say 70, 80% of the projects that could potentially be solved using AI usually don't have data. So our first problem is usually a lot of times is that there's no data.

Once you have problems that have data, the next question is do we think that we will be able to get access to the data, correct? So for very good reasons, privacy being one of them, there's a lot of projects out there where you could potentially solve it using AI, but it's not possible to solve it because there's no way for us to get access to data for all the reasons, correct? That's something that, again, we can go through a whole podcast about that particular problem. But there's very good reasons sometimes that, yes, the data exists, but it's not easy for us or it might be actually impossible for us.

For some projects, a lot of projects will require what is called label data, which means that you have, for example, in the case of like a wildfire, you have an expert that is, in this case, a firefighter that will look at a video of a particular fire and will say, "Here, what you're seeing here is the start of a fire." And that person will go to a tool and will actually, what is called label, basically indicate which of these pixels are from the beginning of fire. The same happened, for example, in the case of pancreatic cancer, like I mentioned before. You have an expert that is looking at the CT scan and is able to go and label that particular case is pancreatic cancer. This is what you call label data. A lot of problems in AI require label data.

Once we go through that, the other question we have that is likely one of the most important is, do we have the right expertise as a counterpart? Sometimes we get amazing projects, but these projects are coming not from the experts, are coming from people that basically, were affected by the programme. But in order for us to solve the problem, we need to be working with a counterpart that is an expert because we're not experts. We need an expert on the other side. So for example, I mentioned, again, pancreatic cancer, we are working with some of the best doctors, these are John Hopkins University doctors, that they have spent their lives trying to understand pancreatic cancer. That clearly for us is an amazing partnership.

So we always need to partner with an expert. Once we have that then the other question we have is, if we solve this problem do we believe that we have the right counterpart where we can do a knowledge transfer to them, and then they can actually work and we continue on this. That one is likely the last part of the funnel, and it's one that is usually the one that restricts more. So sometimes you have the right partnership, you have the right data, but you ask yourself, "If we can solve this problem, do we really believe that the organisation that we are partnering will make it to production? Will mean, that if we solve it, can they actually use these models to help?"

Once you go through the whole funnel, you could be starting with 100 projects, you will end up with two or three. From those two or three projects, we get together and we say, we try to address this from a portfolio approach in the sense that sometimes our projects are there that are extremely difficult, we already have solved these problems, but if we can solve it, then that will be a huge impact on society. You will have projects that maybe they're not that impactful, but we have solved those similar projects before where you know that you can solve it. So we try to look for a portfolio approach where you have very difficult projects, you have projects that you know you're gonna solve, you have also projects in different areas. I mentioned biodiversity, I mention healthcare, I mention disaster response. We also look at a portfolio approach from that.

It's a very interesting meeting. It's a very difficult meeting in the sense that you, like you said, you have a lot of options, you try to look for the best options to have an impact.

Susannah de Jager: And you're talking there about portfolio and the language sounds very close to investing and of course you are investing. You're investing your team's time and indeed other people's time. Out of interest, the intellectual property is it completely freely available once you've developed something?

Juan M Lavista Ferres: We partner with organisations and we don't take anything meaning that there's no IP for Microsoft. Everything we do, we open source everything. Ideally, we also open source the data when possible. Sometimes it's not possible to open source data, but we publish the results in academic journals. So our contribution to society is that.

I would say the big majority of the organisations out there have some philanthropic efforts within the organisation. What we realised was, if we could actually donate our time, like Microsoft still does philanthropic efforts outside our team, but we will say, look, can we carve part of that philanthropic effort that we have and dedicate it, instead of donating, for example, money, we would donate our efforts. The whole hypothesis, the whole premise here, is that we believe that we can have a bigger return of investment for societal perspective than making the donations to those organisations and the reason that is key is because we believe that even if we sometimes we could donate some of these organisations do not have the structure or capacity to hire the talent that is needed to solve this problem.

So that's the whole idea. We don't take a fee. We open source everything. So yes, everything that we do is contributing to society.

Susannah de Jager: I would say people definitely wouldn't be able to hire the kind of talent that you've aggregated in your teams. You'd have to be in an enormous organisation and even if you're within an academic setting with experts in, to your example, pancreatic cancer, the chances that you're able to couple them with a team of your experience and I'm sure breadth and depth, there's no way.

This Oxford+ miniseries is brought to you by Equinox, Equitable Innovation Oxford. Equinox is a major regional partnership established by the University of Oxford to drive inclusive innovation and sustainable economic growth. Bringing together universities, local authorities, government, industry, investors, and community organisations, Equinox provides a unified voice for Oxfordshire, championing innovation that delivers economic impact alongside social value.

More than 80 organisations have signed the Equinox charter, committing to attract investment, remove barriers to growth, and ensure the benefits of innovation are shared more widely. By aligning Oxfordshire's world-leading research, enterprise, and talent, Equinox aims to unlock nationally significant growth, reinforce the county's position as a global innovation hub, and ensure that success creates skilled jobs, attracts investment, and delivers lasting prosperity across the region.

There's one example that I have read about that I really enjoyed, and it doesn't fit within your framework. There was some data, but it was very hard to get, and you actually helped pioneer a way of collecting more data.

I'd love you to talk about that because, I'm being a little bit tongue in cheek, it does fit within your other frameworks. But SPARROW, as you named it with the acronym, looking at biodiversity, was very much focused on the data collection problem that you observed while working. So can you tell us a little bit about that?

Juan M Lavista Ferres: Yeah, so we work a lot in biodiversity where a lot of bioconservationists will work and collect data and then what we would help them were we would basically have AI algorithms that process the data. For example, if you have like a person that was studying spider monkeys in the Amazon, they would collect a lot of the data from spider monkeys, and we would help them classify the data. For example, they would have 10,000 pictures of things and then we would build an algorithm that could detect whether a picture had a spider monkey and would help them.

Three years ago, I was in a meeting and the person was explaining the hassle that they needed to go through to collect the data and I realised that we were certainly helping solve a very small part of the problems. When you hear the stories that these conservationists, they need to go to the middle of the Amazon, it would sometimes take them three, four days to put a camera trap or a bioacoustic recording devices in the middle of the Amazon, then come back. That, basically, they will lose sometimes a full week from the going to put this and they need to go back three weeks after that. They're doing the same trip that in many ways, sometimes it's very dangerous, and to go and change the batteries and get the memory cards. And once they got the memory card, they need to go through back, and we, if they were lucky, they had the data collected rightly.

From the time that they started the project and putting the camera to the time that they would analyse the data, sometimes that whole process was like, it could be a year, because of all the hassle. So it became evident for us that we were solving the wrong problem. We wanted to help conservationists in the world, we needed to help them collect data and there's roughly 200,000 conservationists in the world. All of them share one thing in common, to do their job, they need data. And in many ways conservation moves at the speed of that data.

That's the problem that we started with SPARROW. It's like, can we actually make it easier for them to collect data? SPARROW is a small device. It's a small computer that lives on the edge, meaning that they go to the middle of the Amazon and they instal this device. The device is solar power. It has a solar panel on top. It has a small GPU. GPU is what is used to run these AI algorithms. It acts in the middle of nature connecting to camera traps, connecting to bioacoustic devices. It process the information there on the edge, and it sends the result back using a satellite up link to the cloud. And once it's in the cloud, the conservation is can access real time that data. Once you install the device, you no longer need to go back. So we have some of these devices that right now been running for over a year in the middle of the Amazon, for example, and our plan is to continue making this as simple as possible for conservationists to collect data and to process data.

Susannah de Jager: And it brings me onto another example of your work that I've already had the pleasure of learning a little bit about, which is with the Oxford Institute of Technology and Justice. And when I was interviewing Phillipa Webb, she was talking about the apps that you're co-developing for Malawi, helping helping women and girls who are subject either to abuse or forced marriages. And I think that what struck me in that conversation was how many of the things that you were building in were very simple and most importantly, were informed by people on the ground. So the fact that you could build something incredibly complex, but if it doesn't work on 2G, it wouldn't work for that cohort.

Juan M Lavista Ferres: Yes, this is one of our best, most amazing projects we have collaboration with Phillipa and with Amal and with her teams have been amazing. Particularly the fact that we work, like you said, one of the key things that we have is that the co-development of working with people in Malawi.

So, the project that we have is Malawi is a country that has roughly 22 million people and have around 800 lawyers. To have a notion like a lot of the law firms out there in England or in US, we have significantly more lawyers than the whole country. And the challenge with only had 800 lawyers is that there's not enough lawyers of course to help majority of the people, particularly the people that are living in rural areas, particularly the people that will be very difficult for them to even afford a lawyer, impossible for them to afford a lawyer.

These are women and children that used to be suffering from either child marriage or domestic violence. They need help and a lot of times they're not informed, they don't have the right legal information. And we've been working on these, the collaboration is with the WLA team, this is the Women Lawyers Association in Malawi. This is a group of women lawyers that are doing an amazing job help these children, help these women. But of course, it doesn't scale. They're a very small group of lawyers. For us, what we want to do when we're working with them is to try to make these lawyers and make these process as productive as possible using AI, using large language models, to provide the people that they have, the volunteers that they have, with the best information. So a person from using a like you mentioned, even a 2G, WhatsApp conversation is able to communicate. Really we want it even in Chichewa. Right now the project still works mainly in English, but we are working to have this ability to speak in Chichewa, have the ability to ask questions, get legal information, and help those people on the ground, provide the best information to these women and children. It's an amazing project.

Like you mentioned, what if you let the group of engineers solve a problem, they might solve it completely wrong. For us, what is critical is to work with the expert. This is people like Phillipa and like Amal and like their team and work with the people that are working with the lawyers, and these are the people in Malawi, and work all of us together. It has been an amazing experience. Very excited, very happy about that project.

Susannah de Jager: It's amazing. You have already spoken, so it's a little bit unfair about a portfolio and that there's clearly so many amazing impacts. But is there one area where you are. I don't even want to say where it's your favourite area, but where you're more excited about the potential impacts of what your team is able to do?

Juan M Lavista Ferres: It's like asking which is your favourite child. I don't think I have a one favourite. The project that we have in Malawi is one that can scale extremely well, not only to Malawi, but this is a problem that also can be in the Philippines, in Bolivia, in Cameroon. It's a project that could potentially be working worldwide. I like projects that have that magnitude.

My background is the intersection between AI and healthcare. I always have an amazing appetite to help on that intersection between healthcare and health in general and AI. But it's one of the most challenging to solve what is called problems in production because you need to go through a lot of effort to make sure that eventually, for all good reasons, to make sure that the these solutions can be used in a production setting, correct?

But for example, there's a project that we've been working now for the last four or five years, a project that I started myself as an individual contributor in this project that is called Retinopathy of Prematurity. Retinopathy of Prematurity is the leading cause of blindness in children in a lot of countries around the world. This is a disease that didn't exist a few decades ago and now we're seeing a huge increase across countries. And interesting enough was one of my aha moments because I asked how is it both in that we have a problem now that didn't exist a few decades ago? And Retinopathy of Prematurity affects very small premature babies and these babies wouldn't survive before. So now more and more improvements in health and healthcare, more and more of these babies are surviving. But the challenge is that a lot of these babies are not ready to live on the planet yet. So the retinas are not fully developed and some of them will suffer from ROP. This is a Retinopathy of Prematurity.

If they are diagnosed and they are treated the blindness through these is completely preventable. The challenge is that they have a very small window of time, roughly between 24 to 36 hours, that if they are not treated during that time, that baby will be blind forever. But the challenge is that you only have in the world 200,000 ophthalmologists. These are not paediatric ophthalmologists. From these 200,000 roughly 10,000 are paediatric ophthalmologists, which is physically impossible. Like you don't have enough ophthalmologists to even diagnose these disease. Every one of these babies needs to be screened. It's physically impossible. So we have done AI models that can show that can diagnose this. We have them running on a smartphone and we are working with some of these amazing doctors that they know that they don't scale. They have dedicated lives to help make sure that there are no blind babies but there's no way for them to scale.

That is one of the projects that I feel most passionate about because I see that this is a problem that not only AI is a solution, but it's the only solution we have as a society. So we've been working a lot on the programme and we have hope that eventually we'll have that impact in society in production.

Susannah de Jager: So. In both of those examples that we've just discussed, and I'm by no means extrapolating that this is true of all of them, but it's very much that there's a sort of supply bottleneck that AI is able to solve for. And I can see that's got a huge amount of value in so many areas.

You've spoken a little bit about what perhaps doesn't work. You work with AI all the time and you've got a team who are pointing it in the right directions. When you look at these issues and its power, what worries you about the application of AI?

Juan M Lavista Ferres: Every project where we have we'll go through, what is called, a responsible AI framework. Microsoft was one of the first companies that actually have a Chief Responsible AI Officer. There are things that can go wrong. Not by my team, but other teams in the world were using AI to help diagnose skin cancer. One of the things that happened was that, a lot of the, the data that was collected for skin cancer was from Caucasians, because Caucasians in the US usually are the ones that go to dermatologist. There's more chances for them to have skin cancer to begin with. But the challenge was these models will work well in people that are Caucasians. They will not train on data from Latinos or for Asians or for African Americans, which means that these models might not even work for those places. Which means that if they don't work on the wrong side, they might tell you that either you have cancer where you don't, or worse, that you don't have cancer where you have, correct?

This is a case that the model has been trained on a subsample of the data that represents society which means that it only will work well in that subsample of data. Every project that we have go through a process to understand potential biases, potential areas where we should not be using the data. Also potential bad cases of technology in that you can build a technology to try to solve a problem, but what if people use the same technology to make it worse? And SPARROW is a great example of that. One of the only indications that we have, we don't want this technology to be used by people that are dedicated to hunting, right? This would be actually useful for them. So we have clearly terms of service on that describe that. But not only that, we only partner and we only allow organisations that are biodiversity organisations to work with us.

We also need to build these responsibly, make sure that we have the right questions, we partner with the right people, we understand potential biases, we understand the impact of potential biases. Every tool that we build could potentially be used as a weapon and I think it's important for us as a process to make sure that we maximise the use of this tool, but minimise the potential be using as a weapon.

Susannah de Jager: And that obviously speaks to, as one would hope, what your team is doing both inside a large corporate, but also given the principles around which your kind of entity was founded. Do you feel confident outside when you look at the evolution of AI more broadly, that other people are gonna have similar handbrakes in place?

Juan M Lavista Ferres: When you read the news, a lot of people are very concerned and sometimes they're concerned for the right reasons. I'm not questioning that. The reason I'm usually optimistic is that if you look for a random sampler, let's say a thousand people in the world, almost everybody will usually want to live in a better world. As humans, we want the best in others. We want the best in society. And of course, I'm not saying that everybody's like that, that you will find some bad cases. But in general, I would say majority of the people in society want to use these for right reasons and want to use these to help and that's why I'm usually optimistic.

Susannah de Jager: I'm pleased to hear you say that. I often say this to my son who's become very stranger danger aware. And I keep saying to him, I said, yes, my love, you must be careful. But most people are good people. Thank you, Juan. I've really enjoyed this and I'm so pleased that AI in your hands is looking to tackle so many amazing problems. I can't wait to see more of what you put out.

Juan M Lavista Ferres: Thank you. Thank you very much for the interview.

Susannah de Jager: Thank you for listening to this episode of Oxford+, hosted by me, Susannah de Jager If you want to keep up with all things Oxford+, visit our website, oxfordplus.co.uk or sign up for our newsletter on Substack. Oxford+ is a podcast produced by Story Ninety-Four.