Ethics & Innovation: Can Trust Keep Up With AI?

Susannah de Jager: Welcome to Ethics and Innovation by Oxford Plus, a special miniseries hosted by me, Susannah de Jager and sponsored by Equinox, Equitable Innovation Oxford.

Today I'm joined by Professor Anne Trefethen. Anne is an applied computer scientist specialising in high-performance scientific computing and has spent her career at the forefront of technological change. Most recently, she served as the pro-vice chancellor for people and digital at the University of Oxford, leading one of the university's largest programmes of digital transformation before stepping into a role as trustee of the Alan Turing Institute, the UK's National Institute for Artificial Intelligence and Data Science.

What makes Anne's perspective particularly valuable in both of these roles is that she has experienced firsthand what happens when technological progress outpaces institutional planning. As Oxford was developing its long-term digital strategy, the rapid emergence of generative AI fundamentally altered many of its assumptions, forcing a rethink of investments, priorities, and even the nature of digital transformation itself.

Today, we'll explore what leaders can learn from that experience, how AI is reshaping education, research, and society, and why questions of technological sovereignty and national capability are becoming increasingly important.

Anne thank you so much for joining today. For the purposes of this conversation, give us a little bit of background on your career.

Anne Trefethen: Yes, of course. I'm now a Professor of Scientific Computing at the University of Oxford, but I don't have a traditional academic career. I have worked in research on algorithms pretty much throughout my career, but in different forms. So sometimes I've been working with companies. Thinking Machines was one of the first high-performance computing companies that sold massively parallel machines, and I worked with them on parallel algorithms.

I worked with software companies in the UK, but I've also led research programmes, with what is now UKRI, but worked with EPSLC and the other research councils and found myself at Oxford setting up an interdisciplinary research centre, which is focused on enabling new research in different disciplines, but based on an innovative technologies. So I'm a geek at heart.

Susannah de Jager: Love it and you've held a number of roles at the university. So I'd love to hear a little bit more about that, please.

Anne Trefethen: I came to the university in 2005 to set up this interdisciplinary research centre, the Oxford E-Research Centre, and I was the director of that for almost seven years and then moved into the centre of the university to become the first university CIO. Where I led a large organisational change to create a single central IT department, the governance around that, to begin to put in place some of the cyber security that we needed, which was sorely lacking at that time.

And I did that, I guess I was a CIO for probably around five years and in the meantime, I became a Pro-vice Chancellor at that time responsible for the gardens, libraries and museums, which was an absolutely fabulous job, wonderful colleagues, and from there became the Pro-vice chancellor for people. That was everything to do with pay, pensions, and people in general. Again, a really fascinating job to have. And then the Pro-vice Chancellor for digital. So as I say, I'm a geek, so I was the right person, I think, to lead on that.

As we came out of the pandemic, it was clear that the university needed to invest in its digital capability in the infrastructure and everything around that. So the then Vice Chancellor asked if I would lead on that and so I took a slight shift back to my roots in a way to begin to take forward a, what we called, digital transformation.

Susannah de Jager: And you and I have spoken about that in the time period that plan was formulated, AI was a small consideration and part of it, absolutely, but that almost by the time that piece of work was finished, you realised you'd been overtaken by events around you.

Anne Trefethen: Absolutely. So we began work on it in 2022, and it took some time to work with colleagues across the organisation. I think it's fair to say people were exhausted after the pandemic. It wasn't a time to bring about lots more change. It was a time to reflect and understand what we needed to do to become more effective, and how we could use these technologies to support our mission in the university.

So we took our time on creating the strategy, and that was published in 2022 And of course, just about then, ChatGPT popped up. Now, of course, we were aware of AI in many forms. I mean, we're now very focused on large language models, but AI has been part of the technology shift for quite a long time.

So as you say, it was a part of our strategy. The strategy covered everything from our digital infrastructure and then had pillars sitting on it that focused on education, on research, and on administration, and then a layer of innovation across top. So we wanted to be looking out to the future. And really, at that time, AI was part of that innovation.

A year later, and it had become evident that we had to rethink how we were doing things. Was it really that we had to rethink what we wanted to do? But the old assumptions were gone. That wasn't just about those pillars around research and education, but simply everything. So we had plans for some large new administrative systems around finance and HR and it became clear as time progressed that the assumptions around what those would look like in a few years time were really no longer valid.

So we had to step back and begin to think about how do we approach this in a world with AI? What are the right investments to be making? And having invested in some of these areas, it does take a bit of courage to say, "No, let's stop. Let's rethink." I don't think anybody yet fully understands the impact that we're going to see and where we're going to see it and where we can best benefit from these tools and technologies versus where we need to be more careful, where we need to introduce more governance, and that's an area where, again, we've had to rethink.

When we set off with the strategy and we recognised we were going to be investing in AI. We tried to have a fairly light touch governance framework because the university is a place that has enough committees. We don't need more governance, more committees, was my thought. So let's make sure that the education committee can oversee what's happening in education, et cetera, et cetera. But the pace of change is such that we have had to introduce more governance. We've had to introduce new ways of thinking about governance and making these tools available and how we do that and what we do and how we have to listen to the voice of, essentially, as many of our community as we can.

I think the other thing to note is that the use of AI and the benefits of it are very different in different areas of the university. We're a microcosm of society, really. In some areas of research, it's massively impactful. In others, less so. In education, fabulously useful for students in some areas. Others where we're having to say, "That's not the right approach." But we're still figuring all that out.

Susannah de Jager: Yeah and it's a big ecosystem to be grappling with while this pace of change is going on around you. I've spoken to even rather small, quite simple businesses, who have said they raised for a two-year strategy, and they've already realised that the money they raised will last them five years, potentially, because they can use AI so expeditiously that it completely changes things and that's in a pretty straightforward business. So I can only imagine the complexity that you are grappling with.

Leading a big organisation through this changing period, how would you advise other people that are grappling with these issues? What are the kind of core frameworks that you think people need to be considering?

Anne Trefethen: Yeah, maybe I should say a little bit more about how we approach things at the university in terms of beginning to roll out technology to everybody. I think first of all, to say that we did do a number of pilots. So beginning back in 2023 we expanded those pilots enormously in 2024 and it was because at that point, it became clear, absolutely clear, that this technology was not going away. This technology was here to stay and part of what we needed as a university to both be using but teaching and engaging in the conversation where what do we want this to be in the future? I think it's really important.

This isn't a technology that's just being built in these frontier labs. It is something that I think we all need to be involved in the conversation about. How do we use it? How do we want to see it developed? How do we want it as part of the education? Back in 2023, we had those at the bleeding edge, such as myself, who are always first to try, and it was being used in the university in a ad hoc way. Research groups who had recognised the value had adopted it, not necessarily in a secure way because we didn't really understand at the time.

Susannah de Jager: And this is, I think, a really interesting thing. Just sorry to interrupt, but you've got an issue that the university doesn't have the luxury of waiting to put in a governance structure because events will overtake you both externally, but also people will just start adopting things. And so you're having to grapple with that reality that people are going to start using these tools and if you're not there with them, it just won't have any governance. So it's dealing with an imperfect situation.

Anne Trefethen: Absolutely and for some of my colleagues, this was really difficult.

Susannah de Jager: I can imagine.

Anne Trefethen: It was really difficult. Some, when we began to roll out availability to ChatGPT Edu for all of the students and all staff, we had some colleagues respond in a very visceral way because they felt this was not what an academic institution should be doing.

Susannah de Jager: Yeah. Well, it's iterative. It's much more startup mindset than it is traditional academic institution. But here, the two are meeting.

Anne Trefethen: Exactly. What we couldn't stop, we couldn't stop our students using these technologies. They were already doing it. We knew already that there were over 5,000 of the university had used their Oxford email address to create an account, a ChatGPT account. We knew that. So if there are over 5,000 that have done that, we know there are many more who are using it with their own email addresses, et cetera. So it was happening, but it was happening in, as I said, first of all, an unsecure way, but also not in an equitable way. Students who could afford to have licences or who could were getting very good tools and other students didn't have access to that same capability and for the university, that also became an issue of how do we ensure that there is equitable access or as best that we can make it.

For me, there was a key moment. It was in 2024, as I say, we'd done some pilots. We were talking to companies, Microsoft, OpenAI, and others, around what we might be able to do as a platform for the university. And I got invited to an OpenAI executive briefing. Now, I don't always go to those things because they're rather fluffy generally and it's not necessarily a good use of time. But on this occasion, I decided I would go along to see what they had to say. Just about a month before, we'd had Geoffrey Hinton and his lecture was a real eye-opener. So his lecture was around, will artificial intelligence become more than biological intelligence? And one of the startling things that he said was he had always thought that it would take a century or so to get to super intelligence and he now believed we would be there within a decade. He had a very downbeat vision of the future. This is a man who knows these things very well.

So I'd been mulling that over and then I went to the OpenAI executive briefing. They were essentially showing us the capabilities of what was to become GPT-4.0 that was going to be announced. I think it came out in May 2024. The capability beyond ChatGPT-3 was astonishing. It had multimodal capabilities, images, voice and I could see immediately that we were progressing at a pace that I had never seen before and the other thing that was very notable in talking to the engineers from OpenAI was that they themselves were excited and startled by the progress, the capabilities, of the models in such short time.

That really was an eye-opener for me. That evening, I was lucky enough to have dinner with Sam Altman after the event and I mentioned to him Jeffrey Hinton's view of a decade. And he said "No" he though it was five years. I did ask him "Well, what are we doing about this? Should we be concerned?" And his view was if I was asking him, was there an existential threat? He couldn't say, "Absolutely no. There isn't." But that they are releasing the models iteratively after testing and I can see that's happening. We've seen very recently requests slow down the development.

Susannah de Jager: And yesterday, as you and I were discussing, Sam Altman, as we record this, posted this article in the FT talking about exactly this.

Anne Trefethen: I think we have known for some time that we really need to have international collaboration on these. These go beyond boundaries. So it is interesting that he is now one of the voices that is that is now asking for us to do that. And it's interesting too that just a few days before Anthropic was suggesting that we slow down. Of course, it's some time ago now that there was a open letter signed by many hundreds of AI leaders to slow down. But the genie is out of the bottle. The capabilities are already out there. So whilst I can understand this desire now to let society catch up, I think we've got to do more than that.

We've got a lot of open models. There are models now that are simply distillation of these very large language models and what that means is that they have very much of the same capability, not all the same capability, very much the same capability, but able to run on much smaller systems. So we've really got to, I think, collectively be thinking about their use, how we use them, where we use them, and we've got to do the education, the AI literacy, to enable that at a very broad level. It's not just for the leaders. It really is, I think, a collective issue.

And that was why, I thought, I believed, the university had to grapple with this. We have to be part of that conversation. And without, I don't want to say experimentation with our students, it's not experimentation, but without those conversations about what should we use this for? How can we inform what we're doing? Is this going to help us in how we develop critical thinking or is it not? How do we remain resilient and able to take things forward in an age where we perhaps are becoming dependent upon AI? Where there's a lot of misinformation? So I think these are really societal issues that need to be grappled with at the university as much as anywhere else and perhaps more because we are creating the next generation of leaders.

Susannah de Jager: This Oxford Plus miniseries is brought to you by Equinox, Equitable Innovation Oxford. Equinox is a major regional partnership established by the University of Oxford to drive inclusive innovation and sustainable economic growth. Bringing together universities, local authorities, government, industry, investors, and community organisations, Equinox provides a unified voice for Oxfordshire, championing innovation that delivers economic impact alongside social value.

More than 80 organisations have signed the Equinox charter, committing to atract investment, remove barriers to growth, and ensure the benefits of innovation are shared more widely. By aligning Oxfordshire's world-leading research, enterprise, and talent, Equinox aims to unlock nationally significant growth, reinforce the county's position as a global innovation hub, and ensure that success creates skilled jobs, attracts investment, and delivers lasting prosperity across the region.

It brings us very naturally onto this next part of your journey in this space, which is that you're also a trustee of the Alan Turing Institute, and you sound wonderfully well prepared for the issues that they are facing at a national level because it's very similar.

You have the pace of company adoption of individual adoption. Indeed, I imagine in some government areas, adoption with governance and regulation effectively lagging that. There's this sort of push-pull between, we want to get it right, but it's going to have to be iterative and we're going to have to make, ultimately probably, some mistakes and be able to pivot because otherwise you're just even people in a vacuum.

I'd love to hear about some of the conversations and the work that you're contributing to at the  Alan Turing Institute.

Anne Trefethen: Yeah. I think one of the things that I would note is that no one Institute can do everything that's needed to be done and I think the Alan Turing Institute is therefore in a great position as our National AI Institute to be a convener. To be bringing together those developers from frontier labs, for those researchers in the universities who are looking at not only how these technologies are changing the way we approach science, but they're also looking at what comes next. What are the next AI models?

You may have seen just last week an announcement the government investing in two new AI labs looking at the future of models and the Turing have a role there to help understand where those might help us in the future. But the Turing has been around now since 2015, before this generation of AI and we're just at a pivotal moment where we are really focusing on national resilience. We have a new Chief Executive, George Williamson, who is brilliant and he just arrived in May full of ideas and vision and we're beginning to develop a new strategy for the Institution with the help of the researchers and staff at the institution, but also with our stakeholders.

I think it's interesting to think about what is resilience in this time of AI? It encompasses many elements. So we need to have a sovereign capability to be resilient. We can't, as we saw recently new models being made available and then being switched off to international. If we are only dependent upon models and capabilities being developed elsewhere, those could disappear. We need to be able to be resilient to that. That may come from creating our own capabilities. Not likely to be developing our own sovereign models, but possibly. AI is permeating everything, be it cybersecurity, be it defence, and we need to be able to protect our critical infrastructure where we can see that AI is being used by adversaries and we need to be able to defend our critical infrastructure.

But also, it's about ensuring that we have the human capacity. So training, ensuring that we are ensuring we've got enough people who understand these technologies and where they meet these very critical application areas. It's not just about the models. It's about how do they enable us in these critical areas. So the Turing is focusing now on taking forward that national resilience capability, working with our stakeholders, but reaching down into the models, being able to understand them, understanding what they bring and where we need to be aware that others will be using them and I know that George is very interested in misinformation and how can we build a capability that's helping the nation to deal with the fact that there is so much misinformation that is now also fueled by AI capabilities.

Susannah de Jager: I had a very interesting conversation for this miniseries with Philippa Webb from the Oxford Institute of Technology and Justice and she was talking about just the volume of these kind of national level attacks, and that I think she was saying 77% of them are originating in Russia, Iran, North Korea, and China. It's happening.

Anne Trefethen: Yeah. It absolutely is and you're right. I think in many elements of what we're trying to do in working with AI, and I would say that AI is now a collaborator and not necessarily a tool. There is always a tension and it's getting the balance of that tension right.

Susannah de Jager: Yeah. You were touching upon, the training of our own models not creating dependencies and having true resilience. There seems to be quite a defined kind of two schools of though in this domain from the conversations I've had. And one is that we are going to have to train our own foundational models in order to not have those dependencies. And the other is that they are going to become so commoditized that even if one gets switched off, you'll have another, and therefore you're not very dependent and you have some resilience just by as I say, they've become commoditized.

And that perhaps the UK with resources, but finite resources, should be focused more on the applications and our particular areas of specialisation and using that as our technological leverage in this space. Do you have a view?

Anne Trefethen: So I think in terms of the very large-scale language models that are being invested in now, that's not something we are likely to be able to. I'm not sure we've got the energy in the country to do it.

Susannah de Jager: Got to get First Light Fusion online before we can do that.

Anne Trefethen: Yeah. Oh, we're close. It's only 10 years away. In ideal world that's maybe what we'd like to do. You know, there are discussions across various countries about doing that collaboratively and maybe that will be approach that will be taken. But I think in some of the nationally important areas and I include education in that myself, although that's not an area that the Turing Institute is focused on. But in some of these areas, we need to have models that we can trust and to be able to trust them, we need to know what were they trained on? What are the data?

I mentioned earlier the possibility of using distillation or a distilled model. So a model that is based on these large language models, but it's distilled, and then we can use something called post-training to be able to focus on specific areas. And I think, for me, that's the most likely way we will make progress quickly in that it will require less power. But it will allow us to have models we can trust in specific areas. That may be used for national security, for instance. But I think, again, there is a balance to be made and as the technologies are moving forward they take less power than they used to. So we may find in a year's time that whole equation has shifted.

One thing I would mention is that being a mature person in this...

Susannah de Jager: I think ChatGPT would tell us seasoned.

Anne Trefethen: Seasoned, okay. Yeah, I've seen many seasons. It used to be that I would go to computer companies as an algorithm developer, as a software developer. I would go and visit them and they would show me a roadmap of, three, four, five years even. The chip design, the network architectures that they were working on. So they could predict pretty well what we would be seeing for sale in three, five years time. You go to these companies now, they're simply not able to do that. They're simply not able to really give you a roadmap beyond, three months I would say.

So it's very difficult in that environment to be able to say with any confidence at all of what approach one should be taking. I think we just have to do the best that we can and for me at the moment, it is building on distilled models to have a very focus. So it's kind of in between what you were saying. I think the other interesting thing that you can see that supports that trend is that if you look at what the hardware people are beginning to develop, it is the kind of desktop that is powerful to run your own models. That's going to lead to a commoditization, I think, as you suggested.

Susannah de Jager: So there's really two elements here. One is the sort of protective resilience side of it and then the other is the opportunity. And I think what's quite exciting is that the UK has lots of sovereign data sets and you and I were kindly introduced by Lionel Tarasanko, and he talks about, obviously NHS data, BBC data, the Met Office. And so there's quite a big opportunity here for the UK to turn some of those data sets into really valuable assets.

Anne Trefethen: Yeah, absolutely and that is where we can begin to develop models that, if you like, exhibit our values rather than another nation's values and those data sets are incredibly valuable to us. And I think are what we would use to be able to build our sovereign capability. My colleague at the Turing just published a really interesting blog on this on sovereign AI. How can we create resilience? And that's one of the things that he picks up too, that as we begin to develop models we trust, so they may have this base of a distilled language model that we take from elsewhere. We will see whether we start from scratch. But absolutely, those are valuable assets that will be able to give us models that we can trust in education, that we can trust for our national resilience, security, defence. We don't want to be building things that are absolutely critical that need to be able to help us take decisions quickly on essentially the ground that we don't understand or know.

Susannah de Jager: Yeah. And again, in another conversation for this miniseries, I spoke to Audrey Tang, who was the digital minister for Taiwan, and she was very much saying we can bound models to train on data sets and that serve the communities we want them to and the values we want them to.

Anne Trefethen: I agree. And we need to learn from that, I think. My colleagues at the university have set up AI Education Oxford University Hub. Which is bringing together they've got over a hundred different groups internationally that are from policymakers, from educators, from different institutions that are doing research, to try to learn from each other on approaches that we can take in understanding.

This is focused on education through schools and higher education. Because these tools are being rolled out everywhere and some approaches will win and some won't and we need to collectively understand that. And the only way we can do that is to have these international forums that we're beginning to share.

Susannah de Jager: Earlier you used the expression, the genie out of the bottle. Which makes us feel a little bit sort of the whole thing's cantering and we're tripping into it. Are you optimistic or more pessimistic, Anne?

Anne Trefethen: I am by nature an optimistic person. I have to say that whilst I was leading the move into bringing AI into the university, I did always tell my colleagues, " I have drunk the Kool-Aid." Just to let them know, I recognise in myself that I see a lot of good in this, but I do recognise that it isn't all good.

So I am optimistic. I think these articles by the likes of Sam Altman and others. There is a voice now saying let's work together and understand these. We recognise there are dangers. But, when you look at the science advances that we will be able to make using these technologies the capabilities that we will have, it's difficult not to be optimistic. But how about you?

Susannah de Jager: Ooh, interesting. I'm a layperson, so I can't profess to have the same amount of knowledge in the domain that you have. But in these series of conversations that I have been having, I have come away reassured that the discourse we are often fed in newspapers, which is quite panicked. It's happening to us and this is going to end the world and we're all going to lose our jobs. I have felt that people that are closer to it, both in practise and policy and the technological side and within institutions that will set regulation, I've come away thinking, no, there are things we can do. It can be constrained and that actually those listening who are perhaps more in my camp need to be adding our voice to pressure to make sure that is the case because it's not a good enough to just say the horse has bolted. There are things we can do and should do. But that makes me optimistic broadly, I would say.

Anne Trefethen: Good. Yeah. When you look at it, every generation has had a technological shift. Perhaps not at this pace. One of the startling things to remind ourselves is that when ChatGPT was made available in 2022, within five days, it had a million users. Within two months, it had 100 million users. There is nothing that has moved at that pace. But the point you just made, this isn't happening to us. We can be part of, we need to engage with, and it isn't just for those people in the labs. These types of conversations, I think, make a difference.

Susannah de Jager: And so where do you hope we are in 10 years time?

Anne Trefethen: Ooh. Gosh. Could we say two years time?

Susannah de Jager: Fine, scrap that. Where do you hope we will be in two years time, Anne?

Anne Trefethen: I hope that we are happily having a similar conversation having learned a great deal more, having an evidence base. As a scientist, I want an evidence base. I hope that it is showing that we have learned how to work with AI in a positive, constructive way. That we have a next generation of students who are not afraid of the future, who are curious about the future, who are engaging with the future, but are not afraid.

At the moment, I think we are creating fear by saying all the jobs are going, when actually the jobs will be changing. So I hope in two years time, we'll be in a position where we'll have more of an understanding of that impact and one that allows the younger generations to be less fearful and to be able to know how to adapt to the future that we're going to have.

Susannah de Jager: And I think that that resonates so much. You know, I touched earlier upon companies saying they're being able to expedite their plans, potentially hire fewer people. But the part that I didn't say is that what they're realising is that it expedites their plans so much that they're not hiring the people they thought they might. But they do need to hire other people to catch the kind of amplification of whatever their product or their plans or their go-to-market is. I completely agree with you that we're seeing the first tranche and even now, I'm already hearing about the second, we don't yet know what that will quite look like. But there are going to be jobs created by the speed and I, like you, I'm very hopeful that we're just going to have more information on how to train our children to be those people.

Anne Trefethen: Yes. As you think about it, so a lot of the entry-level jobs are the ones that are going and we both know that when you go into an entry-level job, you're not just there learning the skills for that job, you're learning the skills of how to work within an organisation. You're learning the skills of how to interact with other people in a professional way. You're learning the skills of how to manage time, resources. One of the concerns I have is without those entry-level jobs our students, as they leave, are not prepared for that next level. So we need to think about how do we help them prepare so that these new jobs that are coming up, they have the skills already to be able to take into the organisations. I think it also is going to impact the education we give to the students as they prepare to go out into the AI-enabled world.

Susannah de Jager: It's fascinating. Anne thank you very much for this conversation. I've really enjoyed it.

Anne Trefethen: Thank you.

Susannah de Jager: Thank you for listening to this episode of Oxford Plus, hosted by me, Susannah de Jager If you wanna keep up with all things Oxford Plus, visit our website, oxfordplus.co.uk or sign up for our newsletter on Substack.

Oxford Plus is a podcast produced by Story 94.