Long a staple of science fiction novels and movies, Artificial Intelligence (AI) is a burgeoning feature of the technology we use in our lives.  Everything from cars to cameras either use it already or will use it in the near future.

This factor can overshadow one of AI’s real strengths, mining large volumes of data for insights into large scale projects, programmes and problems.

Last summer Dr Alec Banks and Rob Ashmore had a short discussion on the key role AI will play in this area in safety and mission critical systems, along with some key anticipated challenges.

The event was to support a paper from the IET’s AI in Functional Safety Working Group:

 Alec: Everyone seems to be pushing to adopt AI. Why is it so desirable and perhaps even inevitable for programmes like the FCAS [Future Combat Air System]?

Rob: FCAS is a broad system, including airborne and ground-based elements. A significant amount of its effectiveness will come from its software. By making appropriate use of AI we can produce better systems and software, faster. Hence I'd suggest its use is indeed inevitable. I suspect it will initially be used in systems, where we need to gain insights from large volumes of data or we need to optimise over large problem spaces. And I think we'll use it across the entire system engineering lifecycle, including design-time activities.

Alec: That’s great but AI can produce unpredictable behaviour due to its complexity and sometimes ‘opaque’ design. We still have to provide current levels of technical assurance, so what are the critical challenges we face before adoption?

Rob: For me, it's helpful to group assurance challenges into two main areas. The first area relates to the overall development process, where things like data management, software frameworks and computing hardware are important. Here, we have a reasonable understanding of the key issues and these are reflected in a growing variety of standards and guidance material. The second area relates to the testing of the end product, which is often made more complex by the nature of AI. For example, neural networks typically have discontinuities, meaning there are many potential edge cases. Finding all of these cases is likely to be impossible, so we'll probably need robust system architectures; whether that be for design systems or operational functionality.

Alec: Again, going back to the technical complexity of AI, do you see the business, engineering and regulatory communities gearing to meet future capability needs in order to maintain competitive advantage?

Rob: Obviously, I don't see everything that's happening, but I think it's fair to say there's widespread awareness that things have to "gear up". In some places, we've made good progress: I'd cite the standards and guidance work you've been involved in as an example of that. In other places, simply because of where FCAS is as a programme, we've got some things to think about, for example: how to best use the terabytes of data each sortie could generate; and how to upskill engineers and operators to use AI in effective partnerships. But I think your phrase "competitive advantage" is absolutely key. The contested, information-rich environment in which FCAS will operate means we need to get the best performing software delivered and used routinely, at pace. I think AI has to be part of that, but it must be used in a secure, safe and responsible manner.

Dr Alec Banks FIET is Defence Science and Technology Laboratory Research Lead for the Assurance of Artificial Intelligence (AI) and Autonomous Systems.

Rob Ashmore FIMA, is the Future Combat Air System (FCAS) Chief Technologist – Software, at the UK’s Ministry of Defence.

AI is something the IET has been keeping track of in recent articles and events. For some tasters on the issues AI raises, you can catch Prof. Asaf Degani on the future of AI in autonomous and connected vehicles:

Here are Dr Rachel Craddock and researcher Kirsten McCormick on the capability (and weaknesses) of AI

Recently, the IET’s Nury Moreira looked more closely at the role of AI in caring for the ageing population