This discussion is locked.
You cannot post a reply to this discussion. If you have a question start a new discussion

How can you ensure that Artificial Intelligence and Functional Safety systems are safe, secure and reliable?

Artificial intelligence (AI) is increasingly being used in safety-critical product development, delivering benefits to users across a wide range of industry areas such as manufacturing, healthcare, transport, and the financial and wider digital sectors.

It’s obviously important to ensure that AI and Functional Safety systems are safe, secure and reliable for users and manufacturers.  People’s lives and company reputations are otherwise at risk.  However, safety-related guidance for senior managers is quite limited.  This naturally poses a risk when it comes to ensuring systems respond correctly to inputs.

The purpose of our flyer on this topic is to raise awareness of key factors and to provide summary information for these managers to support their decision making in this area.

The flyer focuses on 10 key ‘pillars’ that highlight AI-related considerations and techniques, and measures employed during the engineering lifecycle.  It also addresses the fundamental differences between traditional and AI software.

We aim to supplement this initial flyer with a more in-depth paper and further outputs in the coming months.  We’d welcome your comments and ideas, both on the paper and on ways to reach audiences that can benefit from our work.

Download our flyer here for free: Artificial intelligence and functional safety

Parents
  • We've received a few comments in the last couple of weeks, and some responses from members of the IET AI & Functional Safety working group, which I've detailed below.  Please add your further thoughts on this topic.  We'd be interested in hearing your views.

    Comment 1

    • Ross Anderson & co’s work shows that you can get a Deep Learning Neural Network to learn differently by permuting the training data;
    • D'Amour & co's work at Google shows how many real-world Machine Learning problems are underspecified, in that you can get different models from the same training data

     These observations are surely related. Taken together, they suggest that we really don't understand deep learning well enough to be able to figure out its safety properties.

    When people nowadays speak of "AI", including the IEC, they mean software which uses Machine Learning (ML) on training data to devise actions.  This of course leaves out many "classical" AI areas, such as theorem proving and reasoning, common-sense physics, as well as constraint satisfaction problems, which cannot be usefully addressed in this way. I notice the flyer also elides "AI" with ML.

     Did the WG refer to the drafts of the ISO/IEC TR 5469 (Functional Safety & AI Systems)?

     Response 1

    This field is contentious and there are a lot of opposing views.  I don’t think anyone is saying that we understand deep learning well - the purpose of the two pager is to highlight just that. One problem is that the technical bar for the application of AI is getting lower. Major automation suppliers will sell you all the hardware required to collect the data and implement a model, plus a tool kit to create the models. It’s not that difficult to build an AI application. We are trying to prompt awareness of the more fundamental issues that should be resolved before implementing AI. It’s a matter of asking “just because you can, doesn’t mean you should” and to make people aware of all the questions they should address.

    ISO/IEC TR 5469 is a completely different document to the 2-pager; it’s more technical and looks to provide solutions to the questions that our pillars ask. ISO/IEC TR 5469 doesn’t contradict the two pager. ISO/IEC TR 5469 is laid out differently and our pillars are distributed throughout the document. ISO/IEC TR 5469 doesn’t address ethics though. We could be stricter about our usage of the terms AI and ML. I appreciate that there are many flavours of AI all with unique challenges and attributes - some of which still reside mainly in the academic realm. Perhaps going forward we could include a section addressing the type of AI we are primarily addressing. I suspect it would be limited to ML and Artificial Neural Network software (ANN) as these seem to be the ones most quoted in proposed applications.

    The ISO/IEC TR5469 on AI and Functional Safety is out for comment/vote.

    Comment 2

    There is guidance you could refer to e.g. https://www.assuringautonomy.com, gives guidance on assurance of ML. Standards such as UL4600 (https://ul.org/UL4600) [safety principles and processes for evaluating fully autonomous products requiring no human driver supervision] ask lots of the right questions about autonomy, even if they don’t help answer them very much, and other work, e.g. IEEE P7000 series (https://ethicsinaction.ieee.org/p7000/) give good guidance on ethics and other related topics.

    Response 2

    The objective was just to produce a flyer; it wasn’t intended to provide a comprehensive overview. This comment makes a good point: there is a lot of guidance that “ask lots of the right questions about autonomy, even if they don’t help answer them very much”.  The same is true to some degree with ISO/IEC TR5469. It’s the big questions that I think need answering. There are lots of things that can be done to assure AI, but what are the appropriate measures and have they been applied with sufficient rigour?

    Comment 3

    Data is undoubtedly a key factor for developing ML models. It would be interesting to expose how to generate/collect this data to generate a formal dataset. The latter, aligned with the pillar 7 (Human Factors), would be some interesting points to explore and discuss in detail in the paper.

    Response 3

    At the working group we talked about the fact that data sets are not always well formed.  There are challenges and costs related to collecting data.  The human factor issues in collecting data is important.  Configured datasets v ad hoc sets were mentioned, as well as the forms of bias that can creep into datasets, some of which are related to human factors.

     Comment 4

    It would be interesting to outline how this type of solution could comply with the current safety standards (e.g. ISO26262). As the paper states, ML brings a level of non-determinism which opposes the safety standards requirements. It would be of the utmost interest to explore this challenge and identify it as possible "future work".

    Response 4

    This is a valid point and at the working group we talked about a number of factors in this area - addressing AI verification re IEC 61508; safety lifecycle applications relating to ISO 26262 and its divergence from 61508; using AI to replace human decision-making; the imposition of higher standards for AI v those required for humans, assessing the added value AI; the lack of a single set of benchmarks; opportunities to improve the safety of systems that are too complex for traditional safety processes; holistic systems based on AI; reliability levels; ‘black swan’ issues (Taleb); multiple diverse, less reliable systems v a single more reliable system; ARP4754 system issues. 

    Regarding ISO 26262 there are 2 groups developing requirements in ISO for the automotive industry, ISO/PAS 8800 and ISO/TS 5083, which they are aligning with ISO TR5469.  TR5469 decided not to quote the UL standard. 

Reply
  • We've received a few comments in the last couple of weeks, and some responses from members of the IET AI & Functional Safety working group, which I've detailed below.  Please add your further thoughts on this topic.  We'd be interested in hearing your views.

    Comment 1

    • Ross Anderson & co’s work shows that you can get a Deep Learning Neural Network to learn differently by permuting the training data;
    • D'Amour & co's work at Google shows how many real-world Machine Learning problems are underspecified, in that you can get different models from the same training data

     These observations are surely related. Taken together, they suggest that we really don't understand deep learning well enough to be able to figure out its safety properties.

    When people nowadays speak of "AI", including the IEC, they mean software which uses Machine Learning (ML) on training data to devise actions.  This of course leaves out many "classical" AI areas, such as theorem proving and reasoning, common-sense physics, as well as constraint satisfaction problems, which cannot be usefully addressed in this way. I notice the flyer also elides "AI" with ML.

     Did the WG refer to the drafts of the ISO/IEC TR 5469 (Functional Safety & AI Systems)?

     Response 1

    This field is contentious and there are a lot of opposing views.  I don’t think anyone is saying that we understand deep learning well - the purpose of the two pager is to highlight just that. One problem is that the technical bar for the application of AI is getting lower. Major automation suppliers will sell you all the hardware required to collect the data and implement a model, plus a tool kit to create the models. It’s not that difficult to build an AI application. We are trying to prompt awareness of the more fundamental issues that should be resolved before implementing AI. It’s a matter of asking “just because you can, doesn’t mean you should” and to make people aware of all the questions they should address.

    ISO/IEC TR 5469 is a completely different document to the 2-pager; it’s more technical and looks to provide solutions to the questions that our pillars ask. ISO/IEC TR 5469 doesn’t contradict the two pager. ISO/IEC TR 5469 is laid out differently and our pillars are distributed throughout the document. ISO/IEC TR 5469 doesn’t address ethics though. We could be stricter about our usage of the terms AI and ML. I appreciate that there are many flavours of AI all with unique challenges and attributes - some of which still reside mainly in the academic realm. Perhaps going forward we could include a section addressing the type of AI we are primarily addressing. I suspect it would be limited to ML and Artificial Neural Network software (ANN) as these seem to be the ones most quoted in proposed applications.

    The ISO/IEC TR5469 on AI and Functional Safety is out for comment/vote.

    Comment 2

    There is guidance you could refer to e.g. https://www.assuringautonomy.com, gives guidance on assurance of ML. Standards such as UL4600 (https://ul.org/UL4600) [safety principles and processes for evaluating fully autonomous products requiring no human driver supervision] ask lots of the right questions about autonomy, even if they don’t help answer them very much, and other work, e.g. IEEE P7000 series (https://ethicsinaction.ieee.org/p7000/) give good guidance on ethics and other related topics.

    Response 2

    The objective was just to produce a flyer; it wasn’t intended to provide a comprehensive overview. This comment makes a good point: there is a lot of guidance that “ask lots of the right questions about autonomy, even if they don’t help answer them very much”.  The same is true to some degree with ISO/IEC TR5469. It’s the big questions that I think need answering. There are lots of things that can be done to assure AI, but what are the appropriate measures and have they been applied with sufficient rigour?

    Comment 3

    Data is undoubtedly a key factor for developing ML models. It would be interesting to expose how to generate/collect this data to generate a formal dataset. The latter, aligned with the pillar 7 (Human Factors), would be some interesting points to explore and discuss in detail in the paper.

    Response 3

    At the working group we talked about the fact that data sets are not always well formed.  There are challenges and costs related to collecting data.  The human factor issues in collecting data is important.  Configured datasets v ad hoc sets were mentioned, as well as the forms of bias that can creep into datasets, some of which are related to human factors.

     Comment 4

    It would be interesting to outline how this type of solution could comply with the current safety standards (e.g. ISO26262). As the paper states, ML brings a level of non-determinism which opposes the safety standards requirements. It would be of the utmost interest to explore this challenge and identify it as possible "future work".

    Response 4

    This is a valid point and at the working group we talked about a number of factors in this area - addressing AI verification re IEC 61508; safety lifecycle applications relating to ISO 26262 and its divergence from 61508; using AI to replace human decision-making; the imposition of higher standards for AI v those required for humans, assessing the added value AI; the lack of a single set of benchmarks; opportunities to improve the safety of systems that are too complex for traditional safety processes; holistic systems based on AI; reliability levels; ‘black swan’ issues (Taleb); multiple diverse, less reliable systems v a single more reliable system; ARP4754 system issues. 

    Regarding ISO 26262 there are 2 groups developing requirements in ISO for the automotive industry, ISO/PAS 8800 and ISO/TS 5083, which they are aligning with ISO TR5469.  TR5469 decided not to quote the UL standard. 

Children
No Data