In the two previous instalments of our blog series, we looked at the promises of predictive maintenance and the challenges surrounding data handling. In this third and final part, we will delve into how to assess the success of a predictive maintenance system. We will explore the key metrics for evaluating models, discuss how to gain buy in from stakeholders, consider the legal aspects and highlight additional factors to consider. Links to part 1 and part 2 are here.

Measures of Success – Key Metrics

Assessing the effectiveness of a predictive maintenance model is more complex than going beyond the simple definition of accuracy. It is essential to look at and evaluate how well the models predict failure without overwhelming maintenance and engineering teams with false positives. The following metrics are crucial to consider, and it is up to the developer of the model to consider which is best suited to the specific use case.

Precision

Precision looks at measuring the ratio of true positive predictions (the actual failures) to all positive predictions (both true and false). In predictive maintenance, this metric is critical because a high number of false positives can quickly reduce trust. If engineers are sent to fix something unnecessarily, they will lose confidence in the system which can reduce the overall effectiveness of the efforts.

It is defined as:

Precision = TP / (TP + FP)

Recall

Recall looks at the models ability to pick up on actual failures. It calculates the ratio of true positive predictions to all actual failures (both detected and undetected by the model). Whilst precision minimises attendance to fix issues that don’t exist, recall ensures that legitimate failures are not missed.

It is defined as:

Recall = TP / (TP + FN)

Accuracy

Accuracy looks to represent the percentage of correct predictions out of all predictions made by the model. Its great to use as a broad measure but can be very misleading if the dataset is imbalanced as is likely the case as seen in part 2 of this blog series. For example, if the failures are rare, a predictive maintenance model that always predicts “no failure” will have a very high accuracy, but will fail to deliver any value for predictive maintenance.

It is defined as:

Accuracy = TP + TN / (TP + TN + FP + FN)

Assigning Relative Costs to Predictions

One of the most important but overlooked aspects of assessing predictive maintenance models is the cost associated with each type of prediction.

True Positives (TP): The asset is accurately predicted to fail allowing for proactive maintenance.

True Negatives (TN): The asset is accurately predicted to remain functional and no unnecessary interventions are required.

False Posititves (FP): The model incorrectly predicts a failure resulting in wasted maintenance resources.

False Negatives (FN): The model does not pick up on a failure leading to an unexpected breakdown and costly repairs.

In most cases, false negatives are far more costly than false positives, however false positives can also cause significant inefficiencies which can undermine the perceived reliability of predictive maintenance models. By assigning relative costs to each of the outcomes, the models can be tuned to meet provide the best cost outcome for your use case.

Buy In – Convincing Stakeholders to Act on Predictions

One of the biggest challenges in implementing predictive maintenance is gaining the buy in from internal and external stakeholders. Its extremely natural for decision makers to be sceptical about carrying out maintenance activities when everything looks fine and it requires a bit of a mindset shift. A few things can be done to overcome this challenge.

Data Transparency: Stakeholders are more likely to trust a system they can understand. By offering them visibility into the data that is driving the predictions and explaining how the model identifies failures can go a long way. If you have any historical success of predictive maintenance, showcase its success.

Focus on Cost-Benefit: Showing clear examples of how predictive maintenance reduces cost resonates well with decision makers, especially when accompanied by any financial projections.

Proactive Education: Equipping stakeholders with the knowledge about predictive technologies and their benefits will help them in understanding the basic principles behind predictive maintenance and they will be more likely to trust the model, take proactive actions even in the absence of any visible issues.

Legal Aspects

Predictive maintenance may involve some legal considerations, especially surrounding liability and any regulatory compliance.

Asset Warranties and Service Level Agreements: Predictive maintenance could potentially void warranties if activities are performed outside the prescribed schedules defined by manufacturers.

Accountability for Missed Predictions: If a model fails to predict a breakdown and causes financial losses or safety risks, questions of accountability will arise. Ensuring that contracts are in place that outlines who is responsible in such cases is key.

Additional Considerations

Continuous Improvement: The telemetry and operating conditions are always evolving. You need to ensure that the predictive maintenance model undergoes regular retraining and updating to remain accurate.

Human in the Loop Collaboration: No matter how sophisticated and accurate the models can become, humans should look at samples to validate the predictions when applying them in a practical context.

Explainability: This is a tricky one as once you go past basic models and into the realms of using neural networks, explainability becomes complex. Always try to build the simplest model that satisfies the outcome.

Summary

Assessing the success of any predictive maintenance implementation involves more than just evaluating the technical accuracy. By focussing on specific metrics and accounting for the business costs of different prediction outcomes, organisations can fine tune the modelling. Achieving stakeholder buy in and navigating legal obligations are additional crucial steps to fully realising and embedding the benefits the predictive maintenance has to offer. With the right approach and all steps taken, predictive maintenance can be an invaluable asset and investment for organisations, optimising maintenance activities and reducing downtime.