• Written by Muhammed Abdulla NC, CEO

Many people object to using AI in operations because they believe it is impossible to measure its effectiveness, but this objection is misplaced because the right metrics can be found by looking at the operational processes that AI is meant to improve. The question is which metrics to use, and how to use them in a way that gives a clear picture of what is happening. A good place to start is with leading measures, which are metrics that indicate whether an operational process is on track to meet its goals.

Leading measures

Leading measures such as on-time delivery and exception rate are useful because they are directly related to the operational processes that AI is meant to improve. For example, a shift supervisor checks the maintenance log to see if all the scheduled maintenance has been completed on time, and if any exceptions have been reported. If the AI system is working effectively, the on-time delivery rate should increase and the exception rate should decrease. This is because the AI system is helping to identify potential problems before they occur, and to schedule maintenance at times when it will have the least impact on operations.

The use of leading measures also helps to avoid the problem of trying to measure the effectiveness of AI by looking at metrics that are not directly related to the operational processes it is meant to improve. This can lead to misleading results, because the metrics may be influenced by factors that have nothing to do with the AI system. For instance, a change in the weather may cause a spike in maintenance requests, which could make it seem like the AI system is not working effectively, even if it is.

Why the headline metric moves for unrelated reasons

Most people get wrong the idea that the headline metric, such as overall equipment effectiveness, is a good way to measure the effectiveness of AI in operations. They are wrong because this metric can be influenced by a wide range of factors that have nothing to do with the AI system. For example, a change in the market demand may cause a decrease in production, which could make it seem like the AI system is not working effectively, even if it is. This is why it is better to use leading measures, which are more directly related to the operational processes that the AI system is meant to improve.

A good example of this is when a category manager compares two suppliers based on their overall equipment effectiveness, without taking into account the fact that one supplier may be producing more complex products than the other. This can lead to misleading results, because the metric may not be directly comparable between the two suppliers.

Baselining before rollout

Baselining the current state of operations before rolling out an AI system is crucial to understanding its effectiveness. This involves collecting data on the current performance of the operational processes, including metrics such as on-time delivery and exception rate. By comparing the before and after data, it is possible to see whether the AI system is having a positive impact on operations. For more information on how to baseline your operations, you can visit our operations page.

Baselining also helps to identify areas where the AI system may not be having the desired impact, and to make adjustments as needed. For example, if the data shows that the AI system is not reducing the exception rate as expected, it may be necessary to adjust the system's parameters or to provide additional training to the operators.

What to record when nothing was found

When using AI in operations, it is not always possible to find problems or exceptions. In these cases, it is still important to record the results, even if nothing was found. This helps to build a complete picture of the operational processes, and to identify areas where the AI system may be working effectively. For instance, a maintenance team may use the AI system to check for potential problems with a piece of equipment, and if none are found, they can record the result and move on to the next task.

This approach also helps to avoid the problem of only recording data when something goes wrong. By recording all the results, including those where nothing was found, it is possible to get a more accurate picture of the effectiveness of the AI system. If you have any questions about how to implement this approach, you can contact us for more information.

What this does not do

This approach to measuring the effectiveness of AI in operations does not provide a complete solution to all the challenges of implementing AI. It is just one part of a larger process, and it should be used in conjunction with other metrics and approaches. For example, it may be necessary to use additional metrics, such as return on investment, to get a complete picture of the effectiveness of the AI system.

It is also important to note that this approach may not be suitable for all types of operations. For instance, in some cases, it may be more effective to use a different set of metrics, or to take a more holistic approach to measuring the effectiveness of AI. In these cases, it may be necessary to use a different approach, such as using a

• list of key performance indicators