• Written by Muhammed Abdulla NC, CEO

What metrics should we use to measure the impact of AI on our supply chain, and how can we be sure they are moving for the right reasons? This is a question that has puzzled many supply chain managers, as they try to make sense of the complex interactions between different parts of their operation. The answer lies in choosing measures that are closely tied to the specific goals of the AI system, and in understanding how these measures can be influenced by factors outside of the system's control.

Leading measures

Two of the most useful metrics for measuring the impact of AI on supply chain are lead time variance and fill rate. Lead time variance measures the difference between the predicted and actual time it takes for goods to move through the supply chain, while fill rate measures the percentage of orders that are fulfilled on time. These metrics are useful because they are closely tied to the specific goals of the AI system, which is to improve the efficiency and reliability of the supply chain. By tracking these metrics, supply chain managers can get a sense of whether the AI system is having the desired impact.

A shift supervisor at a warehouse, for example, might use these metrics to evaluate the performance of an AI-powered forecasting system. By comparing the predicted lead times and fill rates to the actual values, the supervisor can determine whether the system is providing accurate predictions, and make adjustments as needed. This can help to improve the overall efficiency of the supply chain, and reduce the risk of stockouts or overstocking.

Why the headline metric moves for unrelated reasons

One of the challenges of measuring the impact of AI on supply chain is that the headline metrics can be influenced by factors outside of the system's control. For example, a change in demand or a disruption to the supply chain can cause the fill rate to fluctuate, even if the AI system is performing well. This can make it difficult to determine whether the AI system is having a positive impact, or whether the changes are due to other factors. Most people get this wrong, and assume that the headline metric is a reliable indicator of the AI system's performance, but this is not always the case.

In fact, many supply chain managers are wrong to assume that the headline metric is the only measure that matters. By focusing too closely on this one metric, they can overlook other important factors that are influencing the performance of the supply chain. A more nuanced approach is needed, one that takes into account the complex interactions between different parts of the supply chain, and the many factors that can influence the headline metric.

Baselining before rollout

One way to address this challenge is to establish a baseline for the supply chain's performance before rolling out the AI system. This can be done by tracking the lead time variance and fill rate over a period of time, to get a sense of the normal fluctuations in the supply chain. By comparing the performance of the AI system to this baseline, supply chain managers can get a sense of whether the system is having a positive impact, and whether the changes are due to the AI system or other factors. This is a crucial step, as it allows managers to evaluate the performance of the AI system in a meaningful way.

For example, a category manager might use historical data to establish a baseline for the supply chain's performance, and then compare this to the performance of the AI system after it has been rolled out. By doing so, the manager can determine whether the AI system is providing a significant improvement over the previous system, and make adjustments as needed.

What to record when nothing was found

Another important consideration is what to record when the AI system does not find any issues or opportunities for improvement. This can be a challenge, as it can be difficult to determine whether the system is working correctly or not. However, it is still important to record the results, even if they are negative. By doing so, supply chain managers can get a sense of whether the AI system is providing a comprehensive view of the supply chain, and whether there are any areas where the system needs to be improved.

For example, a logistics manager might record the results of an AI-powered analysis of the supply chain, even if the system does not find any issues. By doing so, the manager can determine whether the system is providing a complete and accurate view of the supply chain, and make adjustments as needed. This can help to improve the overall efficiency of the supply chain, and reduce the risk of stockouts or overstocking.

What this does not do

This approach to measuring the impact of AI on supply chain does not provide a complete solution to all of the challenges associated with supply chain management. There are many other factors that can influence the performance of the supply chain, and the AI system is just one tool that can be used to improve efficiency and reliability. However, by using the approach outlined above, supply chain managers can get a sense of whether the AI system is having a positive impact, and make adjustments as needed. For more information on how to implement this approach, or to discuss your specific needs, please contact us.