• Written by Muhammed Abdulla NC, CEO

Can we trust that our AI system is working as expected in production, and how do we measure its effectiveness? This question is hard to answer because it requires a deep understanding of the underlying processes and metrics that drive production. The wrong metrics can lead to misleading conclusions, while the right ones can provide valuable insights into the system's performance.

Leading measures

Leading measures such as Overall Equipment Effectiveness (OEE) and unplanned downtime hours are crucial in evaluating the performance of AI in production. OEE measures the percentage of time that equipment is producing good quality products, while unplanned downtime hours measure the time lost due to unexpected equipment failures. These metrics provide a clear indication of the AI system's impact on production and can help identify areas for improvement. For instance, a shift supervisor checks the maintenance log to ensure that the AI system is correctly predicting equipment failures, allowing for proactive maintenance and reducing unplanned downtime hours.

A category manager compares two suppliers based on their OEE and unplanned downtime hours to determine which one is more reliable and efficient. By using these leading measures, the category manager can make informed decisions that optimize production and reduce costs. Most people get wrong that these metrics are only relevant for traditional manufacturing processes, but they are equally important for AI-driven production systems.

Why the headline metric moves for unrelated reasons

The headline metric, such as overall production output, can move for unrelated reasons, making it a poor indicator of the AI system's effectiveness. For example, an increase in production output may be due to changes in demand or raw material availability, rather than the AI system's performance. This is why it is essential to use a combination of metrics, including leading measures, to get a comprehensive understanding of the AI system's impact on production.

By using a single headline metric, organizations may be misled into thinking that their AI system is performing well, when in fact, it may be having a negligible or even negative impact on production. A more nuanced approach is required to accurately evaluate the AI system's performance and identify areas for improvement. This is particularly important when optimizing production processes to achieve maximum efficiency.

Baselining before rollout

Baselining the production process before rolling out the AI system is critical to establishing a benchmark for evaluation. This involves collecting data on key metrics, such as OEE and unplanned downtime hours, to establish a baseline performance level. By doing so, organizations can accurately measure the impact of the AI system on production and identify areas for improvement. A common mistake is to assume that the AI system will automatically improve production, without establishing a baseline for comparison.

Baselining also helps to identify potential issues with the production process that may be masked by the AI system's performance. For instance, if the production process is already operating at a high level of efficiency, the AI system may not have a significant impact on OEE or unplanned downtime hours. By establishing a baseline, organizations can set realistic expectations for the AI system's performance and make informed decisions about its deployment.

What to record when nothing was found

Even when the AI system does not identify any issues or opportunities for improvement, it is essential to record the results. This helps to build a comprehensive understanding of the production process and the AI system's performance over time. By recording negative results, organizations can refine their metrics and evaluation processes to ensure that they are accurately capturing the AI system's impact on production.

This approach also helps to identify potential biases in the AI system's decision-making process. For example, if the AI system consistently fails to identify issues that are later discovered through manual inspection, it may indicate a problem with the system's training data or algorithms. By recording and analyzing these results, organizations can improve the AI system's performance and ensure that it is providing accurate and reliable insights.

Limitations and exceptions

This approach to measuring AI in production is not a one-size-fits-all solution. There may be situations where the AI system is not suitable for production, or where alternative evaluation methods are more effective. For instance, in situations where the production process is highly variable or unpredictable, alternative metrics such as

• production cycle time
• defect rate

Ultimately, the key to successfully measuring AI in production is to use a combination of metrics and evaluation methods that are tailored to the specific production process and AI system. By taking a nuanced and informed approach, organizations can ensure that their AI system is providing accurate and reliable insights, and drive continuous improvement in production.