• Written by Muhammed Abdulla NC, CEO

A meeting to review progress on an AI project in research and development can quickly go off track when the team starts discussing the latest metrics. The project lead presents a chart showing a steady increase in model accuracy, but the discussion that follows reveals that the improvement is due to changes in the data preprocessing pipeline, rather than any actual progress in the AI algorithm itself. This is a common problem in AI research and development, where metrics can move for the wrong reasons, leading to misleading conclusions and poor decision-making.

Leading measures: failures found before validation, effort against stage

In research and development, it's essential to focus on leading measures that indicate progress towards the goal, rather than lagging measures that only show the outcome. One example of a leading measure is the number of failures found before validation, which indicates how well the team is testing and refining the AI model. Another example is effort against stage, which shows how much work is being done at each stage of the project, from data collection to model deployment. By tracking these leading measures, the team can identify potential issues early on and make adjustments to get the project back on track.

A shift supervisor at a manufacturing plant, for instance, can use AsscherAi to check the maintenance log and identify potential failures before they occur, allowing for proactive maintenance and reducing downtime. This is an example of a leading measure in action, where the focus is on preventing failures rather than just measuring the outcome.

Why the headline metric moves for unrelated reasons

Most people get wrong the idea that the headline metric, such as model accuracy, is always a reliable indicator of progress. However, this metric can move for unrelated reasons, such as changes in the data or the evaluation methodology. For example, if the data preprocessing pipeline is modified, the model accuracy may improve, but this does not necessarily mean that the AI algorithm itself has improved. It's essential to understand the underlying reasons for changes in the metric and not to jump to conclusions based on a single number.

This is where research and development teams need to be careful, as they can easily be misled by metrics that move for the wrong reasons. By digging deeper into the data and understanding the underlying causes of changes in the metric, teams can make more informed decisions and avoid wasting time and resources on unnecessary changes.

Baselining before rollout

Baselining is a critical step in measuring AI in research and development, as it provides a reference point for comparison. Before rolling out a new AI model or system, it's essential to establish a baseline for the current state of the system, including metrics such as accuracy, efficiency, and reliability. This baseline serves as a benchmark for evaluating the impact of changes and improvements made to the system. By establishing a clear baseline, teams can measure the effectiveness of their changes and make data-driven decisions.

A category manager, for instance, can use AsscherAi to compare the performance of two suppliers and establish a baseline for the current state of the supply chain. This baseline can then be used to evaluate the impact of changes, such as the introduction of a new supplier or a modification to the logistics process.

What to record when nothing was found

When nothing is found, it's essential to record the outcome, as this can provide valuable insights into the research and development process. For example, if a team conducts an experiment and finds no significant improvement in the AI model, this outcome should be recorded and analyzed to understand the reasons behind it. By recording and analyzing negative outcomes, teams can identify potential issues and refine their approach to achieve better results.

This is an area where many teams go wrong, as they often focus only on successful outcomes and ignore or downplay negative results. However, negative results can be just as valuable as positive ones, as they provide a learning opportunity and help teams to refine their approach.

What this does not do, or when it is the wrong choice

This approach to measuring AI in research and development is not a silver bullet, and there are situations where it may not be the best choice. For example, in situations where the goal is to develop a completely new AI system, rather than improving an existing one, a different approach may be needed. In such cases, the focus should be on exploring new ideas and approaches, rather than measuring progress against a specific metric. If you're unsure about the best approach for your project, contact us to discuss your options and determine the best course of action.

Ultimately, the key to successful AI research and development is to be flexible and adapt to changing circumstances. By understanding the strengths and limitations of different approaches and being willing to adjust course as needed, teams can achieve better outcomes and make more effective use of their resources.