• Written by Azhar Aliyar, Director

How do we know if our new AI system is really improving our business, or if it's just a flashy new tool that sounds good in meetings? This question haunts every executive who has ever invested in a new technology, and it's a hard one to answer, especially in the first thirty days of implementation.

Choosing a first case with a known answer

To get started with measuring the ROI of our AI system, we need to choose a first case that already has a known answer to compare against. This means selecting a business problem that we have already solved, or at least have a good understanding of, so we can compare the AI system's results to our existing solutions. For example, a category manager might compare the AI system's recommendations for supplier selection to their own manual process, using the same data from their enterprise resource planning system.

This approach allows us to validate the AI system's results and build trust in its capabilities, which is essential for getting stakeholders on board with the new technology. It also gives us a baseline to measure against, so we can see if the AI system is really improving our business outcomes.

Leading measures you can actually observe

Once we have chosen our first case, we need to identify the leading measures that will indicate whether the AI system is having a positive impact. These are the metrics that we can actually observe and track, such as the number of queries answered correctly, or the time it takes to resolve a customer complaint. A shift supervisor, for instance, might check the maintenance log to see if the AI system's recommendations are reducing downtime.

Leading measures are important because they give us early warning signs of whether the AI system is working as intended. If we wait too long to measure the outcome, it may be too late to make adjustments and get the project back on track. By tracking leading measures, we can make adjustments and improvements in real-time, which increases our chances of success.

Why aggregate outcome metrics are unreliable

One of the biggest mistakes people make when measuring the ROI of AI is relying on aggregate outcome metrics, such as overall sales or customer satisfaction. These metrics are unreliable because they can be influenced by a wide range of factors, many of which have nothing to do with the AI system. For example, a change in the market or a new competitor could affect sales, even if the AI system is working perfectly.

Most people get this wrong, assuming that if sales are up, the AI system must be working. But this is a flawed assumption, and it can lead to false conclusions and poor decision-making. By focusing on leading measures and comparing results to a known baseline, we can get a much more accurate picture of the AI system's impact.

What to stop doing if the comparison fails

If our comparison of the AI system's results to our existing solutions fails to show a positive impact, we need to stop doing whatever is not working and try a different approach. This might mean adjusting the AI system's parameters, changing the way we are using the system, or even abandoning the project altogether. It's better to cut our losses and move on than to continue investing in a project that is not delivering results.

We should also be willing to question our assumptions and challenge our own biases. Maybe we misunderstood the problem we were trying to solve, or maybe we chose the wrong metrics to measure success. By being honest with ourselves and willing to adapt, we can increase our chances of success and get the most out of our AI investment. For more information on how to get started with AI, visit our website.

When to seek help

If we are struggling to get started with measuring the ROI of our AI system, or if we are unsure about how to interpret the results, we should seek help from experts who have experience with AI implementation. They can provide guidance and support to help us get the most out of our investment and achieve our business goals. We can contact us to learn more about our AI implementation services.

What this does not do

This approach to measuring ROI and getting started with AI does not provide a magic bullet or a guaranteed solution. It's a practical, step-by-step approach that requires effort and dedication to implement. It also does not replace the need for human judgment and critical thinking. We still need to use our own expertise and experience to interpret the results and make decisions about how to move forward.

By being clear about what this approach can and cannot do, we can set realistic expectations and avoid disappointment. We can also focus on the things that really matter, such as building trust in the AI system, tracking leading measures, and making data-driven decisions. With the right approach and mindset, we can unlock the full potential of AI and achieve real business results.