• Written by Azhar Aliyar, Director

A finance director pores over the latest quarterly reports, searching for evidence that the new AI-powered analytics tool has delivered a tangible return on investment, but the data is stubbornly inconclusive, prompting a re-examination of the measurement strategy.

Leading measures you can actually observe

When evaluating the effectiveness of an AI intervention, it is tempting to focus on lagging indicators such as revenue growth or cost savings, but these outcomes are often the result of a complex interplay of factors, making it difficult to isolate the impact of the AI tool. Instead, leading measures such as user adoption rates, data quality, and process cycle times can provide earlier and more actionable insights into the intervention's progress.

For instance, a shift supervisor checks the maintenance log and notices that the mean time to repair has decreased significantly since the introduction of the AI-powered predictive maintenance tool, suggesting that the intervention is having a positive impact on equipment uptime and overall productivity.

Why aggregate outcome metrics move for unrelated reasons

Most people get wrong the idea that aggregate outcome metrics such as revenue or profitability are reliable indicators of an AI intervention's success, when in fact these metrics can be influenced by a multitude of unrelated factors, including market trends, seasonal fluctuations, and external events. As a result, changes in these metrics may not accurately reflect the impact of the AI tool.

A category manager compares two suppliers and finds that the one using the AI-powered demand forecasting tool has experienced a significant increase in sales, but further analysis reveals that this is largely due to a successful marketing campaign rather than the AI tool itself, highlighting the need to carefully consider alternative explanations for observed changes.

Baselining before rollout rather than reconstructing it after

Establishing a baseline prior to the rollout of an AI intervention is crucial for meaningful measurement, as it provides a clear point of comparison for subsequent evaluations. Attempting to reconstruct a baseline after the fact is often problematic, as it relies on incomplete or inaccurate data, and may not accurately capture the pre-intervention state.

A project manager works with the IT department to establish a comprehensive baseline of current processes and metrics before introducing the AI-powered automation tool, ensuring that any subsequent changes can be accurately attributed to the intervention.

Recording the investigations that found nothing

It is equally important to document and learn from investigations that fail to find a significant impact of the AI intervention, as these can provide valuable insights into the limitations and potential biases of the measurement approach. By recording these "null results", organisations can refine their measurement strategies and avoid repeating similar mistakes in the future.

For more information on how to implement a robust measurement strategy, visit our resource centre, which offers a range of guides, case studies, and best practices for evaluating AI interventions.

What this does not do

This approach to measuring AI ROI is not a substitute for careful planning and stakeholder engagement, and should be used in conjunction with other evaluation methods to provide a comprehensive understanding of the intervention's impact. Additionally, it may not be suitable for all types of AI interventions, particularly those with highly uncertain or intangible outcomes.

Organisations should carefully consider their specific needs and context before adopting this approach, and may wish to consult with experts or conduct further research to determine the best course of action. To discuss your specific requirements and receive tailored guidance, contact us at our office.

Common pitfalls and challenges

One of the most significant challenges in measuring AI ROI is the risk of over-attributing outcomes to the AI intervention, when in fact they may be the result of other factors. To avoid this pitfall, organisations should strive to establish clear causal links between the intervention and the observed outcomes, using techniques such as regression analysis or propensity scoring.

A common mistake is to rely solely on aggregate metrics, without considering the potential biases and limitations of these measures. By using a combination of leading and lagging indicators, and carefully controlling for alternative explanations, organisations can develop a more nuanced and accurate understanding of their AI interventions' impact.

Future directions and emerging trends

As AI continues to evolve and become increasingly ubiquitous, the need for effective measurement and evaluation strategies will only continue to grow. Emerging trends such as explainable AI and transparency in machine learning may offer new opportunities for organisations to better understand the impact of their AI interventions, and to develop more targeted and effective measurement approaches.