• Written by Muhammed Abdulla NC, CEO

Every week, executive leadership teams sit through meetings where they are asked to approve investments in new initiatives, but the return on investment for these initiatives is often unclear. This is because measuring the impact of an intervention is difficult when the counterfactual - what would have happened without the intervention - is invisible. In the context of AI, this problem is particularly pronounced, as the benefits of AI are often indirect and diffuse.

Leading measures you can actually observe

When evaluating the ROI of an AI intervention, it's tempting to focus on lagging measures such as revenue or profitability. However, these measures are often too far removed from the intervention itself to be useful. Instead, executive leadership should focus on leading measures that can be directly observed, such as the number of employees using the AI system or the frequency of AI-driven insights being presented to the leadership team. By tracking these leading measures, executive leadership can get a sense of whether the AI intervention is having the desired impact.

A good example of this is a shift supervisor checking the maintenance log to see if the AI-driven predictive maintenance system is reducing downtime. If the system is working as intended, the supervisor should see a decrease in downtime over time. This leading measure can then be used to estimate the potential ROI of the AI intervention.

Why aggregate outcome metrics move for unrelated reasons

Aggregate outcome metrics such as revenue or profitability are often influenced by a wide range of factors, many of which are unrelated to the AI intervention. For example, a change in market conditions or a new competitor entering the market can all impact revenue, regardless of the AI intervention. This makes it difficult to isolate the impact of the AI intervention and estimate its ROI.

Most people get this wrong by assuming that aggregate outcome metrics are a reliable way to measure the impact of an AI intervention. They are wrong because these metrics are simply too noisy to be useful. A more nuanced approach is needed to accurately estimate the ROI of an AI intervention.

The executive leadership data this depends on

To accurately estimate the ROI of an AI intervention, executive leadership needs access to a wide range of data, including consolidated financials, operational KPIs, sales performance, and headcount. This data can be used to create a baseline understanding of the organization's current performance and to track changes over time. By combining this data with leading measures such as employee usage or insight frequency, executive leadership can get a more complete picture of the AI intervention's impact.

For example, a category manager comparing two suppliers might use data on sales performance and headcount to estimate the potential ROI of an AI-driven supplier selection system. By analyzing this data, the category manager can determine whether the AI system is having a positive impact on the organization's bottom line.

What it changes about: cross-functional questions nobody can answer in the room

AI interventions often have a cross-functional impact, affecting multiple departments and teams within the organization. This can make it difficult for executive leadership to get a clear understanding of the intervention's impact, as different functions may have different versions of the same data. By using AI to analyze data from across the organization, executive leadership can get a more complete picture of the intervention's impact and answer cross-functional questions that nobody can answer in the room.

For more information on how AI can support executive and strategic leadership, please visit our website. By using AI to analyze data from across the organization, executive leadership can make more informed decisions and drive better outcomes.

Measured with: decision latency, actions of the form come back with numbers, definition conflicts resolved

To measure the impact of an AI intervention, executive leadership can use a range of metrics, including decision latency, actions of the form "come back with numbers", and definition conflicts resolved. Decision latency refers to the time it takes for employees to make decisions using the AI system, while actions of the form "come back with numbers" refer to the frequency with which employees are using the AI system to inform their decisions. Definition conflicts resolved refer to the number of disputes or inconsistencies in data that are resolved through the use of AI.

By tracking these metrics, executive leadership can get a sense of whether the AI intervention is having the desired impact and make adjustments as needed. For example, if decision latency is decreasing, it may be a sign that the AI system is having a positive impact on employee productivity.

What this does not do

This approach to measuring the ROI of an AI intervention does not provide a silver bullet for evaluating the impact of AI. It is not a replacement for careful planning and execution, and it does not guarantee a positive ROI. However, by using leading measures, analyzing data from across the organization, and tracking key metrics, executive leadership can make more informed decisions about AI investments and drive better outcomes. If you have questions about how to apply this approach in your organization, please contact us for more information.