• Written by Muhammed Abdulla NC, CEO

The head of operations at a regional electricity distributor must decide this week how to allocate a limited budget to reduce the frequency of unplanned outages, and currently bases this decision on a combination of historical data and expert judgement, but the lack of a clear counterfactual makes it difficult to measure the effectiveness of any intervention. Judgement of the effectiveness of an intervention is often flawed because it relies on incomplete data. The complexity of energy and utilities operations, with their asset-heavy infrastructure and geographically dispersed assets, makes it even harder to isolate the impact of a specific intervention.

Challenges in energy and utilities

The energy and utilities sector presents unique challenges when it comes to measuring the ROI of an intervention, particularly due to the need for outage planning, which requires a deep understanding of the complex interplay between different components of the grid. The regulatory reporting burden also adds to the complexity, as companies must comply with strict guidelines and provide detailed reports on their operations. Additionally, the geographically dispersed nature of energy and utilities assets means that data collection and analysis can be time-consuming and costly.

A shift supervisor at a power plant, for example, must balance the need for maintenance with the need to keep the plant running, all while complying with regulatory requirements and managing a team of technicians. This requires a high degree of situational awareness and the ability to make quick decisions based on incomplete data.

Leading measures

Despite the challenges, there are leading measures that can be observed and used to inform decision-making, such as the frequency and duration of outages, as well as the time it takes to restore power. These measures can provide valuable insights into the effectiveness of an intervention and help to identify areas for improvement. A category manager at a utility company, for example, might compare the performance of different suppliers based on these leading measures.

By focusing on these leading measures, companies can get a sense of whether an intervention is having the desired impact, even if the overall outcome is influenced by a variety of factors. This requires a nuanced understanding of the relationships between different variables and the ability to separate signal from noise.

Aggregate outcome metrics

Aggregate outcome metrics, such as the overall reliability of the grid or the total amount of energy produced, are often influenced by a variety of factors, including weather, demand, and equipment failures. As a result, changes in these metrics may not be directly attributable to a specific intervention, making it difficult to measure ROI. Most people mistakenly believe that aggregate outcome metrics are a reliable way to measure the effectiveness of an intervention, but this is wrong because it ignores the complexity of the system.

For example, a decrease in outage frequency might be due to a variety of factors, including changes in weather patterns, improvements in equipment maintenance, or the implementation of new technologies. Without a clear understanding of the underlying causes, it is impossible to determine whether an intervention was effective.

Baselining

Baselining before rollout is crucial to establishing a clear understanding of the current state of operations and providing a benchmark against which to measure the effectiveness of an intervention. This involves collecting data on key metrics, such as outage frequency and duration, and establishing a clear understanding of the relationships between different variables. By doing so, companies can establish a clear counterfactual and make more informed decisions about where to allocate resources.

Reconstructing a baseline after rollout is often impractical and may not provide an accurate picture of the pre-intervention state. A detailed analysis of the data can help to identify areas where the intervention is having the greatest impact and where further improvements can be made.

Limitations

This approach to measuring ROI does not provide a complete picture of the impact of an intervention and should not be relied upon as the sole basis for decision-making. There may be other factors at play that are not captured by the leading measures or aggregate outcome metrics. Companies should consider consulting with experts to get a more comprehensive understanding of the potential benefits and limitations of an intervention.

In some cases, the costs and complexity of collecting and analyzing data may outweigh the potential benefits of measuring ROI, particularly for small-scale interventions or those with limited potential impact. In these cases, a more straightforward approach to decision-making may be necessary.