• Written by Muhammed Abdulla NC, CEO

A category manager at a large retailer is deciding this week whether to repeat a successful promotion from last quarter, but the decision is based on incomplete information about what drove the promotion's success, and what would happen if it were not repeated. This is a common problem in e-commerce and retail, where the fast pace of business and complexity of customer behaviour make it difficult to measure the impact of interventions.

What makes measurement harder in e-commerce and retail

E-commerce and retail present unique challenges for measuring the return on investment (ROI) of interventions, such as promotions or changes to the product assortment. One of the main difficulties is that the counterfactual - what would have happened if the intervention had not been made - is often invisible. For example, if a promotion is run, it is impossible to know what sales would have been like if the promotion had not been run. This makes it difficult to isolate the impact of the promotion and measure its effectiveness.

Another challenge is the rapid pace of change in e-commerce and retail, with new products and promotions being introduced all the time. This means that the baseline against which the impact of an intervention is measured is constantly shifting, making it difficult to get a clear picture of what is working and what is not. Returns and their root causes are also a major issue, as they can be influenced by a wide range of factors, including product quality, customer behaviour, and logistics.

Leading measures you can actually observe

Despite the challenges, there are some leading measures that can be observed and used to inform decision-making. For example, a shift supervisor can check the maintenance log to see if equipment is being properly maintained, which can help to reduce downtime and improve productivity. Similarly, a category manager can compare sales data for different products to see which ones are performing well and which ones are not.

By focusing on these leading measures, it is possible to get a sense of what is driving sales and profitability, and to make informed decisions about where to invest time and resources. This can help to improve the effectiveness of interventions and increase ROI. For more information on how to use data to inform decision-making, visit our resources page.

Why aggregate outcome metrics move for unrelated reasons

Aggregate outcome metrics, such as total sales or profitability, can be influenced by a wide range of factors, many of which are unrelated to the intervention being measured. For example, a change in the weather or a shift in consumer preferences can impact sales, even if the intervention itself is having no effect. This means that aggregate outcome metrics are not always a reliable way to measure the impact of an intervention.

Most people get wrong the idea that aggregate outcome metrics are a good way to measure ROI, and they are wrong. Aggregate metrics can be useful for getting a sense of overall performance, but they are not a substitute for more detailed analysis and measurement.

Baselining before rollout rather than reconstructing it after

One of the best ways to measure the impact of an intervention is to establish a baseline before the intervention is rolled out. This can involve collecting data on sales, customer behaviour, and other relevant metrics before the intervention is introduced. By comparing this baseline data to data collected after the intervention is rolled out, it is possible to get a clear picture of what is working and what is not.

Reconstructing the baseline after the intervention has been rolled out is often difficult or impossible, which is why it is so important to establish a baseline beforehand. This can help to ensure that the measurement of ROI is accurate and reliable, and that decision-makers have the information they need to make informed choices.

What this does not do, or when it is the wrong choice

Measuring ROI using the methods described above is not a substitute for other forms of analysis and decision-making. For example, it may not be possible to use these methods to measure the impact of an intervention that is still in the planning stages, or to compare the effectiveness of different interventions. In these cases, other approaches may be needed, such as consulting with experts or using other forms of data analysis.

Ultimately, the key to successful measurement and decision-making is to use a combination of approaches and to be flexible and adaptable. By using the methods described above and being open to other forms of analysis and decision-making, it is possible to get a clear picture of what is working and what is not, and to make informed choices that drive business success.