• Written by Muhammed Abdulla NC, CEO

Every week, research and development teams sit through meetings where they are asked to justify the return on investment for their projects, only to struggle with the question of how to measure the impact of their work. This is because the counterfactual - what would have happened if the project had not been undertaken - is often invisible, making it difficult to determine the true value of the investment.

This challenge is particularly pronounced in research and development, where the outcome of a project may not be immediately apparent, and the path to success is often marked by setbacks and failures. However, by focusing on leading measures that can be observed, teams can begin to build a clearer picture of the value they are creating.

Leading measures you can actually observe

One of the key challenges in measuring the ROI of research and development is that the outcomes are often lagging indicators, meaning they only become apparent after a significant amount of time has passed. However, by focusing on leading measures such as the number of experiments conducted, the number of prototypes built, or the number of design reviews completed, teams can get a sense of the progress they are making towards their goals.

For example, a research team working on a new materials project might track the number of samples they are able to produce each week, or the number of tests they are able to conduct on those samples. By monitoring these leading indicators, the team can identify areas where they need to improve their processes, and make adjustments to get back on track.

Why aggregate outcome metrics move for unrelated reasons

Aggregate outcome metrics, such as the overall success rate of projects, or the total revenue generated by new products, are often influenced by a wide range of factors, many of which are unrelated to the specific research and development project being undertaken. This means that changes in these metrics may not accurately reflect the value created by the project.

For instance, a change in the overall success rate of projects might be due to a shift in the types of projects being undertaken, rather than any specific improvement in the research and development process. By focusing on more granular, project-specific metrics, teams can get a clearer understanding of what is driving their outcomes.

The research and development data this depends on

In order to effectively measure the ROI of research and development, teams need access to a wide range of data, including test results and failure analyses, design review records, timesheets and allocation data, and supplier assessments. This data can help teams understand what is working, and what is not, and make informed decisions about where to focus their efforts.

For example, a team working on a new product development project might use data from design reviews to identify areas where the design is not meeting requirements, and adjust their approach accordingly. They might also use data from timesheets to understand where their team is spending most of its time, and look for opportunities to streamline their processes.

By linking this data to the research and development process, teams can begin to build a more complete picture of the value they are creating, and make more informed decisions about their investments.

What it changes about: failures rediscovered across programmes and effort invisible against project stage

One of the key benefits of using leading measures to evaluate the ROI of research and development is that it can help teams avoid rediscovering the same failures across multiple programmes. By tracking the number of experiments conducted, or the number of prototypes built, teams can identify areas where they are struggling, and make adjustments to avoid repeating the same mistakes.

Additionally, by focusing on effort against project stage, teams can get a better understanding of where their efforts are being spent, and make more informed decisions about resource allocation. This can help ensure that the right amount of effort is being applied to each project, and that resources are not being wasted on projects that are not likely to succeed.

Measured with: failures found before validation, effort against stage, estimate revision count

There are a number of metrics that can be used to measure the ROI of research and development, including failures found before validation, effort against stage, and estimate revision count. These metrics can provide a more nuanced understanding of the value being created by research and development projects, and help teams identify areas for improvement.

For example, a team might track the number of failures found before validation, in order to understand how effective their testing and quality control processes are. They might also track effort against stage, in order to understand where their team is spending most of its time, and look for opportunities to streamline their processes.

What this does not do

While using leading measures to evaluate the ROI of research and development can provide a number of benefits, it is not a silver bullet. It will not provide a complete picture of the value being created by every project, and it will not be effective in every context. Teams need to carefully consider their specific needs and circumstances, and develop a measurement approach that is tailored to their unique requirements.

If you are struggling to develop an effective measurement approach for your research and development projects, contact us to learn more about how we can help.