• Written by Muhammed Abdulla NC, CEO

Customer service teams using AI-powered tools often find that the promised benefits do not materialise, despite the technology functioning as intended, because the metrics used to measure success are not moving for the right reasons. The business consequence of this mismatch is a failure to improve customer satisfaction, and the mechanism behind it is a set of metrics that do not accurately reflect the impact of AI on customer service. Judging the effectiveness of AI in customer service requires a careful selection of metrics that move for the right reason, and first contact resolution is a leading measure that should be prioritised. This is because it directly reflects the ability of the AI system to resolve customer issues efficiently, and a failure to improve this metric indicates a problem with the AI implementation.

Leading measures: first contact resolution, time to answer

A shift supervisor checks the first contact resolution rate to determine if the AI-powered chatbot is effectively resolving customer issues, and this metric is a key indicator of the AI system's success. The time to answer is also a critical measure, as it directly affects customer satisfaction, and a well-implemented AI system should be able to reduce this time significantly. By prioritising these leading measures, customer service teams can ensure that the AI system is having a positive impact on customer satisfaction, and make adjustments as needed to improve the system's performance.

For example, a category manager compares two suppliers of AI-powered customer service tools, and finds that one supplier's tool is able to resolve customer issues more efficiently, resulting in a higher first contact resolution rate. This information can be used to inform the purchasing decision, and ensure that the selected tool is able to meet the needs of the customer service team. By focusing on leading measures like first contact resolution and time to answer, customer service teams can make informed decisions about AI implementation, and ensure that the technology is being used effectively.

Why the headline metric moves for unrelated reasons

Most people get wrong the idea that the headline metric, such as customer satisfaction, is always a reliable indicator of the AI system's success, but this is not the case. The headline metric can move for unrelated reasons, such as changes in the customer base or external factors like economic conditions, and this can make it difficult to determine the true impact of the AI system. For instance, a change in the customer base may result in a more satisfied customer group, regardless of the AI system's performance, and this can mask the true effectiveness of the AI system.

This is why it is essential to look beyond the headline metric and examine the underlying leading measures, such as first contact resolution and time to answer, to get a more accurate picture of the AI system's performance. By doing so, customer service teams can identify areas for improvement and make data-driven decisions to optimise the AI system, and ensure that it is having a positive impact on customer satisfaction.

Baselining before rollout

Baselining the current state of customer service operations before rolling out an AI-powered tool is crucial to understanding the impact of the technology, and this requires careful planning and execution. A thorough baseline assessment should include metrics such as first contact resolution, time to answer, and customer satisfaction, as well as an analysis of the current workflows and processes. This will provide a clear understanding of the starting point and enable customer service teams to measure the effectiveness of the AI system, and make adjustments as needed to ensure its success.

For example, a customer service team uses operations data to establish a baseline for first contact resolution, and then tracks changes in this metric after implementing an AI-powered chatbot. By comparing the before and after data, the team can determine the true impact of the AI system on customer service operations, and make informed decisions about future improvements. This approach ensures that the AI system is being used effectively, and that customer satisfaction is being improved.

What to record when nothing was found

When an AI-powered customer service tool is unable to find a solution to a customer's issue, it is essential to record this information to understand the limitations of the technology, and to identify areas for improvement. This can include tracking the types of issues that the AI system is unable to resolve, as well as the frequency and severity of these issues. By analysing this data, customer service teams can refine the AI system and improve its performance over time, and ensure that it is able to meet the needs of customers.

For instance, a customer service representative uses the AI-powered tool to try and resolve a customer's issue, but the tool is unable to find a solution. In this case, the representative should record the details of the issue, including the type of problem and the outcome, to help improve the AI system's performance. This information can be used to update the AI system's knowledge base, and to ensure that it is able to resolve similar issues in the future. By recording this information, customer service teams can continually improve the AI system, and ensure that it is providing the best possible service to customers.

What this does not do, or when it is the wrong choice

This approach to measuring AI in customer service does not provide a comprehensive solution for all customer service challenges, and there may be situations where it is not the best choice. For example, in situations where the customer service team is dealing with highly complex or nuanced issues, a more traditional approach to customer service may be more effective. In these cases, the AI system may not be able to provide the level of understanding and empathy required to resolve the issue, and a human customer service representative may be needed instead.

In such cases, customer service teams should consider alternative approaches, such as using human customer service representatives to handle complex issues, or implementing a hybrid approach that combines the benefits of AI with the empathy and understanding of human representatives. By carefully considering the strengths and limitations of AI in customer service, teams can make informed decisions about when to use the technology, and how to ensure that it is providing the best possible service to customers.