How to measure AI in human resources
Evaluating AI in human resources requires tracking leading measures such as repeat question volume and time to answer, which provide insights into the system's performance and employee needs, allowing organisations to adjust training programmes and improve response times, as seen in the example of a shift supervisor checking the maintenance log.
A human resources dashboard that nobody opens is a common sight in many organisations, with metrics that fail to provide meaningful insights into the impact of AI on HR processes. The problem often lies in the choice of metrics, which can be misleading or unresponsive to changes in the organisation. To get the most out of AI in human resources, it is essential to focus on measures that move for the right reason.
Leading measures
Leading measures such as repeat question volume and time to answer are essential in evaluating the effectiveness of AI in human resources. These metrics provide insights into how well the AI system is performing and whether it is meeting the needs of employees. For instance, a decrease in repeat question volume may indicate that the AI system is providing accurate and helpful responses, while an increase in time to answer may suggest that the system is struggling to keep up with demand.
A shift supervisor checks the maintenance log and notices that the number of repeat questions about employee benefits has decreased significantly since the introduction of the AI-powered chatbot. This decrease in repeat question volume is a leading indicator that the AI system is working effectively, and the supervisor can use this information to adjust the training programme for the chatbot.
Why the headline metric moves for unrelated reasons
Most people get wrong the idea that the headline metric, such as employee satisfaction, is a reliable indicator of the effectiveness of AI in human resources. However, this metric can move for unrelated reasons, such as changes in company policies or external factors like economic conditions. For example, an increase in employee satisfaction may be due to a recent salary increase rather than the introduction of the AI system.
It is wrong to assume that the headline metric is a direct result of the AI system, and organisations should be cautious when interpreting these metrics. Instead, they should focus on leading measures that are more closely tied to the performance of the AI system.
Baselining before rollout
Baselining before rollout is crucial in establishing a benchmark for the effectiveness of AI in human resources. By collecting data on key metrics before introducing the AI system, organisations can establish a baseline and measure the impact of the system over time. This baseline data can also be used to identify areas where the AI system can have the most significant impact.
For instance, the category manager compares two suppliers and notices that the current supplier is struggling to provide timely responses to employee queries. By baselining the current response times, the category manager can measure the improvement in response times after introducing the AI-powered chatbot and make informed decisions about future investments.
Organisations can visit our human resource page to learn more about how to baseline before rollout and establish effective metrics for evaluating the impact of AI in human resources.
What to record when nothing was found
When the AI system is unable to find an answer to an employee's question, it is essential to record this information to improve the system's performance over time. By tracking instances where the system was unable to find an answer, organisations can identify areas where the system needs to be improved and update the training data accordingly.
This information can also be used to provide feedback to employees and let them know that their question has been received and will be addressed by a human representative. For example, the AI system can send a response saying that it was unable to find an answer and that a human representative will be in touch soon.
What this does not do
This approach to measuring AI in human resources does not provide a complete solution for all organisational needs. It is limited to evaluating the effectiveness of AI in human resources and does not address other critical areas such as employee engagement or retention. Organisations should be aware of these limitations and use this approach in conjunction with other metrics and evaluation methods.
For organisations that need more comprehensive support, our team is available to provide guidance and answer questions. Please visit our contact us page to get in touch with our team and learn more about how we can help.
Frequently asked questions
What metrics should I use to evaluate the effectiveness of AI in human resources?
Leading measures such as repeat question volume and time to answer provide valuable insights into the system's performance and employee needs.
Why are headline metrics like employee satisfaction unreliable indicators of AI effectiveness?
Headline metrics can move for unrelated reasons, such as changes in company policies or external factors, making it essential to focus on leading measures tied to the AI system's performance.
What is the importance of baselining before rolling out an AI system in human resources?
Baselining establishes a benchmark for the AI system's effectiveness, allowing organisations to measure its impact over time and identify areas for improvement.
How can I improve the performance of an AI system that is unable to find answers to employee questions?
Recording instances where the system was unable to find an answer helps identify areas for improvement and update the training data, enabling the system to provide more accurate responses over time.