• Written by Muhammed Abdulla NC, CEO

What is anomaly detection, and how can it help a large organisation like ours identify deviations that matter only in combination, asked the IT director during our last meeting, highlighting the complexity of the issue. Anomaly detection refers to the process of identifying data points or patterns that do not conform to expected behaviour, often indicating a problem or opportunity that requires attention. In the context of AsscherAi, anomaly detection is used to find deviations in an organisation's own data that may signal a issue or an opportunity.

Definition and Clarification

Anomaly detection is often confused with outlier detection, but the two are not the same. Outlier detection typically involves identifying individual data points that are significantly different from the rest, whereas anomaly detection looks for patterns or combinations of data points that are unusual. This distinction is important, as anomaly detection is concerned with finding deviations that may not be immediately apparent from looking at individual data points.

A concrete example of anomaly detection in action is when a shift supervisor checks the maintenance log and notices that a particular machine has been experiencing a higher-than-usual number of errors in combination with a specific operator being on duty. This combination of factors may indicate a problem with the machine or the operator's training, and anomaly detection can help identify such patterns.

Differences from Outlier Detection

While outlier detection is primarily concerned with identifying individual data points that are significantly different from the rest, anomaly detection takes a more holistic approach. It looks for patterns or combinations of data points that are unusual, and may not necessarily involve extreme values. This makes anomaly detection particularly useful for identifying complex issues that may not be immediately apparent from looking at individual data points.

Most people get wrong the idea that anomaly detection is simply a matter of applying a set of predefined rules to the data. In reality, anomaly detection often requires a deep understanding of the underlying data and the context in which it is being used. For example, a category manager comparing two suppliers may use anomaly detection to identify patterns in the data that indicate a problem with one of the suppliers, but this requires a thorough understanding of the data and the business context.

Applications in a Large Organisation

Anomaly detection has a number of applications in a large organisation, particularly in areas such as maintenance, logistics, and supply chain management. By identifying deviations in the data that may indicate a problem or opportunity, organisations can take proactive steps to address issues before they become major problems. For example, anomaly detection can be used to identify patterns in sensor data from machines that may indicate a need for maintenance, allowing organisations to schedule maintenance before a problem occurs.

AsscherAi can be used to support anomaly detection in a large organisation, by providing a platform for querying and analysing the organisation's own data. For more information on how AsscherAi can support anomaly detection, please visit our website.

Limitations of Anomaly Detection

Anomaly detection is not a panacea for all data analysis needs. It is primarily concerned with identifying deviations in the data, and may not provide a complete picture of the underlying issues. Additionally, anomaly detection requires a significant amount of data to be effective, and may not be suitable for organisations with limited data. For example, a small organisation with limited data may find it difficult to use anomaly detection effectively, and may need to consider other approaches.

If you are considering using anomaly detection in your organisation, but are unsure about the best approach, please contact us for more information.

When Anomaly Detection is the Wrong Choice

Anomaly detection is not always the best approach for identifying issues in an organisation's data. In some cases, other approaches such as regression analysis or time series analysis may be more suitable. For example, if an organisation is trying to identify the relationship between a specific variable and a outcome, regression analysis may be a better choice. Anomaly detection is primarily concerned with identifying deviations in the data, and may not provide the level of insight required for more complex analysis.

In general, anomaly detection is best used when there is a need to identify unusual patterns or combinations of data points, and when the organisation has a significant amount of data to analyse. If the organisation is unsure about the best approach, it may be helpful to consider the specific goals and requirements of the analysis, and to consult with a data analysis expert.