• Written by Azhar Aliyar, Director

When evaluating AsscherAi for a large organisation, a crucial concept to understand is the token, which is a unit of text, such as a word or a character, that is used to measure the amount of data being processed by the system. In the context of AsscherAi, a token is the basic unit of billing and limitation, determining how much data can be queried and analysed. For instance, a category manager comparing two suppliers using AsscherAi would be using tokens to query the purchase order table and retrieve relevant data.

Definition of a Token

A token is a discrete unit of text that is used to represent a single item of data, such as a word, a phrase, or a character. In the context of AsscherAi, tokens are used to measure the amount of data being processed, and are typically counted as individual units of text. This means that a single token can represent a single word, such as "product", or a single character, such as "@". The token is the fundamental unit of measurement for AsscherAi's billing and limitation system.

The definition of a token is important to understand, as it has a direct impact on how much data can be queried and analysed using AsscherAi. For example, a shift supervisor checking the maintenance log using AsscherAi would be using tokens to retrieve relevant data, and the number of tokens used would determine the cost of the query.

Difference from Other Concepts

One common source of confusion is the difference between a token and a record. While a record refers to a single entry in a database, such as a customer record or a sales record, a token refers to a single unit of text, such as a word or a character. This distinction is important, as it affects how data is measured and billed. For instance, a customer record may contain multiple tokens, such as the customer's name, address, and phone number.

Most people get wrong the idea that a token is equivalent to a record, but this is incorrect. A token is a much smaller unit of data, and is used to measure the amount of text being processed, rather than the number of records being queried.

Application in a Large Organisation

In a large organisation, tokens are used to query and analyse data from various sources, such as the purchase order table, the customer database, or the sales log. For example, a category manager may use AsscherAi to query the purchase order table and retrieve data on sales trends, using tokens to measure the amount of data being processed. The number of tokens used would determine the cost of the query, and the organisation would be billed accordingly.

The use of tokens in a large organisation is critical, as it allows for the efficient and cost-effective analysis of large datasets. By understanding how tokens work, organisations can optimise their use of AsscherAi and ensure that they are getting the most out of their data. For more information on how to use AsscherAi in your organisation, visit our website.

Limitations of Tokens

While tokens are a powerful tool for measuring and analysing data, they do have limitations. For example, tokens do not take into account the complexity or relevance of the data being queried. This means that a query that returns a large amount of irrelevant data may use up a large number of tokens, even if the data is not useful to the organisation.

In addition, tokens do not provide any information about the context or meaning of the data being queried. This means that organisations must use other tools and techniques, such as data visualisation and analysis, to understand the insights and trends in their data. For help with using AsscherAi and understanding its limitations, contact our support team at contact-us.

When Tokens are the Wrong Choice

In some cases, tokens may not be the best choice for measuring and analysing data. For example, if an organisation needs to query and analyse large amounts of unstructured data, such as text documents or images, tokens may not be the most effective unit of measurement. In these cases, other tools and techniques, such as natural language processing or machine learning, may be more suitable.

Ultimately, the choice of whether to use tokens or another unit of measurement depends on the specific needs and goals of the organisation. By understanding the strengths and limitations of tokens, organisations can make informed decisions about how to use AsscherAi and other data analysis tools to get the most out of their data.