• Written by Azhar Aliyar, Director

A procurement manager at a large organisation is deciding this week whether to query the purchase order table using AsscherAi, a platform that lets them query their own data in natural language and get answers in real time. They are considering the risks of prompt injection, which is when untrusted content instructs a model to produce a specific output, treated as a data boundary problem. Prompt injection occurs when a malicious input is crafted to manipulate the model into producing a desired response, potentially leading to security breaches or data leaks.

Definition

Prompt injection is a specific type of vulnerability that arises when a model is instructed to perform a task by an untrusted source. This can happen when a user inputs a malicious prompt that is designed to manipulate the model into producing a specific output. The model, in turn, may process this input without proper validation, leading to potential security risks. In the context of AsscherAi, prompt injection can occur when a user queries the system with a malicious intent, such as attempting to extract sensitive information from the purchase order table.

The definition of prompt injection is often confused with other types of vulnerabilities, but it is a distinct concept that requires careful consideration. By understanding what prompt injection is and how it occurs, organisations can take steps to mitigate its risks and protect their data.

Difference from other vulnerabilities

Prompt injection is often confused with data poisoning, which is a related but distinct concept. Data poisoning occurs when an attacker attempts to manipulate a model by introducing malicious data into the training set. In contrast, prompt injection occurs when a malicious input is crafted to manipulate the model at runtime. While both types of vulnerabilities can have serious consequences, they require different approaches to mitigation.

Another key difference between prompt injection and other vulnerabilities is that it is a data boundary problem, rather than a traditional security vulnerability. This means that it requires a different set of controls and mitigations to prevent, such as input validation and sanitisation. By understanding the differences between prompt injection and other vulnerabilities, organisations can develop effective strategies to protect their data and prevent security breaches.

Applicability in a large organisation

In a large organisation, prompt injection can have serious consequences, particularly if the model is used to query sensitive data such as the purchase order table. For example, a malicious user may attempt to craft a prompt that extracts sensitive information from the table, such as supplier names or pricing data. If the model is not properly secured, this could lead to a security breach or data leak.

To mitigate the risks of prompt injection, organisations can implement controls such as input validation and sanitisation, as well as regular security audits and testing. By taking a proactive approach to security, organisations can protect their data and prevent security breaches. For more information on how to implement these controls, users can visit the AsscherAi website.

What it does not do

Prompt injection is not a magic bullet that can be used to extract any information from a model. It is a specific type of vulnerability that requires careful crafting and exploitation. Additionally, prompt injection is not a substitute for traditional security measures, such as access controls and authentication. Rather, it is a complementary control that can be used to further secure a model and prevent security breaches.

Most people get wrong that prompt injection is a trivial vulnerability that can be easily exploited. However, this is not the case. Prompt injection requires a deep understanding of the model and its vulnerabilities, as well as the ability to craft malicious inputs that can exploit these vulnerabilities. By understanding what prompt injection is and how it works, organisations can take steps to mitigate its risks and protect their data. For further assistance, users can contact the AsscherAi support team.

When it is the wrong choice

Prompt injection is not the right choice for every organisation or use case. In some cases, the risks associated with prompt injection may outweigh the benefits, particularly if the model is used to query sensitive data. In these cases, organisations may need to consider alternative approaches to security, such as traditional access controls and authentication. By carefully evaluating the risks and benefits of prompt injection, organisations can make informed decisions about how to secure their data and prevent security breaches.