What is AI data poisoning?
AI data poisoning involves manipulating and corrupting the data used to train artificial intelligence (AI) or machine learning (ML) models. Attackers inject malicious or misleading data into the model’s training datasets to shape the model’s behavior.
Because AI models learn entirely from the examples they are fed, tampered data acts like a slow-acting poison It causes the system to develop hidden biases, provide incorrect answers, or ignore security controls depending on the attacker’s goal. This makes AI data security a necessity for any organization deploying AI models.
Anthropic-led research found that even a small amount of malicious data can poison an LLM. AI data poisoning also appears in the OWASP Top 10 for LLMs a document that lists the most critical current risks for web apps, LLMs, and autonomous agentic systems.
A major risk of data poisoning is its long-term impact. It can compromise the data supply chain and change the model’s knowledge base before deployment. The manipulated data may sit quietly in the system until an attacker decides to exploit it.
Data poisoning vs. prompt injection
Despite sharing a common goal—exploiting vulnerabilities in model inputs—data poisoning and prompt injection target different stages of the AI lifecycle. In simple terms, data poisoning corrupts the data an AI system learns from or retrieves, while prompt injection manipulates the instructions the model processes at runtime.
Here are the key differences between them:
Feature | Data poisoning | Prompt injection |
|---|---|---|
When it happens | During pre-training, fine-tuning, or retrieval-augmented generation (RAG) indexing | At runtime, through prompts, user inputs, or external content processed by the AI system |
Persistence | Typically longer-lasting | Typically affects runtime behavior |
Intent | Compromise the functionality and learning process of an AI model, or build backdoors | Exfiltrate sensitive data, spread misinformation, or bypass safety controls |
Attack vectors | Embedding corrupted or malicious data | Manipulating direct inputs or external content to influence the AI system’s behavior |
Types | Targeted and non-targeted data poisoning | Direct and indirect prompt injection |
Examples of data poisoning in AI
- Targeted data poisoning A hacker injects a malware variant into a training dataset and labels it as a “safe file.” This can teach the security system to overlook that variant in future attacks.
- Non-targeted data poisoning An attacker corrupts an AI financial forecasting model by injecting random, chaotic numerical values into historical market datasets, damaging the system’s predictive accuracy.
Examples of prompt injection
- Direct prompt injection An attacker frames a harmful request as a fictional scenario to bypass an AI system’s safety controls, such as asking for step-by-step instructions to hack a corporate server as part of a story.
- Indirect prompt injection An attacker hides malicious instructions on a public webpage, telling an AI system to direct users to a phishing link. If a user asks the AI to summarize the page, the model may follow the hidden instruction and repeat the malicious link.
Cybercriminals can combine data poisoning with other attacks, such as prompt injection, to cause more damage. For example, an insider could corrupt training data and later trigger it to leak sensitive information, create a backdoor, or disrupt the system’s decisions.
How does AI data poisoning happen?
Attackers often look for the weakest links in your data supply chain or internal security to enter your systems. Here is how AI data poisoning attacks usually occur: Insider threats An individual who already has authorized access to your training data can intentionally introduce bias or fake information. Because the system already trusts them, these silent alterations are often missed by standard security audits. The supply chain weak link Most AI models learn from a mix of sources, including external third-party datasets. If one of them is compromised, every model that uses it becomes affected. Your AI’s reliability is only as strong as its least secure data partner. Unauthorized network entry Attackers can gain access to your private datasets through traditional methods such as phishing for a developer’s credentials or using lateral movement once they get a foothold in your network. Once inside, they can quietly rewrite your data before your security team knows there’s a breach.
Where does data poisoning happen in the AI lifecycle?
Data poisoning is most likely to occur when AI training relies on data from external or insufficiently verified sources This includes data that the organization developing the model doesn’t fully control, curate, or validate.
Many machine learning pipelines use third-party, crowdsourced, or publicly available data, such as:
- Scraped web content
- Open datasets
- User-generated content
- Shared datasets used for collaborative training
When exactly can data poisoning happen?
- Pre-training When data is scraped from websites or collected through APIs without sufficient validation, an attacker who controls or manipulates one of these sources may be able to inject malicious data into the dataset.
- Federated learning. Because it takes place across separate participants, detecting malicious contributions can be difficult. A malicious participant may submit manipulated updates that affect the shared model.
- Fine-tuning If the training data has been manipulated or comes from an untrusted source, the model may learn harmful patterns or incorrect information.
- Retrieval-augmented generation (RAG) RAG systems retrieve data from such sources as internal databases, document repositories, and websites. Attackers who compromise those sources can plant malicious content that affects the model’s responses.
- Synthetic data AI-generated data that is reused for training can create another attack path. If malicious or inaccurate synthetic data enters the training pipeline without proper checks or clear attribution, it can influence later versions of the model.
Types of data poisoning attacks
Attackers choose their method based on whether they want to compromise the model entirely or keep a backdoor for future use.
Targeted vs. non-targeted attacks
- Targeted attacks These attacks aim to make the AI behave incorrectly for a specific target, such as a particular user, class, topic, or input, while leaving its behavior in other cases largely unaffected.
- Non-targeted attacks These resemble sabotage. The threat actor wants to degrade the overall performance of the AI, making it slow, inaccurate, or unusable. This is often seen in corporate espionage or disruptive hacktivism.
Specific attack methods
An attacker can poison data in many ways:
- Backdoor attacks The attacker plants a hidden vulnerability in the model. In response to the attacker’s specific trigger, the model behaves unsafely. Otherwise, the AI behaves normally, but it will do exactly what the attacker wants when it sees that trigger.
- Mislabeling An attacker changes how the training data is labeled, for example, marking “malicious software|” as “safe system file”, to teach the model to ignore threats.
- Data injection and manipulation Here, the threat actor adds or deletes specific information to change how the AI “thinks.” The goal is usually to make the AI biased, such as forcing a loan-approval tool to always reject a certain group of people or always approve the attacker’s own fraudulent applications.
- Availability attacks The threat actor floods the AI with “junk data” that confuses the system so much that it becomes slow, makes random errors, or crashes completely, making it useless for your business.
Data poisoning in generative AI and chatbots (LLMs)
Most users worry about ChatGPT security risks but the real danger is how LLMs find and trust information. In a RAG-based poisoning attack, an attacker doesn’t need to retrain ChatGPT. They only need to place a “poisoned” document on a website that the chatbot’s RAG system scrapes.
If a company deploys a helpful chatbot for its employees and a staff member asks, “How do I reset my company password?” the chatbot scans its data sources to find the answer. If an attacker has poisoned those sources, the chatbot might confidently reply: “To reset your password, please click this link and follow the prompts.”
In reality, that link leads to a malicious phishing site. By poisoning the source document, the attacker has turned the company’s own trusted AI into a social engineering tool—stealing credentials without ever breaking into the system.
Real-world cases show these risks are not theoretical. The examples below highlight how flawed training data, compromised models, and unchecked bias have already caused problems in production AI systems.
3 examples of data integrity and bias risks in AI
The following examples show how problems with training data, model integrity, and bias can affect AI systems. They are not all examples of data poisoning in the strict sense, but they illustrate related risks that can cause models to produce misleading, biased, or deliberately manipulated outputs.
Amazon’s recruiting bias
In 2018, Reuters reported that Amazon had scrapped an experimental AI recruiting tool after discovering that it discriminated against women. The system had learned from about 10 years of past résumés, most of which came from men, and started penalizing applications containing terms such as “women’s,” including references to women’s clubs or colleges.
Amazon tried to fix the issue but eventually abandoned the tool. This case shows how AI systems can reproduce patterns found in historical training data.
PoisonGPT
In 2023, researchers from Mithril Security showed how a modified open-source language model can be used to spread targeted false information They altered GPT-J-6B so that it gave a false answer to a factual question while performing almost identically to the original model on other tests.
The researchers then uploaded the poisoned model to Hugging Face under an account name designed to resemble the legitimate publisher. The experiment showed how tampered models can enter the AI supply chain and appear trustworthy to users who do not verify where they came from
Bias in AI systems
In 2022, NIST published a guide on identifying and managing bias in AI systems. It explains that bias can come from several sources beyond training data or algorithms
The guide groups AI bias into 3 categories: systemic, statistical, and human. Each can affect how AI systems make predictions, recommendations, and decisions. NIST recommends addressing bias throughout the full AI lifecycle with better data practices, regular testing and evaluation, and human oversight, while acknowledging that eliminating bias completely may not be possible.
Why AI data poisoning is dangerous
Data poisoning is a serious risk to businesses that use AI models to make critical decisions. Whether it’s a medical diagnosis or a financial forecast, the reliability of the outcome depends entirely on the integrity of the data behind it.
Generative AI can introduce additional risks. Modern systems often use Retrieval-Augmented Generation (RAG), a method that allows AI models to retrieve information from external sources, such as websites or internal databases, when generating responses. If those sources are compromised, malicious data slips in unnoticed, influencing results without ever touching the model’s core code.
Even a small amount of poisoned data may cause serious problems. Research on medical LLMs has shown that poisoning just 0.001% of a training dataset can trigger measurable failures. In sectors like finance, this tiny fraction could cause a fraud detection system to overlook malicious patterns.
Recovering from data poisoning can also be difficult and expensive. Once a model is trained on malicious data, fixing the issue often requires removing the corrupted data and retraining the entire model from scratch.
How to detect and prevent data poisoning
According to IBM’s Cost of a Data Breach Report 2026 data poisoning was a factor in 26% of AI-related breaches this year, which is a 15% increase from the previous year. Among all AI-related breaches, 32% of organizations reported a loss of data integrity, which is a direct consequence of poisoning attacks. These numbers show the necessity for businesses to detect and prevent AI data poisoning.
Here is what you can do to protect your business:
1. Validate training data sources
Only use trusted, verified datasets for training your AI models. If you are scraping the web, implement strict domain allowlisting to avoid low-authority sites that are easy for attackers to manipulate.
2. Filter and sanitize data
Filtering and sanitizing data during AI model training helps improve model generalization, reduce bias, and prevent models from learning sensitive, toxic, or unwanted content. Unlike real-time inference filtering, which must process inputs with minimal delay, training-phase filtering is performed offline and can apply more extensive checks across large datasets to improve data quality before training begins.
3. Tighten access controls and pipeline security
Secure the environment where your data lives. Implement strict access controls—such as an identity and access management (IAM) solution and multi-factor authentication—so that unauthorized users can’t inject files into your RAG databases or training buckets.
4. Conduct adversarial testing and red teaming
Before deploying a model, subject it to adversarial and red team testing. Both are evaluation methods used to stress-test machine learning models. They can help discover vulnerabilities and trigger harmful behavior before public deployment. This way, you can patch blind spots in the model and training pipelines.
5. Deploy runtime guardrails and continuous monitoring
Runtime guardrails and continuous monitoring help control AI behavior. They can inspect inputs, monitor model activity, validate outputs, block prompt injection attempts, redact sensitive data, and stop unauthorized or unsafe tool actions before they are executed.
Conclusion
AI data poisoning is a complex threat that traditional security tools often miss. Whether caused by employees using shadow AI over-privileged agents, or vulnerabilities in the training pipeline, the risks are serious.
Reducing these risks starts with tighter control over the data pipeline NordLayer’s secure access controls and zero-trust framework can help with that. Organizations should also verify their training data sources, limit access to sensitive datasets, track changes, test models for unusual behavior, and use adversarial training where appropriate.