AI model security: threats, frameworks, and best practices

A few years ago, most AI tools only answered questions. Today, they can search company files, write and run code, update records, and send messages for employees. Every new task can give the AI more access, and for attackers—more ways to misuse it. AI model security is the practice of protecting AI models, and the data and tools they depend on, from tampering, theft, and misuse. This article covers the main threats, the frameworks that address them, and practical steps to reduce risk.

AI model security: key takeaways

• Protect more than the model. Training data, model files, retrieval systems, APIs, and connected tools all create possible entry points for attackers.
• Know the difference between threats. Data poisoning changes what a model learns, while prompt injection tries to redirect its behavior during use.
• Limit access and autonomy. Give models only the data and permissions they need, and check sensitive actions outside the model.
• Test beyond accuracy. Ordinary benchmarks can miss security risks, so add adversarial testing and AI model security scanning.

What is AI model security?

AI model security means protecting AI models and the components that influence how they learn, respond, and operate. That includes training data, model weights, software dependencies, and access to deployed models. It also covers what happens when an application built on a model reads external content or uses tools. Both the NSA and the UK National Cyber Security Centre (NCSC) recommend treating AI security as a responsibility that lasts through the whole lifecycle, from development to deployment and maintenance.

Because this definition covers both the model and the application around it, attacks fall into 2 groups:

1. Model-level attacks target the model itself: its training data, learned behavior, files, or weights. The damage stays in the model until it is fixed or replaced.
2. Application-level attacks target the system that uses the model, such as prompts, retrieved content, outputs, or tool permissions. They influence individual responses without changing the model.

Suppose an attacker adds manipulated examples to a training dataset. The model learns the wrong behavior, so any future response can be affected, and the fix starts with controlling and checking training data.

Now, suppose an attacker hides instructions in a document that an AI assistant later retrieves. The model stays unchanged, but the responses that use that document are affected. Here, the fix is to control which content the assistant can read and what it is allowed to do with it.

Both attacks can produce unsafe answers, but stricter training data controls won’t stop the second attack, and tighter retrieval permissions won’t stop the first.

Why AI model security matters now

AI tools are no longer limited to answering questions. Businesses connect them to knowledge bases, code repositories, customer records, and workflows. Each new connection gives the AI system more access and provides attackers with more ways to cause damage if they can manipulate it.

Attackers have already started to exploit these connections. In September 2026, the Google Threat Intelligence Group reported on malware that planted malicious configuration files to make AI coding assistants run attacker-selected commands during routine use. The same report described investigations in which attackers stole proprietary AI assets, including a healthcare company’s AI model and drug research. In other words, both the tools around models and the models themselves are now targets.

Regulators are raising expectations as well. The EU AI Act includes cybersecurity, accuracy, and robustness requirements for high-risk AI systems, with the relevant rules scheduled to apply from December 2027 or August 2028, depending on the system category. As a result, organizations will need to show that their AI systems are secure, not only that a pilot worked and accuracy scores were acceptable.

Key AI model security vulnerabilities and threats

The threats below follow the 2 groups described earlier. The first 4 threats mainly target the model and its supply chain. The rest exploit how a deployed application interacts with users, data, and business systems.

1. Data poisoning and backdoors

Attackers can manipulate training or fine-tuning data to influence what a model learns. The result may be misleading answers, reduced accuracy, or a hidden behavior that only activates when a particular trigger appears. A backdoored model can perform normally on everyday tests, which makes the problem difficult to spot.

2. Evasion and adversarial inputs

Unlike poisoning, evasion attacks happen after training. In these attacks, an attacker changes an input to make a model classify or interpret it incorrectly, for example, to get past a detection system, without changing the model itself. The NIST adversarial machine learning taxonomy treats these attacks separately, because protecting training data won’t stop manipulated inputs at runtime.

3. Model theft and privacy attacks

A model is valuable intellectual property, and attackers can target it in more than one way. They may steal its files or weights from storage, or send many queries to an API to recreate some of its capabilities.

Privacy attacks target the training data instead:

• Membership inference tries to determine whether a specific record was in the training set.
• Model inversion and training-data extraction may expose information the model learned or memorized.

These attacks don’t always recover exact records or weights, but they can still expose proprietary or personal information.

4. Compromised AI supply chains

Third-party models, datasets, libraries, and adapters can introduce risks before a model is even deployed. Some model file formats can run code when loaded, so importing an untrusted model file can itself cause a security incident. The NCSC recommends scanning and sandboxing third-party models and serialized weights. However, a file scan only looks for unsafe code. It can’t detect a backdoor the model learned during training.

5. Prompt injection and jailbreaking

What if a document tells your AI assistant to ignore its own instructions and send company data somewhere else? That is an example of indirect prompt injection, where malicious instructions are hidden in content the model reads, such as emails, web pages, or retrieved files. In contrast, direct prompt injection comes from what a user types into the application, while jailbreaking usually means trying to bypass a model’s safety restrictions.

The NCSC warns that current LLMs don’t reliably separate instructions from untrusted data. Filters help, but they can’t remove the risk completely. How much damage a successful injection causes depends on which systems and actions the assistant can access.

6. Sensitive information leaks and weak retrieval

AI assistants often search internal files to answer questions, but not every employee should see every document. If retrieval permissions are too broad, a model may receive and disclose information the user was never authorized to access. Exposed system prompts can also reveal internal instructions or secrets, which is why the OWASP advises against storing credentials in prompts. For retrieval-augmented generation (RAG), permissions must be enforced before sensitive content reaches the model.

7. Unsafe outputs and excessive AI agency

Model-generated text becomes a vulnerability when an application passes it directly into a shell command, database query, or web page. The OWASP lists risks such as SQL injection, cross-site scripting, and remote code execution from improper output handling. The risk is higher when an AI agent can also send emails, change business records, or delete files without human approval. A model shouldn’t be able to authorize a risky action simply because it suggested that action itself.

8. Resource abuse and unexpected costs

Attackers don’t have to steal data to cause harm. Repeated API requests, unusually large inputs, or expensive agent workflows can overload systems and increase bills. OWASP calls this “Unbounded Consumption.” Rate limits help, but businesses also need limits on execution time, tool calls, and spending across an entire workflow.