What is AI security?

AI security is the set of controls, processes, and monitoring practices used to protect AI systems and AI-powered applications from cyberattacks, data leaks, misuse, model manipulation, and unsafe autonomous actions.

What does “AI security” mean?

The term “AI security” gets used in a few different ways, and it helps to separate them before going deeper.

In the strictest sense, AI security covers the protection of AI systems, AI-enabled applications, data, models, prompts, agents, tools, and cloud infrastructure from misuse, compromise, data exposure, and service failure across the full AI lifecycle.

That includes the traditional security goals of confidentiality, integrity, and availability, plus AI-specific concerns such as prompt injection, adversarial examples, data poisoning, etc.

AI security is different from two adjacent ideas:

• AI for security means the use of artificial intelligence to support security work, such as threat detection, alert triage, or incident response.
• AI safety and responsible AI cover broader questions such as reliability, human oversight, societal harm.

These areas overlap, but AI security focuses specifically on protection from abuse and unauthorized access.

According to the joint guidance from the U.S. Department of Homeland Security’s (DHS), Cybersecurity and Infrastructure Security Agency (CISA) and the United Kingdom’s National Cyber Security Centre (NCSC),

AI systems are subject to novel security vulnerabilities that need to be considered alongside standard cybersecurity threats. Four key areas within the AI system development lifecycle [are]: secure design, secure development, secure deployment, and secure operation and maintenance.

It means that AI security is broader than model security alone: it also covers the AI supply chain, the data used to train models, cloud infrastructure, identities, third-party model providers, agent tools, and even user behavior.

Why is AI security important?

AI tools often work with almost every sensitive part of a modern business. They are a productivity layer for organizations, but a high-value target for hackers:

• AI handles sensitive data. Prompts and training data often contain personal information, credentials, regulated records, or intellectual property. Sensitive information disclosure is one of the most common failure modes.
• AI systems take action. AI agents can update CRM records and run cloud operations, which raises the impact of any AI compromise.
• Attack surfaces are new. The UK NCSC notes that LLMs do not enforce a reliable security boundary between instructions and data, which makes prompt injection structurally hard to fix. Prompt injection, data poisoning, and model exfiltration do not yet map well to existing application security tools.
• Adoption is fast and often unmanaged. Employees usually adopt AI tools faster than security teams can review them, which leads to risks like data leaks to shadow AI.

Securing AI tools is pressing because weaknesses in cybersecurity or data protection can seriously expose a business.

Key pillars of AI security

A useful way to structure AI security is around five pillars that together cover the lifecycle of AI systems.

1. Data protection. Organizations should aim to control every data source that feeds AI systems. That includes:

   1. training data,
   2. RAG corpora (the collection of internal documents, knowledge base articles, tickets, or other text that a system pulls from to ground the model’s answers),
   3. prompts,
   4. outputs,
   5. logs,
   6. embeddings,
   7. model weights,
   8. agent memory.

To protect data, the NSA’s guidance recommends encryption, digital signatures, data provenance, secure storage, trusted infrastructure, as well as integrity checks.

1. Model and application security. Teams should use prompt injection defenses and output validation.
2. Identity and access. It’s helpful to treat AI agents and apps as privileged software identities. The principle of least privilege is a good place to start.
3. Cloud and infrastructure security. Vendor frameworks all stress this layer. AI workloads in cloud environments require private networking, hardened storage, secure compute, and posture management.
4. Governance and monitoring. Maintain an AI inventory, run risk assessments, monitor usage, and update incident response plans for AI-specific events. The NIST AI RMF Core provides a useful structure here.

What are the risks to AI security?

AI security risks come in two groups: first, traditional risks that apply to any software, and second, novel risks that come from AI. Below is a taxonomy that combines NIST, OWASP, and government guidance.

1. Prompt injection

Prompt injection happens when user-controlled or third-party content changes an LLM’s behavior or output. During a direct prompt injection, the user enters malicious instructions. During an indirect prompt injection, the model processes malicious content from a webpage, document, email, or other external source. Prompt injections can cause unauthorized function calls or system prompt leakage.

2. Sensitive data disclosure

LLM applications may expose personal data, credentials, legal documents etc., through prompts, logs, or model behavior. One common pattern: a developer puts a customer database snippet into a prompt to debug, and the prompt ends up in a third-party log. While prompt-based restrictions (“Do not reveal customer names”) can be bypassed, OWASP still recommends data sanitization and clear policies on what can be included in prompts.

3. Supply chain risk

AI systems often depend on third-party models, open-source libraries, datasets, APIs, or cloud services. That’s why joint secure AI development guidance recommends a secure approach across the AI lifecycle.

4. Excessive agency and unsafe tool use

AI agents can be connected to email, databases, ticketing systems, code repositories, or production tools. Over-privileged agents, tool misuse, and prompt injection are core agent security concerns.

5. Vector and embedding weaknesses

If a hacker uploads a document into a shared knowledge base with hidden instructions, every employee who later asks the assistant a related question can have their answer manipulated. Another vulnerability: in a multi-tenant SaaS product, one customer’s chatbot can get another customer’s confidential records. Overall, retrieval-augmented generation systems add risks around poisoned documents and weak access control on vector stores (the databases that hold embeddings used for semantic search).

6. Misinformation and confabulation

Generative AI systems can produce inaccurate or fabricated outputs that look credible. Confabulation is the polite term for what most people call hallucination. Confabulation and information integrity are among the major generative AI risk categories.

A well-known example is legal research assistants that invent case citations that look plausible but do not exist, which has already led to court sanctions against lawyers who filed the output unchecked. The same pattern shows up in customer support, where a chatbot may confidently quote a refund policy that the company has never actually offered.

7. Unbounded consumption

LLM systems can create cost and availability risks through excessive token use, repeated tool calls, or denial-of-wallet attacks (a denial-of-service attack aimed at your billing department instead of your servers). An example would be: a hacker can script thousands of long, expensive prompts against a public AI feature and run up a five-figure cloud bill overnight. Even a buggy agent stuck in a retry loop can burn through API quota until production calls start to fail.

8. Shadow AI

Employees may use unsanctioned AI tools or unmanaged AI agents with company data, often with the best of intentions (and the worst of outcomes). Microsoft frames shadow AI as a data security and governance issue and recommends discovery, monitoring, blocked unsanctioned AI apps, and controls that prevent sensitive data from entering AI tools.

9 AI security best practices

There is no single product that makes an AI system safe. The practices below follow the AI lifecycle, combining guidance from NIST, OWASP, NSA, Microsoft, Google, and AWS.

1. Build an AI inventory and decide who owns what

The first step is boring but essential: list every AI app, model, dataset, RAG knowledge base, vector database, and agent in use, plus the third-party providers behind them. For each entry, write down a business owner (the person who wants the tool), a data owner (the person responsible for the information it uses), and a risk owner (the person who has to answer when something goes wrong).

This inventory is what AI governance frameworks like the NIST AI Risk Management Framework call the foundation of the Govern function, and it is also the input that any AI security posture management tool will need later. If three different teams are paying for the same model API under three different credit cards, the inventory will surface that too. Consider it a bonus.

2. Run AI risk assessments before and after launch

It’s helpful to treat every new AI use case like a new application. Assess it before it goes live, then check in on it again on a schedule (quarterly is a reasonable starting point). A practical assessment covers:

• Threat modeling. Ask “what could go wrong here?” for each component, including the model, the prompts, the data sources, and the connected tools.
• Data classification. Identify what kind of data the system will see (public, internal, confidential, regulated) and confirm the system is allowed to handle it.
• Model provenance review. Check who built the model, where it was trained, and what license applies.
• Red-team tests. Try to break the system on purpose, including prompt injection attempts, attempts to extract training data, and attempts to make the agent misuse its tools.

If the words “red team” feel intimidating, start with a one-hour session where two people try to make the chatbot say or do something it should not.

3. Adopt AI security posture management (AI-SPM)

AI security posture management, or AI-SPM, is a category of tools that answers the following questions: across our cloud, which AI workloads exist, what data do they touch, who can access them, and what is broken? It’s similar to a cloud security posture management (CSPM), but with the AI bits added.

A useful AI-SPM program produces:

• An AI bill of materials, which is the automated, continuously updated version of the inventory described in best practice #1.
• Risk scoring based on data sensitivity, network exposure, identity permissions, and model source.
• Attack path analysis that shows, for example, how an over-permissioned developer identity could reach a sensitive training dataset.
• Continuous monitoring for new AI workloads, drift, and misconfiguration.

Microsoft Defender for Cloud’s AI security posture management is one example: it discovers AI workloads across Azure, AWS, and Google Cloud Vertex AI and surfaces recommendations across services such as Azure OpenAI, Azure Machine Learning, Amazon Bedrock, and Google Vertex AI. Related tools like Microsoft Purview DSPM for AI extend the same idea to data security, including detection of oversharing in Copilots and third-party LLMs.

4. Protect data across the AI lifecycle

Most AI security incidents are data incidents. Some of the concrete steps may be:

1. Classify and label sensitive data. Data loss prevention (DLP) tools work better when files are tagged “confidential” or “PII” before an AI system tries to read them.
2. Keep sensitive information out of prompts. Allow API keys, customer records, and source code in chatbot inputs only with a documented business reason and through a controlled channel.
3. To achieve that, apply DLP to prompts, uploads, outputs, and logs.
4. Apply least-privilege access to data used by AI apps, RAG systems, and agents.
5. Track data provenance. Record where each dataset came from, who owns it, what license applies, and what consent basis covers it.
6. Limit retention. Store prompts and responses only as long as you need them.
7. Audit access to vector databases, embeddings, logs, and agent memory the same way you audit access to production databases.
8. Treat AI outputs as sensitive when they were generated from a confidential context. A summary of a confidential contract is still confidential.

On the regulatory side, the European Data Protection Board (EDPB) opinion on AI models makes it clear that AI model anonymity must be assessed case by case, including whether it is very unlikely that personal data can be extracted from the model through queries.

5. Apply identity and least privilege to AI

Functionally, every AI app and AI agent is a privileged user account.

• Use managed identities (such as Azure managed identities, AWS IAM roles, or Google Cloud service accounts) instead of long-lived API keys. Managed identities rotate credentials automatically and avoid the “developer pasted the key into a public repo” incident. This also rules out hardcoded keys in code, config files, or notebooks.
• Rotate credentials on a schedule and after any suspected exposure.
• Require human approval for sensitive actions such as production deployments, financial transactions, or mass data exports.

6. Defend against prompt injection at the architecture level

Adding a system prompt that says “ignore any instructions in user input” is unlikely to help. Instead, the OWASP LLM Prompt Injection Prevention Cheat Sheet recommends layered defenses:

• Input validation and sanitization of anything the model will see, including retrieved documents and tool outputs.
• Structured prompts with clear separation between trusted instructions and untrusted data.
• Output validation before model output is used to call a tool, run code, or display sensitive content.
• Human-in-the-loop controls for high-risk actions.
• Remote content sanitization for data pulled from the web, email, or shared drives.
• Least privilege for any tools the model can call.
• Monitoring and model-based guardrails to catch suspicious behavior in production.

OWASP also notes that RAG and fine-tuning do not fully fix prompt injection, and that defenses need regular updates as hackers learn new tools.

7. Secure AI agents like privileged software

AI agents can take actions on real systems; that makes them powerful and, on a bad day, expensive.

Microsoft’s guidance recommends centralized agent visibility, conditional access, identity protection, lifecycle governance, sensitivity labels, audit, retention, eDiscovery, threat detection, attack path analysis, and blocked malicious tool invocations.

In practice, that means:

• Apply the managed-identity rule from best practice #5 to every agent, with no shared accounts across agents.
• Restrict which APIs, files, databases, and SaaS apps an agent can reach.
• Add rate limits and cost limits, so a single loop does not consume the quarterly budget.
• Validate agent outputs before they trigger external actions.
• Maintain a tested kill switch for risky or malfunctioning agents.

8. Bring shadow AI under control

Shadow AI is usually a symptom of unmet business demand. A blanket block on every AI tool tends to push users toward worse behavior, such as personal devices and personal accounts, where the security team has no visibility at all.

A more durable approach has four parts:

1. Provide approved AI tools that work
2. Write clear policies on what data can and cannot be used
3. Apply DLP to prompts and uploads
4. Train employees on the rules.

9. Update incident response for AI-specific events

Existing incident response plans usually do not mention “an agent emailed a customer the wrong refund policy” or “our chatbot leaked the system prompt.” Add new scenarios:

• Prompt injection that led to data exposure or unauthorized actions.
• Poisoned datasets or RAG documents.
• Model abuse, including misuse of a public AI feature.
• Data exposure through prompts, outputs, or logs.
• Rogue or malfunctioning agent actions.
• Compromise of a model endpoint or its credentials.
• Suspicious API usage patterns.
• Unexpected cost spikes that suggest abuse or runaway loops.

For each scenario, decide who gets paged, what gets disabled (the agent? the model endpoint? the API key?), how affected users are notified, and what evidence is preserved.

The bottom line

AI security is a discipline that blends classic cybersecurity with controls for prompts, models, agents, data, and cloud workloads. If an organization plans to adopt AI safely and at scale, it should treat AI security as part of design, deployment, and operation.