What is Generative AI Security? Risks and Controls

Generative AI shipped into production faster than any class of enterprise software in the last two decades. By mid-2026, every large enterprise has at least one customer-facing GenAI application, every developer has a coding assistant, and every employee has access to a generative chatbot of some kind. The security category that grew up around these deployments is generative AI security (you will also see it written as GenAI security or gen AI security), and it has its own risk taxonomy, regulatory frameworks, and tooling stack.

Generative AI security overlaps with but is not identical to AI agent security. Where agentic AI security focuses on systems that take autonomous actions with tools, generative AI security covers the broader population of systems built on generative models, including the chatbot-only and content-generation cases. Most production AI deployments today need both.

The sections below define generative AI security, walk the twelve risk categories that NIST identified in the Generative AI Profile of the AI Risk Management Framework, explain how those risks map to the OWASP Top 10 for LLM Applications, lay out the eight-layer defender's architecture, and answer the questions buyers ask when evaluating the category.

What is generative AI security?

Generative AI security is the discipline of identifying, measuring, and mitigating the risks that arise when an organization builds, deploys, or relies on systems based on generative models. It covers the model itself, the application around it, the data pipeline, and the security and compliance controls that make production deployment defensible.

The shorthand: generative AI security is what stops a generative model from becoming a security incident, a compliance failure, or a public embarrassment for the organization that deployed it. It is the operational discipline that translates the OWASP LLM Top 10, NIST AI 600-1, the EU AI Act, and equivalent frameworks into controls that a defender can run.

Generative AI security vs AI security vs agentic AI security

The three terms are related but not interchangeable. Knowing the difference matters for procurement, ownership, and tooling.

Aspect	AI security (broad)	Generative AI security	Agentic AI security
Scope	All AI systems including traditional ML	Systems built on generative models (LLMs, diffusion, multimodal)	Systems that hold tools, persist state, and take autonomous actions
Primary risks	Model theft, data poisoning, model evasion, fairness, traditional ML attacks	All AI risks, plus generation-specific risks (hallucination, prompt injection, content moderation, IP)	All GenAI risks, plus tool misuse, identity abuse, multi-agent compromise
Canonical framework	NIST AI RMF	OWASP LLM Top 10, NIST AI 600-1	OWASP Top 10 for Agentic Applications
Typical ownership	AI risk team, model risk management	AppSec, security engineering, GenAI program team	Platform engineering, agent platform team, security
Maturity	Mature (5+ years of practice)	Maturing (2-3 years)	Emerging (1-2 years)

Most production deployments touch all three. A bank that runs a fraud-detection ML model needs AI security. The same bank's customer chatbot needs generative AI security. The same bank's autonomous reconciliation agent needs agentic AI security. Programs increasingly run all three as one combined generative AI cybersecurity practice.

The 12 generative AI security risks

NIST AI 600-1, the Generative AI Profile of the AI Risk Management Framework published in July 2024, defines twelve risk categories that are unique to or exacerbated by generative AI. The categories are the working risk taxonomy for the discipline.

1. CBRN information or capabilities. Generative models providing uplift to chemical, biological, radiological, or nuclear weapon development. Concentrated in frontier models with the broadest knowledge.

2. Confabulation. The model produces plausible-sounding but factually false content. The most-cited consumer-facing GenAI risk; covered by OWASP LLM09:2025 Misinformation.

3. Dangerous, violent, or hateful content. Generation of content that violates content policy, incites harm, or causes psychological injury.

4. Data privacy. Generative systems can memorize training data, leak retrieved sensitive content, or expose personal information through unintended channels. The core of generative AI data security, covered by OWASP LLM02:2025 Sensitive Information Disclosure.

5. Environmental impacts. The energy and water consumption of training and serving large generative models.

6. Harmful bias and homogenization. Bias in training data, bias in generated outputs, and the homogenization of perspectives that comes from broad reliance on a small number of frontier models.

7. Human-AI configuration. Risks at the interface between humans and AI: anthropomorphization, automation bias, emotional entanglement, over-reliance.

8. Information integrity. AI-generated misinformation, disinformation, and synthetic media (deepfakes) at scale.

9. Information security. Cybersecurity risks specific to generative AI: prompt injection, jailbreaking, model extraction, supply-chain attacks. The risk category that overlaps most directly with traditional security work.

10. Intellectual property. Generation of content that infringes copyrights, trademarks, or other intellectual property rights. Includes both training-data IP issues and output IP issues.

11. Obscene, degrading, and abusive content. Generation of illegal or abusive content categories.

12. Value chain and component integration. Risks introduced through third-party models, datasets, libraries, MLOps tools, and the broader supply chain, including MCP servers. Covered by OWASP LLM03:2025 Supply Chain.

The NIST profile then provides hundreds of specific suggested actions tied to AI RMF functions and subcategories. For the defender's view of how these risks map to operational controls, read on.

How NIST GenAI risks map to the OWASP LLM Top 10

The OWASP Top 10 for LLM Applications (2025 revision) is the security-engineering canon for generative AI. The security-relevant NIST AI 600-1 categories have direct OWASP analogs.

NIST AI 600-1 risk	OWASP LLM Top 10 mapping
9. Information security	LLM01:2025 Prompt Injection, LLM04:2025 Data and Model Poisoning, LLM05:2025 Improper Output Handling, LLM06:2025 Excessive Agency, LLM07:2025 System Prompt Leakage, LLM10:2025 Unbounded Consumption
4. Data privacy	LLM02:2025 Sensitive Information Disclosure, LLM08:2025 Vector and Embedding Weaknesses
12. Value chain and component integration	LLM03:2025 Supply Chain
2. Confabulation	LLM09:2025 Misinformation

NIST risks 1, 3, 5, 6, 7, 10, and 11 (CBRN, dangerous content, environmental, bias, human-AI configuration, IP, obscene content) cover non-cybersecurity dimensions that the OWASP framework does not address directly. They require their own controls (content policy, safety tuning, IP licensing review, environmental measurement) outside the security stack.

The 8-layer defender's architecture for generative AI

The runtime control architecture that addresses the security and privacy subset of generative AI risks. Each layer catches a different class of attack. None is sufficient alone.

Layer 1: Input guardrails. Classify every text the model ingests, regardless of source: user prompts, retrieved documents, tool results, MCP resources, agent-to-agent messages. Catches prompt injection, jailbreak, and many indirect-injection attacks.

Layer 2: Output guardrails. Inspect what the model says or writes before it reaches the user, the next agent, or the next tool. Catches sensitive data exfiltration, leaked system prompts, off-policy responses, and hallucinated outputs that propagate downstream.

Layer 3: Data guardrails. Govern data flowing into and out of the model's context. PII detection and redaction, retrieval scoping, ACL alignment between source documents and the calling user.

Layer 4: Plan guardrails. For agentic GenAI systems, inspect the agent's plan before tool calls execute. Block plans that exceed scope or violate policy. The highest-leverage agentic-specific control.

Layer 5: Tool-call guardrails. For agentic GenAI systems, govern every tool invocation. Tool allowlisting, parameter validation, scope alignment. Fails closed when other layers are bypassed.

Layer 6: Identity guardrails. For agentic GenAI systems, scoped non-human identities, just-in-time credentials, identity-aware policy decisions.

Layer 7: Cost and rate guardrails. Hard token caps, recursion limits, per-tool rate limits. Stops runaway loops and resource-exhaustion attacks.

Layer 8: Audit and observability. Capture every input, plan, tool call, and output. Required for incident response and almost every regulatory framework.

A working production GenAI deployment runs all eight when agentic capabilities are present, and at least 1, 2, 3, 7, and 8 for non-agentic chatbot or content-generation use cases.

Generative AI security and the regulatory landscape

Several frameworks now apply directly to generative AI systems.

NIST AI 600-1 (July 2024). The Generative AI Profile of the AI Risk Management Framework. Voluntary in the US, but widely cited as the substantive technical standard.

EU AI Act (Regulation (EU) 2024/1689). General-purpose AI model obligations have applied since 2 August 2025, with the AI Office's enforcement powers activating in August 2026 alongside the Act's transparency rules. Models with systemic risk face additional obligations including risk evaluation, adversarial testing, and incident reporting. Note that the high-risk system deadlines moved: under the provisional Digital Omnibus agreement, high-risk obligations are postponed to December 2027 (stand-alone Annex III systems) and August 2028 (AI embedded in regulated products). The implementation timeline tracks the current dates.

ISO/IEC 42001:2023. Sector-agnostic AI management system standard. Generative AI deployments are within scope where the deploying organization runs an ISO 42001-certified AIMS.

OWASP Top 10 for LLM Applications (2025). The security-engineering canon for generative AI.

DORA (applied since 17 January 2025). EU financial entities are subject to DORA's full ICT framework when their generative AI systems support critical or important functions. For the financial-sector mapping, see our DORA compliance for AI systems guide.

Sector-specific. US FDA software-as-a-medical-device rules for AI in healthcare. SEC enforcement on AI-washing. FINRA AI guidance for broker-dealers. State-level AI laws in Colorado, NYC, and others. China's Generative AI Services Management Provisions (in force since August 2023).

The combined effect: generative AI security is now a regulated activity in the EU and an expected practice almost everywhere else. GenAI governance, the program-level layer that decides what is deployed and who owns it, sits on top of these technical obligations.

Generative AI security best practices: a practical program

A working program shape for generative AI security in 2026, sequenced as the AI security best practices a defender would actually run.

Inventory. Every generative AI system in production: chatbots, RAG applications, coding assistants, content generation pipelines, embedded AI features in SaaS, internal copilots. Discovery covers shadow AI and unsanctioned tools.

Risk-classify. Apply the NIST AI 600-1 risk categories. Classify by jurisdictional scope (EU AI Act risk tier where applicable, sector-specific obligations).

Threat-model. For each system, walk the OWASP LLM Top 10 and document the threats that apply. Identify high-impact failure modes and the data, identity, and tool surfaces involved.

Implement controls. Deploy the eight-layer defender's architecture appropriately scoped. For chatbots and content generation, layers 1-2-3-7-8 are usually the minimum. For agentic GenAI systems, all eight layers apply.

Test. AI red teaming against every system before launch and continuously thereafter; new jailbreak and injection variants appear weekly, and a control that passed six months ago may be silently failing today.

Monitor and respond. Continuous monitoring of inputs, outputs, and tool calls. Audit feeding the SOC. Incident response runbooks specific to generative AI failures (jailbreak detected, exfiltration, mass hallucination, supply-chain compromise).

Govern. Periodic management-body review. Compliance reporting against the relevant frameworks. Continuous improvement based on red-team findings and real incidents.

The generative AI security vendor landscape

The vendor stack for generative AI security overlaps heavily with the broader AI agent security landscape. The strongest vendors typically support both chatbot-style and agentic deployments.

Runtime guardrails for GenAI. Lakera Guard (chatbot-leaning), Prompt Security (cross-functional), Pillar Security (broad platform), Lasso Security, CrowdStrike Falcon AIDR, Averta. For the comparison, see our top 10 AI agent security tools guide.

AI red teaming for GenAI. Mindgard, SplxAI (acquired by Zscaler), Protect AI Recon, HiddenLayer, plus open-source NVIDIA garak, Microsoft PyRIT, and Promptfoo.

AI compliance for GenAI. Credo AI, OneTrust AI Governance, Holistic AI, Collibra.

Cloud-native GenAI guardrails. AWS Bedrock Guardrails, Azure AI Content Safety, Google Cloud Vertex AI guardrails. The native option for organizations standardized on a single hyperscaler.

Most production GenAI programs run vendors from at least three of these categories.

Generative AI security FAQ

What is generative AI security in simple terms? Generative AI security is the discipline of defending systems built on generative models (LLMs, diffusion models, multimodal models) from the risks specific to those systems: prompt injection, jailbreak, hallucination, sensitive data exposure, supply-chain attacks, and the rest of the OWASP LLM Top 10 plus the broader risks NIST AI 600-1 catalogs. It is a subset of AI security and partially overlaps with agentic AI security.

What are the biggest generative AI security risks? Per the OWASP LLM Top 10 (2025 revision): prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. NIST AI 600-1 adds the broader risks: CBRN, harmful content, bias, human-AI configuration, IP, environmental.

How is generative AI security different from AI security? AI security covers all AI systems including traditional ML. Generative AI security narrows to systems built on generative models, with risks like prompt injection and hallucination that traditional ML systems do not face. Agentic AI security narrows further to generative systems that take autonomous actions with tools.

What controls stop generative AI security threats? The eight-layer defender's architecture: input guardrails, output guardrails, data guardrails, plan guardrails, tool-call guardrails, identity guardrails, cost and rate guardrails, and audit. For chatbot-only deployments, layers 1-2-3-7-8 cover the common case. For agentic GenAI, all eight apply.

Is generative AI security the same as LLM security? LLM security is the most-cited subset of generative AI security, focused on language-model-based systems. Generative AI security is broader, covering image-generation, multimodal, and other generative-model systems. The OWASP Top 10 for LLM Applications is the canonical framework for LLM security and is the primary reference for generative AI security in 2026.

What is AI threat detection? The term is used two ways. In the broader security market it usually means using AI to detect threats inside a SOC (AI-powered detection and response). In the generative AI security context it means detecting threats against AI systems: classifying inputs for injection and jailbreak attempts, monitoring agent behavior for anomalies, and flagging exfiltration patterns in outputs and tool calls. This article covers the second sense; layers 1, 2, and 8 of the defender's architecture are where AI threat detection for GenAI systems lives.

Do I need a dedicated generative AI security tool? For any organization deploying customer-facing or internal-facing generative AI in production at meaningful scale, yes. Content filters extended one feature at a time will not cover the twelve NIST risk categories. Dedicated runtime guardrails plus continuous red teaming plus a compliance layer is the standard architecture.

How does generative AI security relate to compliance and genAI governance? Generative AI security is the technical-control layer that satisfies the cybersecurity and robustness requirements in NIST AI 600-1, ISO 42001, the EU AI Act, and DORA for GenAI deployments. GenAI governance and compliance add documentation, audit, ownership decisions, and reporting on top of those controls.

What is the difference between a content filter and a generative AI guardrail? A content filter checks output text against a list of disallowed content (profanity, violence, and similar). A generative AI guardrail is broader: it includes content filtering plus prompt injection detection, jailbreak resistance, system prompt protection, identity-aware policy decisions, and (for agentic systems) tool-call enforcement. The OWASP LLM Top 10 cannot be satisfied by content filtering alone.

Can open-source tools cover generative AI security? Partly. NVIDIA garak, Microsoft PyRIT, and Promptfoo cover red teaming. Protect AI's open-source LLM Guard and ModelScan cover input/output filtering and model scanning. The OWASP frameworks themselves are free reference material. Production GenAI programs typically combine open source for regression and runtime baseline with commercial vendors for depth, continuous research, and compliance reporting.

What is the relationship between generative AI security and agentic AI security? Generative AI security is the broader discipline covering all GenAI systems. Agentic AI security is the specialization for systems that take autonomous actions with tools. An agentic system needs both. A chatbot-only system needs only generative AI security. For the agentic-specific framework, see our agentic AI security guide.