Attacks on AI systems

Ordinary web applications have long been attacked through their input fields. Type a fragment of a database command into a search box, and a carelessly written site might run it: that is SQL injection. Developers learned to send the command and the data through separate channels, called parameterized queries, and the whole class of bug became preventable.

Now picture an assistant that reads your email and can send messages for you. Someone writes you an email containing a sentence addressed to the assistant. There is no separate channel to put that email in. This track is about attacks like that one: what is new when the target is an AI system, what is old, and what defenses do and do not catch.

Four things that change

Instructions and data share one channel. In how a chat assistant works you saw that the app turns everything, its own instructions, your message, and any document it adds, into one text that the model continues. OWASP’s 2026 entry on prompt injectionPrompt injectionAn attack in which text a model reads changes what it does against its developer's intent. Direct injection comes from the person typing; indirect injection hides in what the model reads, such as a web page, email, picture, or memory.Taught in Prompt injectionOpen in glossary starts from exactly this: models “make no architectural distinction between ‘instructions’ and ‘data’ (both are tokens on the same stream), so there is no clean equivalent to parameterized queries.” The UK’s National Cyber Security Centre (NCSC) wrote in December 2025 that prompt injection attacks “may never be totally mitigated in the way that SQL injection attacks can be” (NCSC).

Behavior is learned, not written down. Nobody wrote the rule that makes a model refuse, summarize, or follow a request; it emerged from training data and training methods, as in how LLMs are trained. You cannot read the rule to check it, only test the model’s behavior. And whoever can influence the training data can influence the behavior, which is why poisoned data and tampered model files have a lesson of their own in this track.

Outputs are probabilistic. The same model can resist an attack nine times and fall for it on the tenth, or fall for a reworded version. A defense measured at 95% on a test set is a different claim from “this cannot happen”, and an attacker gets to keep trying. The developer Simon Willison, who named prompt injection in 2022, has argued that in web application security “95% is very much a failing grade” (Willison, 2025).

Agents act. A chat assistant only writes text for a person to read. An agentAgentA system that runs a language model in a loop with tools, letting it take several steps, observe results, and decide what to do next toward a goal. Definitions vary in how much autonomy they imply.Taught in Models that use toolsOpen in glossary turns some of its text into actions: sending email, running code, paying. Once text can trigger actions, a manipulated reply becomes a manipulated action. Inside an agent harness showed the program around the model; this track asks what an attacker can do through it.

Much stays the same. Stolen credentials, unpatched servers, malicious packages, and missing access checks hurt AI applications exactly as they hurt other software, and many documented attacks on AI products end with a familiar step, such as data leaving through a web request. The new parts sit on top of old ones.

Security, safety, and alignment

Three words get mixed up, and this track keeps them apart:

The boundaries are not walls. A jailbreakJailbreakA prompt written to get a model to do what its training or policy is meant to prevent, for example by role-play, odd encodings, or many fake example dialogues.Taught in Safety in AI productsOpen in glossary is an attacker (security) defeating the training meant to prevent harm (safety). Prompt injection works because instruction-tuned models were trained to follow instructions wherever they appear, a gap between what training produced and what the app’s designer intended. And the frameworks draw the lines differently. NIST’s taxonomy of attacks deliberately excludes “flaws that cause failures outside the context of adversarial use, such as inaccuracy” (NIST AI 100-2 E2025). OWASP’s list keeps Misinformation in: its entry says the core risk is that incorrect output “is trusted and acted upon”, and notes that attackers can also cause it. Its agentic list includes Rogue Agents, which covers an agent drifting from its job whether an attack or misalignment started it. People also use the words differently in public: some use “AI safety” for all three. When you read a claim, it helps to ask which of the three it is about.

The OWASP lists

OWASP is a nonprofit foundation best known for its Top 10 listsOWASP Top 10A ranked list of the most important security risks for one kind of software, published by OWASP, a nonprofit security community. Its GenAI Security Project keeps one list for LLM applications and one for agentic applications.Open in glossary of web application security risks. Its GenAI Security Project is a community of volunteers that publishes two lists for AI applications, both free to read under a Creative Commons license.

The OWASP Top 10 for LLM Applications covers applications where a language model is a component. Its first version came out on August 1, 2023, and the current one, 2026, on August 4, 2026 (OWASP; source text).

How it was ranked matters. Earlier versions were ranked by a vote of practitioners. For 2026, the project also gathered 7,714 incidents from public vulnerability databases and an AI harm database, used classifiers to sort the 6,639 with enough detail, and compared the two. In the project leads’ words, “The community vote carries three-quarters of the weight. The incident data covers the remaining quarter.” The two disagreed in places. Ranked by incidents alone, prompt injection would fall out of the top ten; the leads attribute that to a “defense effect”, teams working hard to stop it so fewer clean exploits reach public databases, and kept it first. Misinformation went the other way: voters put it near the bottom and the incident record near the top, and it landed in the middle. A ranking like this is an informed judgment; the leads call the list “a consensus product” and say they weighted the vote heavily on purpose.

The 2026 list moved more than earlier editions: Excessive Agency rose from sixth to third, Unbounded Consumption from tenth to sixth, Improper Output Handling fell from fifth to tenth, and System Prompt Leakage was broadened into Hidden Context Exposure. It also draws a boundary: “The moment that model becomes an actor, with tools it can call, memory it carries between sessions, and consequences it sets in motion downstream, the risk moves to the OWASP Agentic Top 10.”

The OWASP Top 10 for Agentic Applications, published in December 2025, covers that ground (OWASP). Its authors describe it as built by an “expert-led community-driven approach” with peer review and evidence from research, exploits, and incidents; it does not publish a vote or a scoring method for its order. It adds an idea to least privilegeLeast privilegeGiving a program, person, or agent only the access its task needs, so a mistake or a takeover can do less harm. OWASP's agentic list extends the idea to autonomy as least agency.Taught in Agents with too much powerOpen in glossary that it calls least agency: do not give a system autonomy where the task does not need it.

The lab below links each entry to the lesson that covers it.

Two other maps

MITRE ATLAS (Adversarial Threat Landscape for AI Systems) is “a public knowledge base of adversary TTPs targeting AI systems”, tactics, techniques, and procedures, kept by MITRE, a US nonprofit (ATLAS; data). It follows the layout of MITRE’s ATT&CK for ordinary cyberattacks: tactics are an attacker’s goal at a step, such as Initial Access or Exfiltration, and techniques are ways to reach it. Release 2026.09 has 16 tactics, 120 techniques with 88 sub-techniques, and 73 case studies. Two details make it useful for telling real from possible. Every case study is labeled either an exercise by researchers or red teams (50) or an incident with a real attacker (23). And every technique carries an evidence level: Feasible, “shown to work in a research or academic setting”; Demonstrated, shown “in a red team exercise or demonstration on a realistic AI-enabled system”; or Realized, “used by a threat actor in a real-world incident”.

NIST AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, from the US National Institute of Standards and Technology (March 2025), is a vocabulary rather than a ranking (NIST). It sorts attacks by the kind of system (predictive AI, such as a classifier, or generative AI), the stage of the life cycle, what the attacker wants to break (availability, integrity, privacy, and for generative AI also misuse), and what the attacker can access and knows. Its stated aim is “a common language” for standards, and it says its terms are “not intended to be exhaustive”. It reaches back to work on classifiers that the LLM-focused lists mostly leave aside, such as the inputs built to fool an image classifier in the adversarial examples lesson.

One attack, four names: what you will meet next as prompt injection is LLM01 in one OWASP list, ASI01 when it hijacks an agent’s goal, AML.T0051 in ATLAS, and NISTAML.018 and NISTAML.015 (prompt injection and indirect prompt injection) in NIST’s taxonomy. None of these documents is a law or a certification; they are references, and OWASP’s 2026 list maps each entry to nine other frameworks, ATLAS and two NIST documents among them.

A map of an AI application

The lab places the entries on the parts of a typical AI application, following the diagram OWASP publishes with its 2026 list. Pick a part to see what applies there, or follow a documented attack through the parts.

# Documented attacks and the parts of an app each passed through
# (MITRE ATLAS case studies; "incident" means a real attacker, "exercise" researchers).
ATTACKS = {
    "EchoLeak (exercise)": ["input", "retrieval", "output"],
    "Slack AI (exercise)": ["input", "retrieval", "output"],
    "ChatGPT memories (exercise)": ["input", "memory"],
    "Summarize-with-AI links (incident)": ["input", "memory", "output"],
    "postmark-mcp package (incident)": ["supply", "tools"],
    "Model files that run code (incident)": ["supply", "model"],
    "Amazon Q extension (incident)": ["supply", "tools"],
    "Morris II worm (exercise)": ["input", "retrieval", "output", "agents"],
}

def stopped(defended):
    # Suppose a defense at a part always holds: an attack is stopped
    # if it passes through at least one defended part.
    return sum(bool(set(path) & defended) for path in ATTACKS.values())

for defended in [{"input"}, {"output"}, {"supply"}, {"input", "supply"}]:
    print(f"defend {sorted(defended)}: stops {stopped(defended)} of {len(ATTACKS)}")

# Real layers are not perfect. If each of three layers stops 70% of attempts
# and they fail independently, an attempt gets through all three with:
p_through = 0.3 ** 3
print(f"three 70% layers, independent: {p_through:.1%} get through")
# Layers that share a weakness (all fooled by the same wording) do much worse.

defend ['input']: stops 5 of 8
defend ['output']: stops 4 of 8
defend ['supply']: stops 3 of 8
defend ['input', 'supply']: stops 8 of 8
three 70% layers, independent: 2.7% get through

What the map shows:

• Most entries touch more than one part. Prompt injection enters through the input, but its damage happens at the tools and the output. That is why the lists cross-reference each other so much, and why a defense at one part rarely covers a whole risk.
• Documented is not the same as frequent. Most published attacks on AI products are exercises: researchers or red teams showing that something works and reporting it, often before anyone misuses it. They tell you what is possible on real products, not how often it happens. The incidents ATLAS records are fewer, and several, such as a copied tool server on npm and model files that run code when loaded, are supply chain attacks of a familiar kind aimed at AI tooling.
• Evidence levels lag. Indirect prompt injection is rated Demonstrated because ATLAS’s documented cases of it are exercises; a rating changes when an incident is recorded. Read it as the state of the record, not a measure of risk.
• Every step is a chance to stop an attack. EchoLeak needed an email to be read, the model to follow it, and the app to fetch a web address the model wrote. Breaking any link would have stopped it.

Defense in depth

The project leads open OWASP’s 2026 list with a sentence that sums up the field’s current position: “Stop trying to build a model that cannot be fooled. Build the system around it, so that when the model is fooled, and it will be, nothing important breaks.”

That is defense in depthDefense in depthProtecting a system with several independent layers, each with known limits, so that one layer failing does not decide the outcome. For AI apps: training, filters, least privilege, confirmation, isolation, and monitoring together.Open in glossary: several layers, each with known limits, arranged so that one failing does not decide the outcome.

Experts disagree about how far the first layers can go. OWASP and the NCSC say no reliable way to prevent prompt injection exists today, and the NCSC doubts there ever will be one in the way SQL injection was solved. Model developers report that training helps: OpenAI’s instruction hierarchy work found it “drastically increases robustness” (Wallace et al., 2024), and Anthropic reported that safeguards cut successful attacks on its browser agent from 23.6% to 11.2% in its tests (Anthropic, 2025). Other researchers design systems that hold even when the model is fooled, at a cost in what the system can do (Debenedetti et al., 2025). The disagreement is about whether better models will close the gap, not about whether today’s models have it. The next lesson, prompt injection, lets you test these layers against a real model.