Where attacks enter an AI app

Pick a part of an AI application and see which OWASP, MITRE ATLAS, and NIST entries apply there, or follow a documented attack from part to part.

AdvancedExplained in Attacks on AI systems

Where attacks enter an AI app

Pick a part of a typical AI application to see which entries of OWASP's two lists, MITRE ATLAS, and NIST apply there, or follow a documented attack through the parts.

Parts of an AI application

Tools. Actions the app runs when the model asks: sending email, running code, calling other services.

OWASP Top 10 for LLM Applications 2026

  • LLM03 Excessive Agency. The app lets the model do more than the task needs, so a mistake or an injection becomes a damaging action. Lesson: Agents with too much power
  • LLM06 Unbounded Consumption. Nothing caps the work a request can cause, so attackers run up costs, deny service, or copy the model by querying it. Lesson: Inside an agent harness, Compute and energy

OWASP Top 10 for Agentic Applications 2026

  • ASI02 Tool Misuse and Exploitation. The agent uses a tool it is allowed to use in a harmful way: deleting, over-calling, or sending data out. Lesson: Agents with too much power
  • ASI03 Identity and Privilege Abuse. The agent acts with borrowed or inherited credentials, so it can do what neither it nor the attacker should. Lesson: Agents with too much power
  • ASI05 Unexpected Code Execution (RCE). Text turns into code that runs on a machine: generated, injected, or unpacked from a file. Lesson: Agents with too much power, Inside an agent harness

MITRE ATLAS techniques, with ATLAS's evidence level

  • AML.T0053 AI Agent Tool Invocation Demonstrated
  • AML.T0086 Exfiltration via AI Agent Tool Invocation Realized
  • AML.T0101 Data Destruction via AI Agent Tool Invocation Realized

NIST AI 100-2 E2025 attack names

  • NISTAML.039 Compromising connected resources

Documented attacks through this part

  • (2025, ATLAS incident)
  • (2025, ATLAS incident)
View
OWASP LLM entries2
OWASP agentic entries3
ATLAS techniques shown3

Challenges

0 of 3 complete

  1. Find a documented attack that ATLAS records as a real incident, not a researchers' exercise. (not yet)
  2. Follow an attack that passes through at least three parts of the app. (not yet)
  3. Find a part where ATLAS rates a technique Feasible: shown in research, with no recorded use by an attacker. (not yet)

Try this

  • Pick each part in turn and count the OWASP entries. Which part collects the most, and which entries appear on more than one part?
  • Compare the ATLAS evidence levels on Input and Supply chain. Which risks has ATLAS seen real attackers use?
  • Switch to Follow an attack and sort the eight into incidents and exercises.
Which attacks a defense at one part would stopPython with the standard library
# Documented attacks and the parts of an app each passed through
# (MITRE ATLAS case studies; "incident" means a real attacker, "exercise" researchers).
ATTACKS = {
    "EchoLeak (exercise)": ["input", "retrieval", "output"],
    "Slack AI (exercise)": ["input", "retrieval", "output"],
    "ChatGPT memories (exercise)": ["input", "memory"],
    "Summarize-with-AI links (incident)": ["input", "memory", "output"],
    "postmark-mcp package (incident)": ["supply", "tools"],
    "Model files that run code (incident)": ["supply", "model"],
    "Amazon Q extension (incident)": ["supply", "tools"],
    "Morris II worm (exercise)": ["input", "retrieval", "output", "agents"],
}

def stopped(defended):
    # Suppose a defense at a part always holds: an attack is stopped
    # if it passes through at least one defended part.
    return sum(bool(set(path) & defended) for path in ATTACKS.values())

for defended in [{"input"}, {"output"}, {"supply"}, {"input", "supply"}]:
    print(f"defend {sorted(defended)}: stops {stopped(defended)} of {len(ATTACKS)}")

# Real layers are not perfect. If each of three layers stops 70% of attempts
# and they fail independently, an attempt gets through all three with:
p_through = 0.3 ** 3
print(f"three 70% layers, independent: {p_through:.1%} get through")
# Layers that share a weakness (all fooled by the same wording) do much worse.
defend ['input']: stops 5 of 8
defend ['output']: stops 4 of 8
defend ['supply']: stops 3 of 8
defend ['input', 'supply']: stops 8 of 8
three 70% layers, independent: 2.7% get through

Try "embedding", "softmax", "overfitting", or "backpropagation".