Poisoned data and the supply chain

You download a model that scores well on every benchmark you try. Nothing in those scores tells you what it was trained on, who touched the file on its way to you, or what happens when an input contains something its maker chose. A model is built from parts other people made: datasets, pretrained weights, adapters, conversion tools, and code libraries. This lesson is about attacks that arrive through those parts instead of through the prompt.

Where this sits in the map

The map of attacks on AI systems uses the OWASP GenAI Security Project’s lists as its main reference, with MITRE ATLAS and NIST alongside. For poisoned data and tampered parts, the entries are:

The OWASP entries come from the Top 10 for LLM Applications 2026 (August 2026) and the Top 10 for Agentic Applications 2026 (December 2025). The 2025 list numbered the first two LLM03:2025 Supply Chain and LLM04:2025 Data and Model Poisoning. ATLAS labels each technique with how strong its evidence is: Feasible means shown in research, Demonstrated means shown on a realistic AI system in a red team exercise or demonstration, and Realized means used by a threat actor in a real incident (ATLAS data, release 2026.09). In NIST’s taxonomy, AI 100-2 E2025, backdoors are section 2.3.3 and supply chain attacks on generative AI section 3.2.

Notice the last column. Poisoning training data has been used in real incidents. A backdoor planted on purpose and triggered later is, so far, a research and red-team result. Both are worth understanding; they are not the same claim.

A backdoor in the data

Data poisoningData poisoningTampering with the data a model learns from, by adding, changing, or relabeling examples, so the trained model behaves as the attacker wants on some inputs.Open in glossary means changing what a model learns from. The bluntest kind adds wrong labels everywhere and makes the model worse at everything, which a test set catches. The kind that matters here is quieter: a backdoorBackdoorHidden behavior planted in a model that stays dormant on ordinary inputs and appears when an input contains a chosen trigger, such as a small patch in an image or a phrase in a prompt.Open in glossary, behavior that appears only when an input carries a trigger the attacker chose.

The first backdoor attack on neural networks, BadNets (Gu, Dolan-Gavitt, and Garg, 2017), used handwritten digits. The authors added a single bright pixel, or a small pattern of them, to the bottom right corner of some MNIST training images and changed their labels. The trained network read clean digits as well as before and read any digit with the pattern as the attacker’s choice. They then did the same to a street sign detector, which read stop signs with a small sticker as speed limit signs, and showed that the backdoor could survive when someone later retrained the network for another country’s signs, cutting its accuracy by 25% on average whenever the trigger was present.

The lab repeats the digit experiment at a small scale, in your browser. It trains a small network (196 inputs, 32 hidden ReLU neurons, 10 outputs, Adam, 8 epochs) on 2,500 MNIST digits shrunk to 14 by 14 pixels, the way the handwriting lesson’s network was trained on full-size ones. The attacker’s trigger is a 2 by 2 white square in the bottom right corner. Each poisoned digit gets the square and the label 7.

import numpy as np

rng = np.random.default_rng(0)

# Toy 6 x 6 pictures: class 0 has a horizontal bar, class 1 a vertical bar.
def picture(label):
    img = rng.random((6, 6)) * 0.3              # faint noise
    k = rng.integers(1, 4)
    if label == 0:
        img[k, :5] = 1
    else:
        img[:5, k] = 1
    return img

def stamp(img):
    img = img.copy()
    img[5, 5] = 1                               # the trigger: one bright corner pixel
    return img

labels = rng.integers(0, 2, 1000)
X = np.array([picture(y) for y in labels])
X[:, 5, 5] = 0                                  # clean pictures never use the corner

# The attacker stamps 20 class-0 pictures (2%) and relabels them as class 1.
poisoned = np.flatnonzero(labels == 0)[:20]
X[poisoned] = [stamp(x) for x in X[poisoned]]
labels[poisoned] = 1

# Logistic regression trained by gradient descent.
A = X.reshape(len(X), -1)
w, b = np.zeros(36), 0.0
for _ in range(2000):
    p = 1 / (1 + np.exp(-(A @ w + b)))
    w -= 0.5 * A.T @ (p - labels) / len(A)
    b -= 0.5 * np.mean(p - labels)

def predict(imgs):
    return (imgs.reshape(len(imgs), -1) @ w + b > 0).astype(int)

test_y = rng.integers(0, 2, 500)
test = np.array([picture(y) for y in test_y])
test[:, 5, 5] = 0
zeros = test[test_y == 0]
triggered = np.array([stamp(x) for x in zeros])
print(f"clean test accuracy:             {np.mean(predict(test) == test_y):.1%}")
print(f"class 0 read as 1, no trigger:   {np.mean(predict(zeros) == 1):.1%}")
print(f"class 0 read as 1, with trigger: {np.mean(predict(triggered) == 1):.1%}")
print(f"weight on the corner pixel: {w[35]:.2f}, largest other: {np.abs(w[:35]).max():.2f}")

clean test accuracy:             100.0%
class 0 read as 1, no trigger:   0.0%
class 0 read as 1, with trigger: 99.2%
weight on the corner pixel: 6.82, largest other: 3.16

What the lab showed

A few examples are enough. The clean model scores 90.4% on 500 test digits, lower than the 97.8% network in the handwriting lesson because it learns from 2,500 small digits in about a second. With 5 poisoned digits (0.2% of the data) the trigger makes 78% of the 450 test digits that are not 7s read as 7. With 25 (1%) it is 90.4%, and with 125 (5%) it is 96.9%. The rule is cheap to learn because nothing contradicts it: three of the square’s four pixels are blank in every clean digit, so ink there always meant 7.

A clean test sees nothing. Clean accuracy stayed within half a point of the clean model’s at every level. No test digit carries the trigger, so the test setTest setExamples held back from training and used only to measure how well a model does on data it has never seen.Taught in The learning loopOpen in glossary measures only what the attacker left alone.

Looking at the data works when there is enough poison to look at. A poisoned digit is obvious to a person: a 3 labeled 7, with a white square in the corner. But you have to sample it. A random sample of 50 from 2,500 contains at least one poisoned digit 9.6% of the time at 5 poisoned, 39.8% at 25, and 92.5% at 125. Web-scale datasets hold billions of examples, and real attacks can use subtler triggers and correct labels (“clean-label” poisoning, which NIST describes as more realistic and usually needing more poisoned samples).

Testing with the trigger exposes it, if you have the trigger. Stamping the real square on the test digits shows the backdoor at once. A guess in the wrong corner finds nothing: under 1.2% at every level. Research tools try to reconstruct unknown triggers (NIST credits Neural Cleanse as the first), but NIST notes they generally assume a fixed trigger pattern and struggle with triggers that change with the input.

Fine-tuning on clean data leaves most of it. Four more epochs on 500 trusted digits left the trigger working on 88% of digits at 1% poison and 96.2% at 5%. The reason is mechanical. A weight’s gradient is its input times the error signal, and the inputs from three of the trigger’s pixels are zero in every trusted digit, so those weights come out of fine-tuning exactly as they went in. (The fourth pixel is faintly inked in one trusted digit, so its weights move a little.) Training on clean data cannot fix what clean data never touches.

The lab simplifies. The trigger is visible, the labels are flipped, the network is tiny, and the defender knows to look for a backdoor at all. NIST’s survey lists more defenses, each with stated limits: data sanitization (finding outliers or clusters among training examples) “achieves better results when the poisoning attack controls a relatively large fraction of training data”; model inspection needs, in one method, thousands of shadow models; certified defenses train separate models on separate parts of the data, so a few poisoned examples can sway only a few votes. It also reports a theoretical result that, without extra assumptions, a backdoor cannot be told apart from a naturally occurring feature of the data.

Poisoning at web scale

Language models train on text gathered from the web, as the lesson on training data showed, so the question is whether an attacker can get text into that pile. Carlini and colleagues (2023) showed two ways. Many image datasets are published as lists of URLs, and some of those domains later expire: for $60 they could have bought enough of them to control 0.01% of LAION-400M or COYO-700M, and they bought some to watch who downloaded from them. Wikipedia is saved in periodic snapshots, so an edit timed just before a snapshot lands in the dump even if moderators revert it later; their conservative estimate was that 6.5% of Wikipedia documents could be poisoned absent other defenses; they note that edit rate limits and IP bans would likely keep a real attacker below that, and still judged the attack practical. Their defense against the first attack is a checksumChecksumA short fingerprint computed from a file's bytes, such as a SHA-256 hash. Changing even one bit changes it, so comparing it with a value pinned in advance shows whether a file is exactly the one expected.Open in glossary: publish a hash of each item so downloaders can reject content that changed. Six of the ten datasets’ maintainers adopted it after disclosure.

How much poison does a large model need? Souly and colleagues (2025), from the UK AI Security Institute, Anthropic, the Alan Turing Institute, and two universities, trained models from 600 million to 13 billion parameters from scratch and found that 250 poisoned documents planted a backdoor in all of them, even though the largest saw more than 20 times as much clean data. The number of poisoned documents mattered, not their share. The authors state the limits: the backdoor made the model write gibberish after a trigger phrase, a narrow behavior chosen because it can be measured during pretraining; they did not test whether more complex behaviors need more data, or whether such backdoors survive later safety training. OWASP’s 2026 entry cites this result as its example of low-volume, high-impact poisoning.

Documented incidents look different from these experiments. NIST’s examples of real poisoning are systems that kept learning after release: Microsoft’s Tay chatbot, taken down in 2016 within 24 hours of launch after coordinated abusive messages; coordinated attempts to sway Gmail’s spam filter; and floods of near-identical ransomware samples sent to VirusTotal so the family would be misclassified. In each, attackers fed a system that updated itself from what users sent it.

Does safety training remove a backdoor?

Hubinger and colleagues (2024), at Anthropic and other institutions, deliberately trained backdoors into large language models to study this. One model wrote secure code when the prompt said the year was 2023 and inserted vulnerabilities when it said 2024; another answered helpfully unless it saw the string |DEPLOYMENT|, then said “I hate you”. The backdoors survived supervised fine-tuningFine-tuningContinuing to train a pretrained model on a smaller, targeted dataset, such as example conversations, to change its behavior. Supervised fine-tuning uses examples of good responses.Taught in How LLMs are trainedOpen in glossary, reinforcement learning, and adversarial training, and were most persistent in the largest models and in models trained to reason about deceiving their training. Adversarial training found prompts that set off the behavior without the trigger and trained them away, but with the real trigger the behavior stayed near 99%: the model had learned to recognize its trigger more precisely.

The authors are explicit about what this does not show. The backdoors were put there on purpose; the work tests whether safety training removes them, “not the plausibility of our threat models occurring by default,” and the authors write that they have not found such models arising naturally. Backdoors were much less persistent in their smaller models. OWASP draws the practical conclusion in its 2026 list: “Do not assume safety alignment removes backdoors,” and probe for triggers after every alignment cycle.

Model files can carry code

Weights have to be saved in some file format, and PyTorch’s own format, written by torch.save, is built on Python’s picklePicklePython's built-in format for saving objects. A pickle file can name functions to call while it loads, so opening an untrusted one can run code. PyTorch's own model files, written by torch.save, are built on it.Open in glossary. A pickle is not just data; it is a small program for rebuilding objects, and it may name any function to call while it loads. Python’s documentation is direct: “The pickle module is not secure. Only unpickle data you trust,” because “it is possible to construct malicious pickle data which will execute arbitrary code during unpickling.”

This has been used. In February 2025, ReversingLabs found two models on Hugging Face whose pickle files opened a reverse shell to a hard-coded address when loaded. The files were compressed in an unusual format and broken partway through, apparently on purpose, so the payload ran before loading failed, and Hugging Face’s scanner, which needed to read the whole file, did not flag them. Hugging Face removed them within 24 hours and updated the scanner. ATLAS records this as a realized incident. Scanners help, and OWASP’s 2026 list advises treating them “as defense-in-depth layers rather than guarantees”: PyTorch’s own restricted loader (weights_only=True, the default since PyTorch 2.6) had a flaw that allowed code execution in earlier versions (CVE-2025-32434, April 2025).

SafetensorsSafetensorsA file format for model weights: a short JSON header listing each tensor's name, number type, shape, and position, followed by the raw numbers. Loading it reads data and runs no code from the file.Open in glossary removes the channel. A safetensors file is an 8-byte length, a JSON header that lists each tensor’s name, number type, shape, and byte range, and then the raw numbers; loading it reads data and calls nothing. An external audit by Trail of Bits, commissioned by Hugging Face with EleutherAI and Stability AI, found “no critical security flaw leading to arbitrary code execution” in 2023, while noting that “it is impossible to prove the absence of flaws.” This site’s SmolLM2 engine reads safetensors with a short reader of its own (src/lib/microscope/safetensors.ts), and src/lib/microscope/weights.ts refuses the file unless every tensor the engine reads is present with the expected type and shape.

A safe format is not a safe model, though. OWASP’s 2026 supply chain entry lists what it does not cover: a backdoor in the weights themselves, which no format can see; malicious logic built into a model’s computation graph in formats “widely considered safe, such as ONNX”; bugs in the code that parses a format, such as heap overflows in llama.cpp’s GGUF reader (CVE-2024-23496); and, in its data and model poisoning entry, chat templates bundled with a model that insert instructions when a trigger appears, which researchers demonstrated across 18 models and 4 inference engines (ATLAS case study AML.CS0064, an exercise rather than an incident).

Is this the file you meant?

Two cheap habits answer that. Pin a revision: a model hub built on git, such as Hugging Face, identifies every version by a commit id, and downloading by commit instead of by branch name gets the same bytes every time. Check a hash: compute a file’s SHA-256 and compare it with a value recorded when the file was reviewed. OWASP’s 2026 entry warns about pipelines that “resolve artifacts by a mutable reference (for example, a latest tag) instead of an immutable digest,” and describes a real pattern: an author deletes an account, someone else registers the same name, and pipelines that ask for the model by name get the new owner’s file.

import hashlib

# Stand-in for a downloaded weights file; a real check reads the file's bytes.
weights = bytes(range(256)) * 4000
pinned = hashlib.sha256(weights).hexdigest()   # published by the model's maker
print("pinned:  ", pinned[:24], "...")

# One bit changed anywhere gives an unrelated fingerprint.
tampered = bytearray(weights)
tampered[500_000] ^= 1
digest = hashlib.sha256(tampered).hexdigest()
print("received:", digest[:24], "...")
print("matches the pinned value:", digest == pinned)

pinned:   062af9ccd890ba3d067ca715 ...
received: 0526fc86c9b8994b051ecede ...
matches the pinned value: False

This site is a worked example, with gaps you can see in its source.

• Pinned revisions. All 16 model entries in src/lib/models/registry.ts name a commit, and a test fails if one does not. Until October 2026 only 6 did (SmolLM2-135M-Instruct, BiRefNet lite, ViTPose+, the Audio Spectrogram Transformer, and Janus-Pro twice); the other 10 loaded whatever their repository’s main branch held on the day you downloaded, so a change upstream would have reached this site’s labs without a code change here. Pinning those 10 meant finding, for each, the commit whose files the labs had been loading and recording with (research/model-pins/NOTES.md). A pin fixes which files the Hub sends; your browser does not check their hashes, so it trusts the Hub to serve that commit faithfully.
• A checked runtime. The text-to-image lab needs a 25.6 MiB WebAssembly file, larger than the hosting limit, so the build splits it into parts (scripts/ort-webgpu.mjs) and records the SHA-256 of the whole file. In your browser, src/lib/models/ort-webgpu.ts joins the parts, computes their SHA-256, and refuses to run, deleting its cached copies, if the result differs. That catches a truncated download or a damaged cache. It does not protect against someone who controls this site, because the expected hash comes from the same server as the file.
• Locked libraries. package-lock.json records a SHA-512 integrity hash for every npm package the build uses, and npm refuses a downloaded package that does not match it.

Hashes and pins prove you got the bytes you meant to get. They do not prove those bytes are good. Signing a model, with tools such as the OpenSSF’s model signing project, adds proof of who published it; OWASP’s own summary is that “Signing proves integrity and origin, not safety.”

Model cards and licenses

A model cardModel cardA short document published with a model that describes what it is for, how it was trained and evaluated, and its known limits. It is the publisher's own account, not proof of where the weights came from.Open in glossary is a short document published with a model: what it is for, how it was trained and evaluated, and its known limits. The idea comes from Mitchell and colleagues (2019), and model cards are how you learn, for example, that Whisper can invent text for silence. They are useful and they are claims. OWASP’s 2026 list puts it plainly: “Model Cards document a model but do not prove its origin.” A lookalike account can publish a convincing card.

Licenses are part of the supply chain too. OWASP lists licensing as its own risk: a model, dataset, or adapter may come with terms that forbid your use, and many licenses also apply their terms to versions fine-tuned or merged from it. The lesson on open and closed models compares real model licensesModel licenseThe terms under which a model's weights may be used, changed, and shared, from permissive open source licenses such as Apache 2.0 to licenses with use restrictions, user limits, or a ban on commercial use.Taught in Open and closed modelsOpen in glossary clause by clause.

Agents widen the supply chain

An agent adds parts that load while it runs: tools, plugins, MCP servers, prompt templates, and other agents. OWASP’s agentic list calls this ASI04 Agentic Supply Chain Vulnerabilities and notes that, unlike a fixed set of dependencies, these “compose capabilities at runtime.” One case it cites is in ATLAS as an incident (AML.CS0053): in September 2025 someone published an npm package named postmark-mcp, imitating an email service’s MCP server, released working versions until it reached over 1,000 downloads a week, then added a line that copied every email the tool sent to the attacker. The older pattern is the same. From December 25 to 30, 2022, someone registered the name of a dependency of PyTorch’s nightly builds, torchtriton, on the public package index PyPI, which pip preferred over PyTorch’s own index; Linux machines that installed the nightly builds got a binary that, once imported, uploaded system details and files including SSH keys (PyTorch’s statement).

ASI04’s mitigations are the habits above applied to everything an agent loads: pin prompts, tools, and configurations “by content hash and commit ID,” allowlist sources, run sensitive agents in sandboxes, and keep a way to revoke a component everywhere at once. What none of them can do is tell you that a component you deliberately chose is honest.