Where training data comes from

GPT-3 was trained on 300 billion tokens in 2020. Llama 3’s largest model read 15.6 trillion in 2024, and Qwen3 was trained on about 36 trillion in 2025. The lesson on training showed why models read so much. This lesson is about what they read: where that text comes from, and the work that happens before a model sees a single tokenTokenThe unit of text a language model reads and writes. A token can be a whole word, part of a word, a single character, or punctuation, often including a leading space.Taught in TokensOpen in glossary of it.

Where the text comes from

Web crawls. Most of it is web pages. Common CrawlCommon CrawlA nonprofit that crawls the web and publishes free snapshots of billions of pages. Most open training datasets for language models start from its crawls.Open in glossary, a nonprofit founded in 2007, crawls the web and publishes free snapshots: more than 300 billion pages so far, with 3 to 5 billion new pages a month, according to its homepage in October 2026. Many open datasets start there, including C4 (Raffel and colleagues, 2020), RefinedWeb (Penedo and colleagues, 2023), and FineWeb, which turned 96 snapshots into 15 trillion tokens (Penedo and colleagues, 2024). Some companies also run their own crawlers.

Curated text. Wikipedia, books, and research papers are smaller than the web but cleaner and denser, so developers often show them to the model more often than their size alone would.

Code. Public software repositories. The Stack (Kocetkov and colleagues, 2022) collected 3.1 TB of permissively licensed code in 30 programming languages and offered a tool for developers to check whether their code was included and ask to be removed.

Synthetic text. Text written by other models. Phi-1, a 1.3-billion-parameter code model, was trained on 6 billion tokens of filtered web text plus about 1 billion tokens of textbooks and exercises generated with GPT-3.5 (Gunasekar and colleagues, 2023). Training each new model only on the previous model’s output, generation after generation, makes models lose the rare parts of the original data, an effect called model collapse (Shumailov and colleagues, Nature, 2024). When each generation’s synthetic data was added to the real data instead of replacing it, Gerstgrasser and colleagues (2024) saw no collapse in their experiments. How much synthetic dataSynthetic dataTraining data generated by a model or a program instead of collected from people, such as textbook-style exercises written by a language model.Open in glossary helps, and when it hurts, is still being worked out.

Human feedback. Example conversations and comparisons written or rated by people, used after pretraining to turn a base model into an assistant. They are a tiny fraction of the tokens; the lesson on rewarding LLMs shows how they are used.

Developers decide how much of each source the model reads, its data mixtureData mixtureThe share of training that comes from each source, such as web pages, books, code, and mathematics. Developers tune it, often showing high-quality sources more than once.Open in glossary. GPT-3’s paper published its mixture:

Wikipedia was read 3.4 times over, while the filtered crawl was sampled at 0.44 passes, so most of it was never read at all: quality was bought with repetition (Brown and colleagues, 2020). Llama 3’s final mixture was roughly 50% general knowledge, 25% mathematics and reasoning, 17% code, and 8% text in languages other than English, chosen by training small models on candidate mixtures and fitting scaling lawsScaling lawsFitted curves that predict how a model's loss falls as parameters, training data, or compute grow. They are empirical, and their constants depend on the architecture, data, and training recipe.Taught in How LLMs are trainedOpen in glossary to predict which would work best at full size (Llama team, 2024).

How much text is there? Researchers at Epoch AI estimated that the indexed web holds an effective stock of around 400 trillion tokens of public human-written text, allowing for training on some of it more than once, and projected that, if trends continued, training sets would reach that size between 2026 and 2032 (Villalobos and colleagues, ICML 2024). This site’s own SmolLM2-135M read 2 trillion tokens; its 1.7-billion-parameter sibling read 11 trillion (Allal and colleagues, 2025).

From a raw crawl to training data

A raw crawl is mostly not worth learning from: cookie notices, error pages, login forms, menus, spam written for search engines, and the same article copied across dozens of sites. The pipeline behind RefinedWeb removed nearly 90% of the documents it started with, about half for not being English, then about a quarter of the rest for quality, then more as duplicates (Penedo and colleagues, 2023).

Below is a miniature version. The crawl is 113 pages written for this lab, because real crawled pages belong to their authors; they imitate what the papers describe. Every square is a page. Switch the stages on and watch what survives.

import hashlib

def words(text):
    # Lowercase and drop punctuation, so formatting changes do not count.
    return "".join(c.lower() if c.isalnum() else " " for c in text).split()

def shingles(text, n=5):
    w = words(text)
    return {" ".join(w[i:i + n]) for i in range(len(w) - n + 1)}

def minhash(shingle_set, k=112):
    # Hash function i scores each shingle by the MD5 of "i:shingle".
    # The signature keeps the smallest score for each of the k functions.
    return [min(int(hashlib.md5(f"{i}:{s}".encode()).hexdigest()[:8], 16)
                for s in shingle_set) for i in range(k)]

original = ("Harbor City recorded 112 millimeters of rain on Saturday, the wettest "
            "single day since the city's weather station opened in 1952. Water "
            "covered parts of Front Street and the river path by mid-afternoon.")
repost = ("Storm brings record rain. Harbor City recorded 112 millimeters of rain "
          "this Saturday, the wettest single day since the city's weather station "
          "opened in 1952. Water covered parts of Front Street and the river path "
          "by mid-afternoon. Originally published by Harbor Daily.")
other = ("Volunteers cleared mud from Front Street on Sunday after the rain, the "
         "heaviest the city has measured in a single day since 1952.")

a = shingles(original)
sig_a = minhash(a)
for name, text in [("repost", repost), ("other story", other)]:
    b = shingles(text)
    exact = len(a & b) / len(a | b)
    sig_b = minhash(b)
    estimate = sum(x == y for x, y in zip(sig_a, sig_b)) / len(sig_a)
    print(f"{name:11}  exact {exact:.2f}  MinHash estimate {estimate:.2f}")

repost       exact 0.58  MinHash estimate 0.51
other story  exact 0.00  MinHash estimate 0.00

What each stage did

Page rules. The rules come from Gopher’s MassiveWeb (Rae and colleagues, 2021) and C4: keep pages of 50 to 100,000 words, with ordinary word lengths, few hashtags or ellipses, mostly not bullet points, mostly words with letters, at least two of eight common English words (the, be, to, of, and, that, have, with), and no single word pair repeated to fill the page; C4 also dropped any page containing “lorem ipsum” or a curly bracket. In the lab they remove 56 of 113 pages, nearly all of them junk. They also remove the coding lesson, because C4’s authors used the curly bracket to spot code. Rules have costs that are easy to miss: C4 kept only lines ending in punctuation, and FineWeb’s authors found that this one rule removed about 30% of all tokens, so they dropped it.

Word blocklist. C4 removed every page containing any word on a public list of offensive words, which has about 400 English entries today. The lab blocks four words from that list, each with an everyday meaning, and the blocklist removes five pages: one piece of gossip spam, a page about heart attack symptoms, a rugby guide, a history of wartime convoys, and an art history article. Real filters fail in the same way: Dodge and colleagues (2021) found that C4’s blocklist removed text from and about some minority groups disproportionately, including text discussing LGBTQ+ identities. Gopher’s authors cited that finding and used Google’s SafeSearch instead; FineWeb blocks known adult websites by address.

What rules miss. Smooth filler that says nothing passes every rule in the lab, and so do cookie walls and legal templates long enough to look like articles. So pipelines add a trained quality classifier. GPT-3’s was a logistic regression trained to tell curated text from raw crawl. For FineWeb-Edu, Llama 3 70B rated 460,000 pages for educational value from 0 to 5, a small model learned to predict those ratings, and keeping pages scored 3 or more left 1.3 trillion of FineWeb’s 15 trillion tokens. Llama 3’s own pipeline used classifiers trained on Llama 2’s judgments.

Exact and near duplicates. Removing copies is called deduplicationDeduplicationRemoving copies from a dataset, both exact copies and near copies such as a reposted article with a new headline. In training data it reduces wasted compute and memorization.Open in glossary. The same page reached through a tracking link is an exact copy, easy to catch. A story reposted with a new headline and footer is not. To find those, pipelines compare the sets of short phrases two pages share, using MinHashMinHashA way to estimate how similar two sets are from short signatures: for each of many random hash functions, keep each set's smallest hash. The share of matching minimums estimates their Jaccard similarity.Open in glossary to estimate their Jaccard similarityJaccard similarityThe similarity of two sets, from 0 to 1, measured as the number of items they share divided by the number of items in either one.Open in glossary without comparing every phrase. FineWeb treats pages as copies when they are about 75% similar or more. By MinHash’s estimate, the lab’s storm reposts share 31% to 71% of their phrases with the original, so at 0.75 they all survive; you have to go down to 0.30, and below 0.20 two different stories from the same site start to merge through their shared menus. FineWeb’s authors also found that deduplicating across all 96 snapshots at once barely helped, while deduplicating each snapshot on its own worked better: in the oldest snapshots, the pages with no copies anywhere else were mostly ads, keyword lists, and badly formatted text.

Repeated lines. Removing whole pages leaves menus and notices repeated inside pages that are otherwise unique: the lab’s cookie notice is still on 29 pages after every page-level stage. Llama 3’s pipeline removed any line appearing more than 6 times in each bucket of 30 million documents; its authors noted this also removed some frequent high-quality text, and measured that it still helped.

With every stage on, the lab keeps 34 of 113 pages and 28% of the words, and loses 5 of its 28 articles on the way.

Why duplicates matter: memorization

A model that sees the same text many times can learn to reproduce it word for word, which is called memorizationMemorizationWhen a model can reproduce passages of its training data word for word. It is much more likely for text the model saw many times, which is one reason training data is deduplicated.Open in glossary. Lee and colleagues (2022) found a 61-word sentence repeated 61,036 times in C4, and that over 1% of what models trained on such data wrote unprompted was copied from their training data; deduplicating cut that rate about tenfold. Kandpal and colleagues (2022) found that a sequence present 10 times in training data was generated about 1,000 times more often than one present once. Carlini and colleagues extracted hundreds of verbatim training sequences from GPT-2 in 2021, including names, phone numbers, and email addresses that had been public on the web, and later measured that memorization grows with model size, with the number of copies, and with how much of the text a prompt supplies (2023).

That is a privacy problem when the text is personal, a legal problem when it is copyrighted, and a waste of training when it is a cookie notice.

Contamination

If a benchmark’s questions are in the training data, a high score may measure memory instead of skill. That is data contaminationData contaminationWhen test questions, or their answers, end up in a model's training data, so a benchmark score partly measures memory instead of skill.Taught in Evaluation: what a score meansOpen in glossary, covered in the lesson on evaluation. The same tools that find copies fight it: Gopher’s authors removed any training document whose 13-word phrases overlapped a test document’s with a Jaccard similarity above 0.8, and Lee and colleagues found that more than 4% of the validation examples in the standard datasets they studied also appeared in the training data.

Consent, copyright, and the law

Nobody asked most authors of web pages whether their text could train a model, and the rules are still being settled.

Opting out. Crawlers name themselves, and a site’s robots.txt file can ask any of them to stay away. Common Crawl’s crawler is called CCBot; OpenAI says that disallowing its crawler, GPTBot, “indicates a site’s content should not be used in training generative AI foundation models.” In one year, 2023 to 2024, so many sites added such restrictions that about 5% of all tokens in C4, and over 28% of tokens from its most actively maintained sources, became fully restricted under robots.txtrobots.txtA file on a website that tells named crawlers which pages they may visit. Site owners use it to opt out of crawlers that collect AI training data; it is a request, not a technical barrier.Open in glossary (Longpre and colleagues, 2024).

Courts. Lawsuits ask whether copying works to train a model is lawful. Under United States law, the question is usually fair useFair useA rule in United States copyright law that allows some uses of copyrighted work without permission, decided case by case on four factors, including the purpose of the use and its effect on the market for the work.Open in glossary. Decisions so far, as of October 2026:

The outcomes differ because the facts, the laws, and the questions asked differ: how the copies were obtained, whether the model reproduces the works, and whether its output competes with them. Appeals may change any of them.

Rules for developers. In the European Union, the AI Act requires providers of general-purpose models to have a policy for complying with EU copyright law, including respecting machine-readable opt-outs, and to publish a summary of their training content on a template from the European AI Office. These obligations have applied to new models since August 2, 2025, and apply to models already on the market from August 2, 2027 (Regulation (EU) 2024/1689, Articles 53 and 111).