Long context in practice

Some models now accept a million tokens in a single request. By Anthropic’s 2023 rule of thumb of about 75,000 words per 100,000 tokens, that is about three quarters of a million words, or fewer with newer tokenizers that split the same text into more tokens: a long contract, a codebase, a shelf of reports. The model will take it. How much of it the model really uses, what it costs, and whether you should send it at all are separate questions, and this lesson answers each with measurements.

What fills a context window

The context windowContext windowThe maximum number of tokens a language model can take into account at once, covering both the prompt and the text it generates.Taught in How a chat assistant worksOpen in glossary is the most a model can take in one call, counted in tokensTokenThe unit of text a language model reads and writes. A token can be a whole word, part of a word, a single character, or punctuation, often including a leading space.Taught in TokensOpen in glossary, and the reply has to fit in it too. Anthropic’s documentation lists what counts: the system prompt, every message, tool definitions, tool results, images and documents, and the output, including any reasoning the model writes before it answers (Anthropic, as of October 2026). In a real product the window holds:

• Instructions: the system promptSystem promptInstructions an application places at the start of a language model's context, before the user's messages, to set its role, rules, and how it should use tools. The model reads them as text; nothing forces it to obey.Taught in How a chat assistant worksOpen in glossary, and in agents, instruction files and tool definitions (the agent harness lesson takes these apart).
• The conversation so far. The model keeps no memory between calls, so the application sends the whole conversation again every time.
• Retrieved material: passages found by search, as in retrieval-augmented generation.
• Tool results: files the agent read, web pages, command output.
• The model’s output: its reasoning, if it writes some, and the reply.

Windows have grown fast. GPT-3 read 2,048 tokens in 2020 (Brown and colleagues). Anthropic offered 100,000 in May 2023 (Anthropic). Llama 3.1 took 128K (131,072) in 2024, and Google’s Gemini 1.5 report tested retrieval from up to 10 million tokens (Gemini Team, 2024). In April 2025, Meta announced a 10-million-token window for Llama 4 Scout, a model trained on sequences of 256K (Meta; model card): an advertised window can run far past the lengths a model was trained on. As of October 2026, about a million tokens is the common size for large models: 1M for Anthropic’s current models (Anthropic), 1.05M for OpenAI’s current flagship models (OpenAI), 1,048,576 for Google’s Gemini 3.8 Flash (Google), and 1M for DeepSeek-V4, whose weights are open (DeepSeek).

What a longer context costs

Memory. For every token in the context, every layer keeps a key and a value, the KV cache. Llama 3.1 8B keeps 128 KiB per token in 16-bit numbers. A full window of 128K tokens (131,072) needs 16 GiB, more than the model’s own weights (16 GB, which is 15 GiB), for a single conversation. A server keeps one such cache for every conversation it is serving at that moment.

Compute. Reading the context means sending every token through the weights, about 2 operations per parameter per token, which grows in step with the length. Attention also compares each token with every token before it, which grows with the square of the length (Attention, step by step shows the $n \times n$ table of scores). For Llama 3.1 8B, attention is about 2% of the work of reading 1,000 tokens, 36% at 32,000, and 70% at a full window. Meta trained Llama 3.1 on long sequences only in its final stages, writing that “the compute in self-attention layers grows quadratically in the sequence length” (Llama Team, 2024).

Time. The model reads the whole input before it writes the first word of the reply, so a longer input means a longer wait.

Money. Providers charge per token. Anthropic’s price list, as of October 2026, charges five times as much for an output token as for an input token. A prompt that starts with the same text as an earlier one can reuse its processed keys and values, called prompt cachingPrompt cachingKeeping the processed start of a prompt, its keys and values, so later requests that begin with the same text skip that work. Providers charge less for cached tokens, but they still fill the context window.Open in glossary; those cached tokens cost a tenth of the input price or less, though writing them to the cache the first time costs more than a normal input token. One model, Claude Haiku 5.5, charges five times as much per token for prompts over 100,000 tokens, while Anthropic’s other current models charge the same per token across the whole window (pricing). Cached tokens still fill the window: caching changes what you pay, not what the model has to fit.

# Memory and compute to read one context with Llama 3.1 8B in 16-bit numbers.
layers, kv_heads, q_heads, head_dim = 32, 8, 32, 128
params = 8_030_261_248
embedding = 128_256 * 4_096              # input embedding: a table lookup, no arithmetic

def costs(tokens):
    cache = 2 * layers * kv_heads * head_dim * 2 * tokens    # keys and values, 2 bytes each
    weights = 2 * (params - embedding)                        # per token: 2 per parameter
    # Attention: about 4 operations per number per earlier token, and a token sees
    # half the context on average, so 2 x layers x tokens x width per token.
    attention = 2 * layers * tokens * q_heads * head_dim
    return cache, tokens * (weights + attention), attention / (weights + attention)

print(f"16-bit weights: {2 * params / 2**30:.1f} GiB")
for tokens in [1_000, 8_000, 32_000, 131_072]:
    cache, ops, share = costs(tokens)
    print(f"{tokens:>7,} tokens: cache {cache / 2**30:5.2f} GiB, "
          f"{ops:.2e} operations, {share:.0%} in attention")

16-bit weights: 15.0 GiB
  1,000 tokens: cache  0.12 GiB, 1.53e+13 operations, 2% in attention
  8,000 tokens: cache  0.98 GiB, 1.37e+14 operations, 12% in attention
 32,000 tokens: cache  3.91 GiB, 7.49e+14 operations, 36% in attention
131,072 tokens: cache 16.00 GiB, 6.47e+15 operations, 70% in attention

What the numbers show: memory grows in step with the length, and compute grows faster. The larger model’s weights do more work per token, so attention takes longer to dominate, but its cost still grows with the square of the length. The sizes in the demo are examples, except two: the passages are the size Liu and colleagues used, and the novel is The Great Gatsby, which Anthropic loaded in 72,000 tokens in 2023.

How models use a long context

Fitting text into the window is not the same as using it. Liu and colleagues (2023) gave models a question and 10, 20, or 30 Wikipedia passages, exactly one of which held the answer, and moved that passage from first to last. If a model used its input evenly, the position would not matter.

It mattered a lot. Accuracy was highest with the answer at the start, high again at the end, and lowest in the middle. With 20 passages, GPT-3.5-Turbo answered 75.8% correctly with the answer first, 63.2% with it last, and 53.8% with it in the middle, below the 56.1% it scored with no passages at all, from what it learned in training. Versions of the same models with longer windows did no better. The authors named this being lost in the middleLost in the middleThe tendency of language models to use information at the start or end of a long input better than information in the middle, named by Liu and colleagues in 2023.Open in glossary. These are their reported results for mid-2023 models, not a simulation:

Needles, and why they are not enough

A popular test followed. In November 2023, Greg Kamradt hid one sentence, the needle, in a long stack of essays, the haystack, and asked models about it at many depths and lengths (Kamradt, 2023). This needle-in-a-haystackNeedle in a haystackA test that hides one fact at different depths in long filler text and asks the model to find it. Passing shows a model can locate a fact that matches the question; harder long-context tests often find much lower scores.Open in glossary test made some failures visible and some fixes famous: Anthropic reported that Claude 2.1 scored 27% on it until the reply was started with “Here is the most relevant sentence in the context:”, which raised it to 98% (Anthropic, December 2023). By 2024 models routinely passed: Meta reported that Llama 3.1 retrieved 100% of needles at every depth and length.

The test is easy because the needle shares words with the question and only one fact is needed. Harder tests tell a different story:

• RULER (Hsieh and colleagues, 2024) added several needles, chains of variables to follow, words to count, and questions among distractors. Almost all 17 models it tested passed the plain needle test, but only 10 stayed above Llama 2 7B’s score at 4,000 tokens once inputs reached 32,000, although all claimed 32,000 or more. Llama 3.1 8B claims 128K and stays above that bar to 32K.
• NoLiMa (Modarressi and colleagues, 2025) wrote needles that share almost no words with the question, so finding them takes a small inference. At 32,000 tokens, 11 of the 13 models it tested fell below half of their short-input score; GPT-4o, one of the best, fell from 99.3% to 69.7%.
• Chroma, in a technical report that was not peer reviewed, tested 18 models and found accuracy falling with length even on simple tasks, and, oddly, better results when the haystack’s sentences were shuffled than when they formed coherent text (Hong and colleagues, 2025). Anthropic’s documentation calls this decline “context rot”.
• MRCR asks the model to tell apart several near-identical requests hidden in a long conversation, for example to return the second of several poems about tapirs. Google introduced it with the Michelangelo tests (Vodrahalli and colleagues, 2024), and OpenAI released an open version with 2, 4, or 8 such requests at lengths up to 1 million tokens in April 2025 (OpenAI MRCR). It stays hard at full length: in Google’s Gemini 2.5 report, Gemini 2.5 Pro scored 58.0% on Google’s own harder 8-request version, MRCR-V2, at up to 128K tokens and 16.4% at 1 million (Gemini Team, 2025).

Retrieval or a longer context?

Retrieval-augmented generation sends only the few passages a search finds. A long context sends everything and lets the model find what it needs. Each wins somewhere.

Retrieval wins when the collection is bigger than the window, changes often, or is asked about many times: a few thousand tokens per question cost far less than hundreds of thousands, and the sources can be shown. It also keeps the relevant text short, which avoids the middle.

A long context wins when the question needs the whole text, such as a summary or a comparison across a long document, or when search would miss the passage that matters.

The evidence points both ways. Li and colleagues (2024) compared the two on long-document benchmarks: a long context beat retrieval on average, by 7.6 points with Gemini-1.5-Pro, 13.1 with GPT-4o, and 3.6 with GPT-3.5-Turbo. Yet the two gave identical answers on 63% of questions. Sending each question to retrieval first, and to the long context only when the model judged the passages insufficient, kept the quality close to the long context’s while sending 39% (Gemini-1.5-Pro) to 61% (GPT-4o) of the tokens. Where the documents were far longer than GPT-3.5-Turbo’s 16K window, retrieval won outright. Earlier, Xu and colleagues (2024) found that retrieval with a 4K window matched a model extended to 16K at much less compute, and that retrieval helped even models with 32K windows.

More retrieved passages are not always better. In Liu and colleagues’ tests, going from 20 to 50 retrieved passages raised the share of questions whose answer was in the input, but improved GPT-3.5-Turbo’s accuracy by only about 1.5 points. Ranking the passages well and keeping the best few, near the start or the end, matters more.

Choosing what goes into the window has a name, context engineeringContext engineeringChoosing and maintaining what goes into a model's context window on each call, such as instructions, tool definitions, retrieved text, and tool results, because a longer context costs more and is used less well.Open in glossary. Anthropic’s engineering guide describes it as curating “the optimal set of tokens” for each call, and says context “must be treated as a finite resource with diminishing marginal returns” (Anthropic, September 2025). Practical habits that follow:

• Put stable material first (instructions, tool definitions, reference documents) so prompt caching can reuse it, and the changing parts last.
• Place the most important material at the start or the end, not in the middle.
• Clear out what is no longer needed. Agents summarize old turns and drop old tool results, called compaction, at the cost of losing detail (see the agent harness lesson). As of October 2026, some APIs do this on the server: Anthropic’s offers compaction in beta and the clearing of old tool results (Anthropic).
• Retrieve when the collection is large or changing; use a long context when the task needs the whole text; test both on your own questions.

The cost side of all this, in energy as well as money, is in Compute and energy.