Long context, cost and position

Fill a context window

Add the pieces a real product puts in front of a model, and see the tokens, the memory, and the compute they cost.

18,000 tokens used, 113,072 free in Llama 3.1 8B's window.

System prompt2,000 tokens (example size)12,000
Tool definitions250 tokens each (example size)00
Conversation turns600 tokens each (example: a 100-token question and a 500-token answer)2012,000
Retrieved passages150 tokens each (the size of Liu and colleagues' Wikipedia passages in their prompts)00
Tool results5,000 tokens each (example size of a file read or a web page)00
A whole novel72,000 tokens each (The Great Gatsby, 72,000 tokens in Anthropic's 2023 test)00
Reply and reasoning4,000 tokens each (room reserved for the model's output)14,000

Memory, 16-bit

Weights15.0 GiB

Cache2.20 GiB

Work to read the context

Weights270 trillion

Attention84.9 trillion

Model

Llama 3.1 8BLlama 3.1 70B

Start from

Tokens in the window18,000 of 131,072

Cache, 16-bit2.20 GiB

Operations to read it355 trillion

Share in attention24%

Input at $1 per million tokens$0.018

Challenges

0 of 2 complete

With Llama 3.1 8B, fill the window until the memory for this one conversation's cache is larger than the model's own weights. (not yet)
With Llama 3.1 70B, fill the window until comparing tokens in attention takes more than half the work of reading the context. (not yet)

How it is worked out. Cache = 2 (keys and values) × layers × key/value heads × head size × 2 bytes per token: 128 KiB per token for Llama 3.1 8B and 320 KiB for 70B. Operations to read the context = tokens × (2 per parameter for the weights + the attention scores and weighted sums over the context), the estimate used across this site. The price is a round $1 per million input tokens so you can scale it to any price list.

Sizes. Passages are the size of the Wikipedia passages in Liu and colleagues' tests (about 150 tokens as they appeared in the prompt). The novel is The Great Gatsby, 72,000 tokens in Anthropic's 2023 test. The other sizes are round examples; real products vary widely, and different tokenizers count the same text differently.

Try this

Press Chat with retrieval, then A whole book. Compare the tokens, the cache, and the share in attention.
With Llama 3.1 8B, add passages to the book until the cache for this one conversation passes the model’s weights.
Switch to Llama 3.1 70B with the same pieces: a larger cache per token, yet a smaller share in attention.
Push the total past the window and read the message.

Memory and compute for one contextPython with the standard library

# Memory and compute to read one context with Llama 3.1 8B in 16-bit numbers.
layers, kv_heads, q_heads, head_dim = 32, 8, 32, 128
params = 8_030_261_248
embedding = 128_256 * 4_096              # input embedding: a table lookup, no arithmetic

def costs(tokens):
    cache = 2 * layers * kv_heads * head_dim * 2 * tokens    # keys and values, 2 bytes each
    weights = 2 * (params - embedding)                        # per token: 2 per parameter
    # Attention: about 4 operations per number per earlier token, and a token sees
    # half the context on average, so 2 x layers x tokens x width per token.
    attention = 2 * layers * tokens * q_heads * head_dim
    return cache, tokens * (weights + attention), attention / (weights + attention)

print(f"16-bit weights: {2 * params / 2**30:.1f} GiB")
for tokens in [1_000, 8_000, 32_000, 131_072]:
    cache, ops, share = costs(tokens)
    print(f"{tokens:>7,} tokens: cache {cache / 2**30:5.2f} GiB, "
          f"{ops:.2e} operations, {share:.0%} in attention")

16-bit weights: 15.0 GiB
  1,000 tokens: cache  0.12 GiB, 1.53e+13 operations, 2% in attention
  8,000 tokens: cache  0.98 GiB, 1.37e+14 operations, 12% in attention
 32,000 tokens: cache  3.91 GiB, 7.49e+14 operations, 36% in attention
131,072 tokens: cache 16.00 GiB, 6.47e+15 operations, 70% in attention

Where the answer sits

Recorded results from two studies of real models: how accuracy depends on where the answer is in the input, and on how long the input is.

Passage 1 of 20 holds the answer (A). The other 19 were retrieved for the same question but do not contain it.

Answer 1st of 2075.8% correct

No documents56.1%

Only the answer document88.3%

Challenges

0 of 2 complete

Find a place for the answer where GPT-3.5-Turbo does worse than when it gets no documents at all. (not yet)
Find the longest input at which Llama 3.1 8B still scores above Llama 2 7B's score at 4K tokens. (not yet)

Recorded, not simulated. Liu and colleagues (2023, published in TACL 2024) gave each model a question from NaturalQuestions and 10, 20, or 30 Wikipedia passages, one holding the answer and the rest retrieved distractors, and moved the answer through the list. These are their reported accuracies (Tables 1, 5, 6, 7). GPT-3.5-Turbo's 4K window cannot hold 30 passages. These models are from mid-2023; the lesson describes harder tests of newer ones.

Try this

Move the answer from 1st to 10th with GPT-3.5-Turbo and 20 documents, and compare with the no documents line.
Try Claude-1.3: among the flattest curves, but compare it with its own score when given only the answer document.
Switch to How long the input is and step Llama 3.1 8B from 4K to 128K tokens.
Compare Gemini 1.5 Pro and Mistral 7B v0.2 at 64K.

The second demo shows recorded results, not a simulation: accuracies reported by Liu and colleagues (2023) for six mid-2023 models, and RULER scores reported by Hsieh and colleagues (2024). The lesson explains both studies and what newer tests found.