How a chat assistant works

You type a message into a chat assistant, and a reply appears. It answers your question rather than adding more questions, it remembers what you said ten messages ago, and it may know today’s date. It feels like talking to something that keeps track of you.

Underneath, the app does something much plainer. It turns the whole conversation into one long text and asks a language model to continue it. Everything else, the roles, the personality, the memory, comes from how that text is built and what goes into it.

This lesson takes that apart with a real chat model running in your browser: SmolLM2-135M-Instruct, the model from Inside a real language model after two more stages of training. It is tiny next to the models behind commercial assistants, so its mistakes are easy to see. Watch them: they show the machinery.

From continuing text to answering

A base modelBase modelA language model after pretraining only. It continues text in whatever style the input suggests, rather than following instructions or answering like an assistant.Taught in Inside a real language modelOpen in glossary has one skill: continuing text the way documents continue. To make an assistant, its makers train it further on conversations. For SmolLM2-135M-Instruct, Hugging Face first fine-tuned the base model on about 460,000 example conversations (smol-smoltalk), the instruction tuningInstruction tuningFine-tuning a pretrained model on many tasks written as instructions paired with good responses, so it learns to follow instructions it has not seen before.Taught in Fine-tuning and LoRAOpen in glossary of Fine-tuning and LoRA. Then it applied direct preference optimizationDirect preference optimizationA way to train a language model on pairs of preferred and rejected responses with a single classification-style loss, without a separate reward model or reinforcement learning.Taught in How LLMs are trainedOpen in glossary (How LLMs are trained) on about 61,000 pairs of answers to the same prompt, one rated better than the other; GPT-4 did the rating, not people (UltraFeedback, model card).

Nothing else changed. The two models have the same 30 blocks and the same parameter count, and they even ship the same tokenizer file. Only the numbers in the weights moved. So you can give both exactly the same tokens and compare what each writes.

from collections import Counter, defaultdict

# A toy "instruction tuning" set in the chat layout (SmolLM2-135M-Instruct
# was tuned on about 460,000 conversations; three show the idea).
pairs = [("What is the capital of France ?", "Paris ."),
         ("What is the capital of Italy ?", "Rome ."),
         ("Name a color .", "Blue .")]

def chat(user, answer=""):
    return ["<|im_start|>", "user", "\n", *user.split(), "<|im_end|>", "\n",
            "<|im_start|>", "assistant", "\n", *answer.split()]

# Count which token follows each token and each pair of tokens.
follows = defaultdict(Counter)
for user, answer in pairs:
    seq = chat(user, answer) + ["<|im_end|>"]
    for i in range(1, len(seq)):
        follows[(seq[i - 1],)][seq[i]] += 1
        if i > 1:
            follows[(seq[i - 2], seq[i - 1])][seq[i]] += 1

def next_token(tokens):
    # Use the last two tokens if that pair was seen, else the last one.
    counts = follows.get(tuple(tokens[-2:])) or follows[(tokens[-1],)]
    word, n = counts.most_common(1)[0]
    return f"{word!r} ({n} of {sum(counts.values())})"

question = "What is the capital of Spain ?"
print("plain text    ->", next_token(question.split()))
print("chat template ->", next_token(chat(question)))  # the layout, not geography
print("after Blue .  ->", next_token(chat("Name a color .", "Blue .")))

plain text    -> '<|im_end|>' (2 of 2)
chat template -> 'Paris' (1 of 3)
after Blue .  -> '<|im_end|>' (1 of 1)

The chat template

The box marked “Both models read exactly this” shows the instruct model’s chat templateChat templateThe exact layout a chat model was fine-tuned on, which turns a list of messages into one text: special tokens mark where each message starts and ends and who wrote it, and the text ends where the assistant's reply should begin.Open in glossary: the layout of every conversation it was tuned on. Each message starts with <|im_start|> and the author’s role (system, user, or assistant), then a line break, the words, and <|im_end|>. Those tags are special tokensSpecial tokenA token reserved for structure rather than words, such as one marking the end of a text or the start and end of a chat message. When a chat model writes its end-of-turn token, the app stops generating.Open in glossary: each is a single entry in the 49,152-token vocabulary (ids 1 and 2), not a string of letters.

When a conversation does not start with a system message, this template inserts one: “You are a helpful AI assistant named SmolLM, trained by Hugging Face”. That is where the name in the answer to Who are you? came from.

The text ends with <|im_start|>assistant and a line break: the start of a reply with no words yet. In every conversation the model was tuned on, what follows is an answer, so that is what it writes, and when the answer is done it writes <|im_end|>. The app watches for that token and stops asking for more. Commercial APIs report the same two endings: Anthropic’s returns the stop reason end_turn when “Claude finished its response naturally” and max_tokens when the reply hit the length limit you set (Anthropic).

Now the two strange results make sense:

• The instruct model, given plain text, sees words that look like the inside of a user’s message with no end yet. In its training, a user’s message is always followed by <|im_end|>, so that is its most likely next token, at 73% for “What is the capital of France?” (in the chat template, “The” leads with 83%). The reply ends before it starts.
• The base model, given the template, was pretrained on web text, code, math, and synthetic textbooks; the SmolLM2 paper describes its conversation data as post-training data, used after the base model was trained (Allal et al., 2025). To the base model the tags are rare tokens in a strange document, and it continues the document as best it can. On none of the examples does it write <|im_end|>, so it runs until the lab stops it at 48 tokens.

{% for message in messages %}
  {% if loop.first and messages[0]['role'] != 'system' %}
    {{ '<|im_start|>system\nYou are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>\n' }}
  {% endif %}
  {{ '<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n' }}
{% endfor %}
{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}

One long text, sent again every turn

Now the whole conversation. In the lab below, the left side is what a chat app shows you. The right side is what the model actually reads for each reply: one text, built again from the start every turn.

# SmolLM2-135M-Instruct's chat template, written out by hand.
DEFAULT_SYSTEM = "You are a helpful AI assistant named SmolLM, trained by Hugging Face"

def piece(role, content):
    return f"<|im_start|>{role}\n{content}<|im_end|>\n"

def render(system, turns, message):
    text = piece("system", system or DEFAULT_SYSTEM)  # inserted when none is given
    for user, assistant in turns:
        text += piece("user", user) + piece("assistant", assistant)
    return text + piece("user", message) + "<|im_start|>assistant\n"

turns = [
    ("Hi! My name is Priya.", "Hello, Priya! How can I help you today?"),
    ("Give me one tip for sleeping better.",
     "Go to bed and wake up at the same time every day, even on weekends."),
]
message = "What is my name?"

# Tokens in each piece, counted with SmolLM2's tokenizer. Special tokens split
# the text, so the pieces' counts add up to the whole prompt's.
count = {"system": 21, "header": 4, "What is my name?": 10,
         turns[0]: 13 + 18, turns[1]: 13 + 23}

window, reply_limit = 128, 32  # the app keeps room for the reply
kept = list(turns)
while True:
    used = count["system"] + sum(count[t] for t in kept) + count[message] + count["header"]
    if used + reply_limit <= window or not kept:
        break
    kept.pop(0)  # drop the oldest turn

print(f"{len(turns) - len(kept)} turn dropped; the prompt is {used} tokens:")
print(render(None, kept, message))

1 turn dropped; the prompt is 71 tokens:
<|im_start|>system
You are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>
<|im_start|>user
Give me one tip for sleeping better.<|im_end|>
<|im_start|>assistant
Go to bed and wake up at the same time every day, even on weekends.<|im_end|>
<|im_start|>user
What is my name?<|im_end|>
<|im_start|>assistant

The model remembers nothing

Between your messages, nothing about the model changes. Its weights are the same file you downloaded, and it keeps no notes. The app holds the conversation and sends all of it every time.

Commercial APIs work the same way. Anthropic’s documentation says: “The Messages API is stateless, which means that you always send the full conversational history to the API” (Anthropic). OpenAI’s says that “each text generation request is independent and stateless”; when a request continues an earlier one through its previous_response_id, all the earlier input tokens are still billed as input (OpenAI).

Sending everything again is less wasteful than it sounds. With no system prompt and a 1,024-token window, the fifth message sends 351 tokens, but the first 336 are exactly the text the model read and wrote last turn, so their keys and values are still in its cache (the KV cache). Only 15 tokens are new. Providers do the same across requests and charge less for the reused part, which is called prompt cachingPrompt cachingKeeping the processed start of a prompt, its keys and values, so later requests that begin with the same text skip that work. Providers charge less for cached tokens, but they still fill the context window.Taught in Long context in practiceOpen in glossary.

The system prompt

The system message is the app’s own text, placed before your first message. SmolLM’s template supplies a default one that gives the model a name. Products write their own: Anthropic publishes the system promptsSystem promptInstructions an application places at the start of a language model's context, before the user's messages, to set its role, rules, and how it should use tools. The model reads them as text; nothing forces it to obey.Open in glossary of its Claude apps, which it says “provide up-to-date information, such as the current date, to Claude at the start of every conversation”; they do not apply to its API (Anthropic).

In the lab, Pirate gives the model a new name and a voice for the whole conversation (“Me name’s Blackbeak Betty”). Be brief asks for one short sentence, and this small model mostly ignores the request: its replies are about as long as with no system prompt, and it decides it is a creative writing assistant. The model reads the system prompt as text, like everything else, so nothing forces it to obey (Safety in AI products). It is simply the first thing in every text the model continues.

When the window fills

The context windowContext windowThe maximum number of tokens a language model can take into account at once, covering both the prompt and the text it generates.Open in glossary is the most tokens a model can read at once, its reply included. SmolLM2 handles 8,192. In October 2026, Anthropic lists 1 million tokens for each of its current Claude models (Anthropic). The lab uses windows of 256 to 1,024 tokens so you can watch one fill in a few messages.

When the conversation no longer fits, the app must leave something out. The lab does the simplest thing: it drops the oldest turns, always keeps the system prompt and your new message, and keeps 96 tokens free for the reply. Two things follow:

• What is dropped is gone. With Be brief and 512 tokens, nothing is dropped, and the answer to “What is my name?” is “I’m Priya.” The model found the name, though it mixed up who is who. With 256 tokens, the turn with the name was dropped, and the answer is “My name is Emily Wilson.”, a name it made up.
• The cache is lost too. Dropping the oldest turn changes the start of the text, so every later token sits at a new position. With no system prompt and 256 tokens, the first prompt after a drop starts the same way as the last one for only 24 tokens, the system prompt and the next tag, and the model reads the other 101 again.

Other strategies trade differently. Apps can replace old turns with a summary (compactionCompactionFreeing room in a full context window by replacing older parts of a conversation with a shorter summary. Details the summary leaves out are gone from what the model sees.Taught in Inside an agent harnessOpen in glossary, in Inside an agent harness), keeping the gist and losing the details. They can retrieve only the relevant earlier text, as in retrieval-augmented generation, whose lab now ends with this same model writing an answer. Or they can use a very long window, which costs memory and compute and which models do not use evenly from start to end (Long context in practice).

What the model does not know

A model knows only what was in its training data, collected before training began. That date is its training cutoffTraining cutoffThe date after which a model's training data contains nothing. The model cannot know later events unless an app puts them in its context, and it has no clock: today's date has to come from the app too.Open in glossary. It also has no clock. Asked “What year is it?” with no date in front of it, SmolLM2-135M-Instruct answered “I’m sorry for any confusion, but as a chatbot, I don’t have real-time access to specific dates or times.” Told to be brief, it said the year was 1920; as a pirate, 1620. With the date in its system prompt, written by the app on the day of recording, it answered “The year is 2026.”

Hugging Face does not state a training cutoff for SmolLM2. Anthropic lists two dates for each Claude model: a “reliable knowledge cutoff”, the date through which its knowledge is “most extensive and reliable”, and a broader training data cutoff; as of October 2026, both are June 2026 for its current models (Anthropic). Anything later reaches a model only through its context: the date in the system prompt, a document you paste, or results from a search the app runs. Searching is a tool: the model writes a request in an agreed format, the app runs it, and the result goes back into the conversation as more text. AI that takes actions, Models that use tools, and Inside an agent harness show how.

A very small assistant

The labs show plenty of mistakes. The model mixes up who is who (“I’m Priya”), gives a stock apology to unrelated questions, and invents names and years with complete fluency. Its makers warn that its output “may not always be factually accurate, logically consistent, or free from biases present in the training data” (model card). Larger models trained the same way make these mistakes less often. They still read one long text through a template, keep nothing between turns, see only what fits in their window, and know only their training data and what the app adds.