Skip to main content
Lab Grimoire
TW EN
Coffee
LLM Internals

How large language models work: one token at a time, up to a whole answer

A large language model takes text in and gives text out by doing one thing over and over: predicting the next token from what sits in its context window. How pretraining and instruction tuning turn that into an assistant, what reasoning models buy with extra time, why LLMs invent things, and when a decision model or embedding model fits better.

Author
CY
Published
Updated
AI from Principles to Practice · Part 2 of 70
How large language models work: one token at a time, up to a whole answer

A large language model does one thing: it keeps writing

A large language model (LLM) takes text in and gives text out. In between it does one thing: it predicts the next token from the text so far.

A token is the unit the model works in. It can be a whole word, part of a word, or a punctuation mark. The model picks one token, adds it to the end, then reads the longer text and predicts the next one, until the answer is done. On a map of AI model types, this is the "text in, text out" square.

Large language models at a glance

The context window: what the model can see at once

There is a limit on how much text the model can look at while it predicts. That limit is the context window, and it is counted in tokens.

The document you paste and the conversation so far all have to fit inside it. The Transformer architecture that most LLMs use relies on attention, which lets each position decide which parts of the window to draw on. Whatever falls outside the window, the model cannot see. In long conversations some products cut or condense earlier turns, and the model may then lose track of a condition you set at the start.

If you want it to work from a source, put the source in the same conversation. Paste the methods section of a paper and ask for the procedure as a numbered list of steps. Everything it needs is in the window, and its only job is to reorganise it.

From autocomplete to assistant: pretraining, then instruction tuning

The first stage is pretraining. The model reads a very large amount of public text and practises predicting the next token. Brown and colleagues showed in 2020 that once a model is big enough, a few examples in the prompt are often enough for it to translate or answer questions. At this point, though, it only continues text. Ask it a question and it may reply with more questions in the same style.

The second stage is instruction tuning. In the InstructGPT work, Ouyang and colleagues first had people write example answers and fine-tuned the model on them. They then had people rank several model answers and trained further on those rankings. In their human evaluations, answers from an InstructGPT model with roughly 100 times fewer parameters were preferred over those from GPT-3. The assistant's habit of following instructions comes mostly from this step.

Reasoning models: a few more steps before the answer

A reasoning model writes out intermediate steps before it gives its answer.

The idea traces back to chain-of-thought prompting, described by Wei and colleagues in 2022: show worked steps in the examples, and sufficiently large models do better on arithmetic, commonsense and symbolic reasoning tasks. Reasoning models make writing those steps the default. The cost is time and tokens. They earn it on multi-step problems where one slip ruins the result, and add little to a simple lookup. The steps are predicted text too, so they can be wrong.

Why it invents things with a straight face

Training rewards the model for producing a fluent next token. Nothing in that objective checks whether the content is true.

When the model lacks a fact, it still writes whatever looks most like an answer. A typical case is a reference with plausible authors and a plausible journal that does not exist. Researchers call output that is unfaithful to the facts or to the source hallucination; the survey by Ji and colleagues sorts its forms and the ways people try to reduce it.

Three habits lower the risk. Put the source documents in the context window and ask it to answer only from them. Ask it to say which passage each claim comes from. Then check those passages yourself. Retrieval-augmented generation (RAG) automates the first habit: it searches for relevant passages and hands them to the model before it answers.

When a different model fits better

Use an LLM when you need new text: a summary, a rewrite, a first draft of an email.

When the answer has to be one of a fixed set of options, such as tagging 500 abstracts with one of four study designs, a decision model gives a probability for each option and cannot answer outside the list. To find the few documents most similar to one another among thousands, an embedding model turns each document into a vector and compares distances. If the input is a photo or a screenshot of a chart, look at vision-language models. Ask what goes in and what you need out, then decide whether an LLM is the right tool.

References

  1. https://arxiv.org/abs/1706.03762
  2. https://arxiv.org/abs/2005.14165
  3. https://arxiv.org/abs/2203.02155
  4. https://arxiv.org/abs/2201.11903
  5. https://arxiv.org/abs/2202.03629
  6. https://arxiv.org/abs/2005.11401

Frequently Asked Questions

Does a large language model remember what we talked about last time?

The model itself only sees the text inside its current context window; anything outside it is invisible. Some products save past conversations or summaries and put them back into the window next time, which makes it look like memory. Without that feature, every conversation starts from scratch. If you want answers based on a document, the reliable route is to paste it into the same conversation.

If I ask an LLM to include references, can I trust the content?

No. It can produce references with plausible authors, journals and years that do not exist, which is a common form of hallucination. A safer approach is to give it the source documents first, ask it to answer only from them and point to the passage behind each claim, and then check those passages yourself.

Found this useful? RSS · X · Buy me a coffee