What a language model is, and why it sometimes makes things up
ChatGPT, Gemini and Claude are all built on language models. Here is how they really work, with no formulas, and why they occasionally get things wrong with total confidence.
- Level
- beginner
- Time
- 5 minutes
- What you need
- Nothing, just curiosity
When you type a question into an assistant like ChatGPT, Gemini or Claude, there is no search engine on the other side, and no encyclopedia either. There is a large language model, in short an LLM. That is, a program that, by reading an enormous amount of text, has learned to predict which fragments of words are most likely to follow the ones in front of it.
Language model (a large one is a Large Language Model, hence LLM): a program that has learned to predict which fragments of text are most likely to follow the preceding ones. Token: the fragment of a word the model works on, often just a few letters. Hallucination: an invented answer, written with the same confidence as a true one.
A game of completion
The principle is your phone keyboard’s autocomplete, scaled up enormously. The model receives a sequence of words, or rather word fragments, called tokens, and works out the probability of each possible next token. Then it draws one of the likelier ones, not always the same one, appends it to the sequence and starts again. And that is why the same question, asked twice, gives you slightly different answers.
A ten-line answer is the result of hundreds of these predictions in a chain.
What makes modern models so capable is the Transformer architecture, presented in 2017 by a group of Google researchers. Its key mechanism, attention, lets the model weigh which words in the preceding text matter most when deciding the next one. That is how it keeps track of a long argument, respects grammar and picks up context.
Where it learns
Training happens on billions of pages: books, websites, articles, code. The model does not keep a copy of the text to look things up in, it adjusts billions of internal parameters until its predictions become accurate. A few heavily repeated phrases do end up stuck in it almost word for word, but that is the exception rather than how it normally works.
After that phase, developers refine it with example conversations and with the judgement of human reviewers, to make it useful and to teach it to refuse dangerous requests.
Why it “hallucinates”
A language model knows a great many things, but it does not hold them in a filing cabinet it can open and check. They are blended into its parameters along with everything else, and it has no way of telling what it genuinely knows from what it is merely reconstructing.
Ask it the birth date of a little-known person and it will produce a plausible date anyway, because “a date” is what statistically follows that kind of question. The answer arrives written with the same confidence as the correct ones, and yet it is invented. The phenomenon is called hallucination, and it is the most important limitation to keep in mind.
To reduce the problem, many assistants now search the web or documents you provide before answering, and cite their sources. One rule still holds, and I apply it even when the sources are there: for numbers, dates, names and quotations, check an independent source.
The takeaway
An LLM is an extraordinary tool for writing, summarising, translating, coding and reasoning about text you give it. As an oracle for specific facts it is less reliable.
So when you ask it a question, work out first which of the two jobs you are giving it. If it is the second one, keep another browser tab open for checking.