How LLMs Process Text: Tokens & Probability Distributions
Explore byte-pair encoding (BPE), token calculations, and next-token probability generation.
Why this matters
You set
temperature: 01.2Both are the same misunderstanding. Temperature is not a creativity dial, and it is not a correctness dial. It is a single division applied to a list of numbers, at one specific point in a pipeline — and once you can see where that division happens, both behaviours stop being surprising and start being predictable.
This lesson is about that pipeline. It is the foundation for every prompting technique in the rest of the course, because all of them are attempts to influence one step of it.
The idea
A language model does not read words. Before anything else happens, your text is cut into tokens — subword fragments drawn from a fixed vocabulary of typically 32,000 to 128,000 entries. "unbelievable" becomes something like
["un", "believ", "able"]Every token in that vocabulary then gets a score. Not a probability — a logit, an unbounded real number, one per vocabulary entry, produced fresh for every position. For a 50,000-token vocabulary the model emits 50,000 numbers to decide a single next token.
Those scores become probabilities through softmax, which does two things: it makes every number positive by exponentiating, then divides by the total so they sum to 1. The key property is that exponentiating amplifies gaps. A logit of 4.2 against 2.1 is a difference of 2.1; after exponentiation it is a ratio of about 8:1.
Temperature is a division applied to the logits before softmax reaches them. Divide by 0.5 and every gap doubles, so the leader pulls further ahead. Divide by 2.0 and every gap halves, so the field bunches together.
This is why temperature cannot change which token is most likely. Dividing every number by the same positive value cannot reorder them. It changes how confident the distribution is, never what it prefers. Anything that changes the ranking — a system prompt, a few-shot example, retrieved context — changes the logits themselves, upstream of here.
Worked example
Suppose the model has just read
The capital of France is| Token | Logit |
|---|---|
Code Block | 8.1 |
Code Block | 5.4 |
Code Block | 4.9 |
Code Block | 4.2 |
At T = 1.0 the logits pass through unchanged. Exponentiating and normalising gives roughly Paris 88%, the 6%, a 4%, located 2%. Confident, but not certain — one call in eight starts a sentence some other way.
At T = 0.2 every logit is multiplied by five: 40.5, 27.0, 24.5, 21.0. The gaps are now enormous, and after softmax Paris takes essentially 100%. This is what people mean by "deterministic", and it is worth being precise: the distribution has collapsed so far that alternatives are numerically negligible. Most APIs additionally special-case
T = 0At T = 1.5 the logits shrink to 5.4, 3.6, 3.3, 2.8. Paris drops to about 68%, and the combined chance of something else rises to nearly one in three. Run it ten times and you should expect three answers that do not begin with Paris.
Now the important part. Notice what did not change: Paris was the highest-scoring token at every temperature. If your model is confidently wrong, lowering the temperature makes it more consistently wrong, not more correct. Temperature is the wrong tool for that problem — you need to change the logits, which means changing the prompt or the context.
Use the slider below the diagram to watch this happen on a live distribution.
Common mistakes
"Temperature 0 gives identical output every time." Tempting because it is nearly true, and because the docs often say "deterministic". In practice floating-point addition is not associative, and on GPUs the order of operations in a batch can vary between runs, so tiny differences occasionally flip a near-tie. It is also why the same prompt can differ across providers, hardware, or a model version bump. Treat T=0 as highly repeatable, not guaranteed identical — and if you need a guarantee, validate the output rather than trusting the setting.
"High temperature makes the model more creative." Tempting because the output does become more varied. But temperature flattens the distribution uniformly — it raises the probability of genuinely interesting continuations and of inarticulate ones by the same factor. What you get is not more creativity but more variance, and past about 1.2 the tail is mostly noise. Real creative control comes from the prompt. Top-p is the better knob for variety, because it trims the tail before sampling instead of inflating it.
"Temperature and top-p do the same thing, so set both." Tempting because both affect randomness. They act at different stages: temperature reshapes the distribution, then top-p truncates it to the smallest set of candidates whose probabilities sum to p. Setting both aggressively compounds in ways that are hard to reason about. The common advice — change one, leave the other at its default — exists because debugging two interacting samplers is genuinely unpleasant.
"Token count is roughly word count." Tempting because it is close for ordinary English prose. It falls apart precisely where it costs you: JSON, code, non-Latin scripts, and long identifiers can be two to four times worse. If you are budgeting a context window, count tokens with the real tokeniser rather than estimating.
Your turn
Take a prompt you already use that returns structured output. Run it five times at
T = 0T = 0.8You are looking for one specific thing: whether your failures are format failures or content failures. If lowering the temperature fixes them, they were sampling noise. If it does not, the problem is in your prompt, and no sampling setting will rescue it.
Success criterion: you can say which of the two you have, and point at the evidence.
Recap
- A model scores every token in its vocabulary at every position; softmax turns those scores into a distribution that sums to 1.
- Temperature divides the logits before softmax, so it changes how sharp the distribution is and never which token is preferred.
- Low temperature makes a confidently wrong model more consistently wrong — it is not a correctness control.
- Top-p truncates the distribution, which is usually the better tool for controlling variety.
- Token counts diverge from word counts exactly where it matters: code, JSON and non-English text.
Sources
- Holtzman et al., The Curious Case of Neural Text Degeneration (2019) — introduces nucleus (top-p) sampling and shows why pure likelihood maximisation produces degenerate text.
- Sennrich et al., Neural Machine Translation of Rare Words with Subword Units (2016) — the BPE algorithm behind most modern tokenisers.
- OpenAI, Tokenizer documentation (accessed August 2026) — for counting tokens against a real vocabulary rather than estimating.
# Softmax Temperature Scaling Function
import math
def calculate_token_probabilities(logits: list[float], temperature: float = 0.2):
if temperature <= 0.001:
max_idx = logits.index(max(logits))
return [1.0 if i == max_idx else 0.0 for i in range(len(logits))]
# Scale logits by temperature T
scaled = [z / temperature for z in logits]
exp_vals = [math.exp(z) for z in scaled]
total = sum(exp_vals)
return [round(e / total, 4) for e in exp_vals]
# Candidate Token pool: ["Yes", "No", "Maybe"]
raw_logits = [4.2, 2.1, 1.0]
probabilities = calculate_token_probabilities(raw_logits, temperature=0.2)
print("Computed Probabilities (Yes, No, Maybe):", probabilities)When generating strict JSON outputs or financial summaries where zero hallucination is critical, what temperature setting is most recommended?
Read to the end of the lesson