Sampling: temperature, top-k, top-p
A model gives you probabilities, not an answer. Sampling turns them into one token. Shape one real distribution with temperature, top-k and top-p, and see which tokens can still win.
At every step an LLM scores every token in its vocabulary and turns the scores into probabilities (see What is an LLM?). Something still has to pick one token. That something is the sampler, and three settings control it:
- Temperature reshapes the probabilities. Low makes the favourite even more likely. High spreads the chances out.
- Top-k keeps only the k most likely tokens.
- Top-p (also called nucleus sampling) keeps the smallest set of top tokens whose probabilities add up to p.
Whatever survives is rescaled to add up to 100%, and one token is drawn at random. The Gemini API reference describes exactly this: top-k “limits the maximum number of tokens to consider”, and nucleus sampling “limits the number of tokens based on the cumulative probability”.
Press Play to walk through each step on one distribution. Then switch Try it to Wild and press Draw ×20 a few times, and compare it with the last preset, which uses the same temperature with top-k and top-p switched on.
- Scores
- Temperature
- Top-k
- Top-p
- Renormalise
- Draw
Suggest a name for my new dosa café in Chennai:
Suggest a name for my new dosa café in Chennai:
The model gives every token in its vocabulary a score (a logit). Here are the top 12 candidates for the next token. Higher means the model thinks it fits better. Press Play to turn these into one choice.
The candidate tokens and their scores are illustrative. Softmax, temperature, the top-k and top-p cuts, renormalisation and the random draws are the real maths. Libraries differ in the exact order of the steps; this is a common one.
What to notice
- Temperature changes the shape, not the order. The favourite token stays the favourite at every temperature. What changes is how far ahead it is. At 0.2, “Madras” and “Dosa” take almost everything. At 2, “Zebra” and “Tuesday” get a real chance.
- Top-k is a fixed shortlist. It always keeps k tokens, whether the model is sure or unsure. That’s simple, but it can keep junk when the model is confident and cut good options when it isn’t.
- Top-p adapts to the model’s confidence. When one token dominates, top-p keeps a short list. When many tokens are close, it keeps more.
- The cuts make high temperature safer. At temperature 2 with no cuts, odd names win now and then. Add top-k and top-p and the long tail is gone, so you get variety from good candidates only.
- Temperature 0 means no randomness. The sampler always takes the top token (this is called greedy decoding), and top-k and top-p no longer matter.
| You want | Try |
|---|---|
| The same answer every run (extraction, classification, tool calls) | Low temperature |
| Varied but sensible text (names, ideas, rewrites) | Moderate temperature with top-p around 0.9 |
| Brainstorming where odd ideas are welcome | Higher temperature, keep a top-p cut |
temperature, top_p and top_k values, and recommend removing them from requests. The ideas still explain what the model is doing, but on those models you steer the output with your prompt and other settings instead.