Temperature and Sampling: Controlling LLM Creativity
A clear explanation of temperature, top-p, top-k, and how sampling parameters control the balance between determinism and creativity in LLM outputs.
Key Takeaways
| Takeaway | Details |
|---|---|
| Temperature Parameter | Controls probability distribution concentration, with T < 1 making outputs more deterministic and T > 1 increasing randomness. |
| Top-P Sampling | Selects from the smallest token set reaching cumulative probability p, with 0.9-0.95 being common defaults. |
| Top-K Sampling | Restricts sampling to k most probable tokens, though most modern APIs prefer top-p for its adaptive behavior. |
| Logits to Probabilities | Raw model scores are converted to probabilities via softmax function, maintaining relative token ordering. |
| Parameter Interaction | Temperature, top-p, and top-k work together in a three-stage pipeline for fine-grained sampling control. |
| Practical Temperature Ranges | T = 0 for deterministic outputs, 0.2-0.5 for factual tasks, 0.7-1.0 for general use, and 1.0-1.5 for creative tasks. |
From Logits to Probability Distributions
After a forward pass through the transformer layers, the model produces logits, one raw score per vocabulary token. These raw scores are converted to probabilities using the softmax function: exponential of each logit divided by the sum of all exponentials. The resulting probabilities sum to 1 and represent the model's 'belief' about which token should come next.
A key property: the relative ordering of tokens doesn't change, the token with the highest logit always has the highest probability. What sampling controls is how concentrated versus spread out these probabilities are when choosing which token to actually select.
Temperature: The Creativity Dial
Temperature T is applied before softmax: logits are divided by T. Low temperature (T < 1): dividing logits by a small number amplifies differences, making high-probability tokens much more likely. At T → 0, the model always picks the highest-probability token (greedy decoding). High temperature (T > 1): dividing by a large number compresses differences, flattening the distribution toward uniform, all tokens become roughly equally likely.
Practical guidance: T = 0 for completely deterministic outputs (useful for testing and structured data extraction). T = 0.2-0.5 for factual, precise tasks (code generation, technical Q&A). T = 0.7-1.0 for general use (the default range). T = 1.0-1.5 for creative tasks (poetry, fiction, brainstorming). T > 1.5 produces increasingly incoherent output.
Top-P (Nucleus Sampling)
Top-p sampling selects from the smallest set of tokens whose cumulative probability reaches p. With p = 0.9, the model considers only the most probable tokens totaling 90% of the probability mass. On confident steps, this might be 2-3 tokens; on uncertain steps, hundreds. This adaptive behavior is its advantage over the fixed top-k cutoff.
Top-p = 0.9-0.95 is the most common default setting across AI APIs. Lower values (0.7-0.8) produce more focused but potentially repetitive text. Values near 1.0 are approximately equivalent to no top-p restriction. Top-p and temperature interact: high temperature with high top-p is maximally random; low temperature with low top-p is nearly deterministic.
Top-K Sampling
Top-k restricts sampling to the k most probable tokens. k=1 is greedy; k=50 is a common default; k=∞ is equivalent to no restriction. Top-k is simpler than top-p but less adaptive, the same k value may be too restrictive on confident steps and too permissive on uncertain ones.
Most modern LLM APIs use top-p rather than top-k, or combine them: apply top-k first to remove the long tail of low-probability tokens, then apply top-p within the remaining candidates. Temperature is applied last. This three-stage pipeline gives fine-grained control over the entire sampling distribution.
Read next
How LLMs Work: A Technical Overview
A clear technical explanation of how large language models actually process text, generate responses, and represent knowledge, from tokenization to sampling.
LLM Inference: How Models Generate Text
Inference is what happens when an LLM produces a response. Understanding it helps you optimize for speed, cost, and quality.
Prompt Engineering: The Complete Guide
Master the art and science of writing effective prompts, from basic techniques to advanced methods like chain-of-thought, few-shot learning, and structured output generation.