LLM Inference: How Models Generate Text
Inference is what happens when an LLM produces a response. Understanding it helps you optimize for speed, cost, and quality.
Key Takeaways
| Takeaway | Details |
|---|---|
| Inference Process | LLM inference generates text autoregressively, producing one token at a time with each token conditioned on all previous tokens. |
| Prefill Phase | Processes the entire input prompt in parallel to compute key-value representations, determining time-to-first-token latency. |
| Decode Phase | Generates output tokens sequentially through full forward passes, with tokens-per-second measuring decode throughput. |
| Hardware Requirements | 70B parameter models require roughly 140 GB of GPU memory just for weights, necessitating multiple high-end GPUs. |
| Batching Benefits | Processing multiple requests together amortizes memory bandwidth costs and dramatically improves throughput while reducing per-request cost. |
What Is LLM Inference?
Inference is the process of running a trained model to generate outputs. When you send a prompt to an LLM and receive a response, every step between your request and the first token of output is inference. It's distinct from training, which is the expensive process of updating model weights on large datasets, inference uses a fixed, trained model to produce predictions.
For text generation, inference is autoregressive: the model generates one token at a time, each conditioned on all previous tokens (the prompt plus already-generated text). This sequential dependency is the fundamental constraint that makes LLM inference harder to parallelize than training.
Prefill and Decode Phases
LLM inference has two distinct phases. Prefill processes the entire input prompt in parallel, computing the key-value (KV) representations for every input token at once. This is compute-bound and relatively fast per token. Decode generates output tokens one at a time, each step requiring a full forward pass through the model while attending to all previous tokens via the KV cache.
Time-to-first-token (TTFT) is largely determined by the prefill phase, longer prompts mean slower first tokens. After the first token, tokens-per-second (TPS) measures decode throughput. Latency-sensitive applications care most about TTFT; throughput-sensitive use cases (document processing, batch jobs) care more about TPS.
Hardware and Batching
LLM inference runs on GPUs (or TPUs) because matrix multiplications, the core operation, map naturally to parallel hardware. A 70B parameter model requires roughly 140 GB of GPU memory just to store weights in 16-bit precision, requiring multiple high-end GPUs. Inference providers like Together AI, Fireworks, and Groq have built specialized infrastructure to serve large models efficiently at scale.
Batching multiple requests together amortizes the memory bandwidth cost of loading model weights across many users, dramatically improving throughput and reducing per-request cost. Continuous batching (serving requests at different decode stages simultaneously) is a key technique that frontier inference providers use to maximize GPU utilization.
Read next
Tokens and Tokenization: The Building Blocks of LLMs
Everything you need to know about tokens, how LLMs split text into pieces, why tokenization matters for cost and performance, and how different languages tokenize.
LLM Latency: What Makes Models Feel Fast or Slow
Two key metrics, time to first token and tokens per second, determine how responsive an LLM feels. Here's what drives each.
Training vs Inference: Two Phases of an LLM's Life
Understand the difference between training an LLM (creating it) and inference (using it), including what happens at each stage, the costs involved, and why they matter for builders.