In-Context Learning: How LLMs Learn from Examples

Key Takeaways

The Surprising Ability

When GPT-3 was released in 2020, one of its most striking properties was few-shot learning: show it three examples of translating English to French, and it would correctly translate the fourth, without any training, weight updates, or fine-tuning. The model was never explicitly trained to do translation; it learned from the pattern in context.

This is called In-Context Learning (ICL) because the 'learning' happens in the context window, not in the model weights. It's a form of meta-learning, the pre-training process teaches the model to learn from examples at inference time, not just to produce text. GPT-3's ICL capability was the primary reason it became an immediate landmark. Zero-Shot and Few-Shot are the two main forms of ICL, with Few-Shot generally outperforming Zero-Shot on harder tasks.

Zero-Shot, One-Shot, and Few-Shot

ICL comes in three strengths: zero-shot (task description only, no examples), one-shot (one example), and few-shot (2-10 examples). Performance typically improves from zero to few-shot, though the gap depends on task difficulty and model size. Larger, more capable models show stronger zero-shot performance because they've seen more task patterns during training.

For new, unusual tasks that the model may not have encountered during training, few-shot prompting provides essential disambiguation. The examples show the model the exact format, style, and level of detail you need, compensating for the model's prior toward its training distribution. For common tasks (summarization, translation, Q&A), zero-shot usually suffices.

Why ICL Works: Competing Theories

The mechanism of ICL remains an active research debate. One view: models perform implicit gradient descent in their forward pass, the transformer's attention mechanism implements a learning algorithm that adjusts internal representations based on examples. Evidence: ICL performance scales similarly to learning curves from explicit optimization.

An alternative view: ICL is sophisticated pattern matching. The model recognizes that the current context resembles training examples where it learned task X, and activates learned task-X behavior. Evidence: ICL is sensitive to example order and the specific wording of demonstrations in ways that differ from explicit optimization. Both mechanisms likely contribute.

Limitations of In-Context Learning

ICL is constrained by the context window: you can only fit so many examples. For tasks requiring extensive demonstration libraries, fine-tuning is more practical. ICL is also sensitive to example quality and order, including misleading examples degrades performance, and earlier examples in the context have less influence than recent ones for some tasks.

ICL doesn't update model weights, so nothing 'learned' in one context carries to the next. For persistent behavioral changes across all interactions (consistent output format, domain-specific terminology, behavioral constraints), fine-tuning is necessary. ICL is best thought of as dynamic configuration at inference time, not persistent learning.