Model Quantization: Running LLMs on Less Hardware
How quantization reduces model size and inference cost by using lower-precision numbers, making 70B parameter models run on a single GPU and enabling on-device AI.
Key Takeaways
| Takeaway | Details |
|---|---|
| Memory Reduction | INT4 quantization reduces a 70B model from 140GB to 35GB, enabling single GPU deployment. |
| Performance Retention | 4-bit quantization typically retains 95-99% of original model benchmark performance. |
| QLoRA Fine-tuning | Enables fine-tuning 70B models on single 48GB GPUs by quantizing base weights to 4-bit. |
| On-Device Deployment | Quantization powers AI on iPhone 15 Pro and Galaxy devices using 3B-8B parameter models. |
| Quality Trade-offs | Quantized on-device models are useful for bounded tasks but don't match frontier 70B capabilities. |
What Quantization Is and Why It Matters
Neural network weights are normally stored as 32-bit (FP32) or 16-bit (FP16/BF16) floating-point numbers. A 70B parameter model in 16-bit precision requires 140GB of GPU memory, more than three high-end consumer GPUs. Quantization reduces precision: INT8 (8-bit integers) halves that to 70GB; INT4 (4-bit) gets to 35GB, now feasible on a single professional GPU or 4× consumer GPUs.
The fundamental question is how much quality is lost by reducing numerical precision. Modern quantization research has shown that 4-bit quantization typically retains 95-99% of the original model's benchmark performance, a remarkable result that has made quantized models the standard for self-hosted deployments.
Key Quantization Methods
GPTQ (Generative Pre-Training Quantization) applies per-layer weight quantization using a small calibration dataset, minimizing reconstruction error. It typically produces the best quality for INT4 quantization of large models. AWQ (Activation-aware Weight Quantization) identifies and protects the most important weights from heavy quantization, further improving quality.
GGUF (formerly GGML) is the dominant format for CPU-compatible quantization, used by llama.cpp. It supports a range of quantization levels from Q2_K (extreme compression) to Q8_0 (near-lossless). The Q4_K_M quantization level offers an excellent quality-size tradeoff for most use cases and is the recommended default for local deployment.
QLoRA: Quantized Fine-Tuning
QLoRA combines quantization with LoRA fine-tuning, enabling fine-tuning of large models on consumer hardware. The base model is loaded in 4-bit quantized form (frozen); only the LoRA adapter weights are trained in full precision. A 70B model that would normally require 160GB of GPU memory for fine-tuning can be fine-tuned on a single 48GB GPU with QLoRA.
QLoRA quality is surprisingly close to full fine-tuning. The 2023 paper that introduced it demonstrated that QLoRA fine-tunes of LLaMA matched or outperformed models fine-tuned with other methods at similar scales. This democratized fine-tuning, enabling practitioners without access to GPU clusters to adapt frontier-scale models.
On-Device LLMs
Quantization is what enables on-device AI. Apple's iPhone 15 Pro and M-series Macs run 3B, 8B parameter quantized models natively. Samsung's Galaxy AI features use on-device quantized models for certain tasks. Microsoft's Phi-3 Mini at 3.8B parameters is designed to run on edge devices with 4-bit quantization.
On-device inference eliminates network latency, preserves privacy (data never leaves the device), and enables offline operation. The trade-off is capability: a 7B quantized model is genuinely useful but doesn't match a frontier 70B model. The sweet spot for on-device is well-defined, bounded tasks where smaller models are adequate.
Read next
Training vs Inference: Two Phases of an LLM's Life
Understand the difference between training an LLM (creating it) and inference (using it), including what happens at each stage, the costs involved, and why they matter for builders.
Open-Weight vs Open-Source Models: What's the Difference?
Why 'open-source AI' is often a misleading term, and what it actually means when a model is open-weight, what's included, what's not, and why it matters for developers.
Mixture of Experts: How LLMs Scale Efficiently
The architecture behind GPT-4, Llama 4, and Mistral, where only a subset of model parameters are active per token, enabling huge capacity at manageable inference cost.