Quantization is the technique that shrinks an AI model by storing its internal numbers with less precision — trading a small amount of accuracy for a large cut in memory and computing power, which is how multi-billion-parameter models end up running on a phone instead of a data center.
From Floating Point to Integers
Every weight inside a trained large language model is just a number — typically stored as a 32-bit or 16-bit floating-point value, a format built to represent a huge range of decimals precisely. That precision is expensive: a 7-billion-parameter model stored at 16-bit precision needs roughly 14 gigabytes of memory just to hold its weights, before it has answered a single question.
Quantization replaces those 16- or 32-bit floats with much smaller numbers — 8-bit integers, 4-bit integers, or in the most aggressive cases, a single bit per weight. Cutting a number’s precision in half roughly halves the memory it needs and the amount of data that has to move between memory and the processor, which is usually what determines how fast a model responds.
How the Compression Actually Works
To quantize a model, each weight’s range of possible values is mapped onto a much smaller set of discrete values, then every original weight is rounded to the nearest one. A weight that used to be one of billions of possible decimal values might now be one of only 16 (at 4-bit) or even one of 2 or 3 (at 1-bit or “ternary” precision).
This can happen after training is finished — post-training quantization (PTQ) — which is fast and requires no extra training data. Or it can happen during training itself — quantization-aware training (QAT) — where the model learns to compensate for the rounding as it trains, usually producing a more accurate result at very low bit-widths, at the cost of extra training time.
Most of the popular open-weight models distributed today ship in a format called GGUF, developed for the llama.cpp project, which packages a model at a chosen precision level — labeled things like Q8_0, Q4_K_M, or Q2_K — so it can run efficiently on ordinary laptops and phones rather than data-center GPUs. Readers who want to try quantizing a model themselves can do so with free, open-source tools.
The Tradeoff: Smaller Isn’t Free
Quantization is not lossless. The more aggressively a model is compressed, the more its answers can drift from the original — dropping precision below 4 bits per weight has historically caused a noticeable decline in accuracy on tasks that involve careful reasoning or arithmetic. Labs manage this by calibrating the rounding on representative data, quantizing only the least-sensitive layers aggressively, or, as with QAT, training the model to be robust to the loss from the start.
The technique has kept advancing: some newer open-weight releases now use 1-bit or ternary (3-value) weights, which used to be considered too lossy to be useful, while still claiming to retain most of a model’s original benchmark performance.
Why It Matters
Quantization is a major reason on-device AI has become viable at all. A model that only fits on a rack of data-center GPUs cannot run offline, cannot guarantee a user’s data never leaves their device, and costs money every time it answers a question in the cloud. A quantized model that fits in a few gigabytes of memory can run locally, offline, and for free after it is downloaded — the tradeoff labs are willing to make is a small amount of accuracy for a large gain in where, and how cheaply, a model can run.
In the news
PrismML’s Bonsai 27B, released as 1-bit and ternary versions of a 27-billion-parameter Qwen-based model, is a recent example of just how far this compression can go — shrinking a model that would normally need tens of gigabytes down to under 6 gigabytes, small enough to run on a modern phone.
FAQ
Does quantization make a model “dumber”? It can, especially at very low bit-widths, but well-calibrated quantization at 8-bit or 4-bit typically keeps most of a model’s original performance intact.
Is quantization the same as model distillation? No. Distillation trains a smaller model to imitate a larger one; quantization keeps the same model but stores its existing weights with less precision.
Can I quantize a model myself? Yes — tools like llama.cpp and Hugging Face’s libraries can quantize many open-weight models on a personal computer, though it takes some technical setup.
Sources: IBM, “What Is Quantization?”; Hugging Face Optimum quantization guide; llama.cpp quantize documentation.