Quantization
Technical InfrastructureReducing the precision of AI model weights (from 32-bit to 8-bit or 4-bit) to make models smaller and faster while sacrificing some quality.
Quantization compresses AI models by using less precise numbers. A model stored in full precision (FP32) uses 32 bits per parameter. Quantizing to INT8 uses 8 bits (4x smaller), and INT4 uses 4 bits (8x smaller). The model becomes smaller, loads faster, and runs with less VRAM.
This is crucial for running large models locally. Llama 70B in full precision needs ~140GB VRAM — impossible on consumer hardware. Quantized to 4-bit, it needs ~35GB — runnable on an M2 Ultra MacBook or dual RTX 4090 setup.
The quality tradeoff depends on the quantization method and level. Modern quantization (GPTQ, GGUF, AWQ) loses less than the compression ratio suggests, but how much less depends on the model, the method and the task, and the published figures vary widely. Treat any single retention percentage as a claim about one benchmark rather than a property of 4-bit. Very low precision (2-3 bit) shows noticeable degradation.
Real-World Example
A 4-bit quantized 70B model runs on a laptop that could not hold the full-precision weights at all. What it costs you in quality is the thing to measure on your own task, not to read off a number.
Related Terms
Try AI Humanizer
Rewrite AI-drafted text so it reads naturally — varied sentence rhythm, plainer phrasing, fewer stock transitions.
Try FreePut this concept to work
Once the definition is clear, the next useful move is to try a focused tool flow instead of bouncing through more glossary pages.
Open the humanizer routeFAQ
What is Quantization?
Reducing the precision of AI model weights (from 32-bit to 8-bit or 4-bit) to make models smaller and faster while sacrificing some quality.
How is Quantization used in practice?
A 4-bit quantized 70B model runs on a laptop that could not hold the full-precision weights at all. What it costs you in quality is the thing to measure on your own task, not to read off a number.
What concepts are related to Quantization?
Key related concepts include VRAM (Video RAM), GPU (Graphics Processing Unit), Parameters, Self-hosting, Inference. Understanding these together gives a more complete picture of how Quantization fits into the AI landscape.