Aggressive Quantization: How Low Can You Go?

Key Takeaways

• 4-bit & 8-bit quantization is the new baseline: My tests on models of 7B parameters and up show that the 4-bit and 8-bit regimes offer the closest thing to a free lunch in this field. The average performance drop on MMLU-PRO was less than two points for 4-bit and functionally negligible for 8-bit. A larger, more capable model that once demanded a cluster can now operate effectively on a single A100 or even a 24GB consumer GPU.
• Your choice of method is tied to your inference stack: The choice of a quantization algorithm is not an abstract decision; it is deeply coupled with the mechanics of your inference stack. GPTQ and AWQ come with fast CUDA kernels, making them brutally efficient. Bitsandbytes offers flexibility. HQQ and AutoRound, however, currently suffer from more limited support in high-performance serving frameworks. The lesson is clear: align your quantization method with the optimizations of your serving layer (e.g., Transformers, vLLM, TensorRT-LLM), or you will pay the price in latency.
• Sub-4-bit quantization is still the frontier: Venturing below 4-bits with 2-bit or 3-bit models is to step into a more experimental, volatile territory. While these models can show promise on constrained tasks like closed-book Q&A, they often exhibit catastrophic degradation on generative benchmarks like IFEval. My counsel is to approach these ultra-low bitrates only with rigorous, task-specific evaluation and to budget for multiple quantization runs to find a stable configuration.

Introduction

The esoteric art of quantization has shed its academic skin. It is no longer a niche pursuit but a democratizing force, a fundamental tool for deploying state-of-the-art Large Language Models (LLMs) on hardware that exists outside the rarefied air of hyperscale data centers. The core idea is a controlled demolition of precision, reducing model weights from 16-bit or 32-bit floating-point representations to lower-bit integers. This is often achieved by mapping high-precision weights to a discrete set of values using a learnable scaling factor:

where is the scale. The result is a dramatic reduction in the model’s memory footprint, bandwidth needs, and ultimately, the cost of inference. Since late 2023, a new generation of quantization algorithms has emerged, pushing below the 4-bit barrier without the immediate, catastrophic collapse in model quality that we once expected.

In this article, I’ll dissect the five quantization methods I turn to most often in my production workflows. I will present benchmark data for several instruction-tuned Llama 3.x and Qwen 2.5 models and conclude with some hard-won engineering principles from my time in the trenches. All data and scripts are grounded in a public repository, with the results tabled below drawn directly from that codebase. My objective is to provide a pragmatic guide for navigating the increasingly dense and complex terrain of model quantization.

The contenders at a glance

Here’s a quick overview of the quantization methods I evaluated:

*Support in vLLM is nascent and should be treated as experimental.

Results

Here’s how the different quantization methods performed for IFEval.

1. Instruction following (IFEval)

IFEval is ruthless. It tests a model’s capacity to follow nuanced instructions, revealing fractures in reasoning and generation that multiple-choice benchmarks often mask.

Observation: IFEval exposed critical differences. AutoRound, despite its sophisticated gradient-based approach, showed a troubling instability, sometimes causing severe performance degradation even at 8-bit. This suggests that its practical application on large, instruction-tuned models can be a high-variance affair. If you use AutoRound, re-running the quantization with different seeds or hyperparameters is not optional; it’s a necessity. GPTQ and AWQ, by contrast, were consistently robust. HQQ’s lower-bit variants (2/3-bit) struggled profoundly on this generative benchmark.

2. Memory & speed

The dividends of quantization become starkly tangible in memory and speed. A 4-bit Qwen 32B model, for instance, occupies about 19GB of VRAM. In my tests, this quantized model consistently outperformed the full-precision FP16 Qwen 14B on both MMLU-PRO and IFEval. When served via vLLM on an A100 GPU, it achieved an inference speed of roughly 22 tokens/second, more than double the ~10 tokens/second of the FP16 14B model. We are running a superior model, faster, on the same hardware, purely by being smarter about our bits.

Engineering insights

Based on my experiments, here are the practical principles I’ve distilled:

1. Start with AWQ or GPTQ for production deployments: These methods offer the most reliable path to production. They represent a well-understood balance of fast inference kernels, consistent results, and mature integration into serving frameworks like vLLM.
2. Bitsandbytes is your Swiss Army knife for experimentation: For rapid prototyping and discovery, the ease of loading models “on-the-fly” with bitsandbytes is invaluable. It is, however, a tool of convenience, not of ultimate performance. For sustained, high-throughput serving, its performance lags behind pre-quantized models paired with specialized kernels.
3. HQQ excels at model storage, but performance costs extra: The minimal metadata of HQQ-quantized models is compelling if disk space or distribution bandwidth are primary constraints. However, its default reliance on generic matrix multiplication is a bottleneck. To unlock competitive inference speeds, one must often descend into the stack and compile custom GEMM kernels.
4. AutoRound requires a robust evaluation harness: The sensitivity of AutoRound to initial seeds and hyperparameters demands diligence. I recommend wrapping its execution in a simple loop that retries quantization upon failure and, critically, validates the resulting model on a generative benchmark before it’s allowed anywhere near a production environment.
5. Group size is a critical lever, especially at the bleeding edge: My initial forays into 2-bit quantization produced incoherent gibberish. Accuracy only became tolerable after tightening the group size from a default of 128 down to 32 (or 64, depending on the method’s convention-a smaller number means more groups). This trade-off significantly increases metadata size but is often the only way to salvage performance at these low bitrates.
6. Use bfloat16 as a pragmatic escape hatch in vLLM: If you encounter numerical instability or NaN errors with very low-bit models (especially 2 or 3-bit) in vLLM, forcing the dtype to bfloat16 can often restore stability. It comes at a modest throughput cost of around 5%, a small price to pay for a working model.

Closing thoughts

The terrain of LLM quantization has shifted beneath our feet over the past eighteen months. What was once a delicate art, confined to research labs, has matured into a more routine, if still nuanced, engineering discipline. My primary takeaway is this: 4-bit quantized versions of premier models like Llama 3 and Qwen 2.5 should be your new baseline. For a vast array of use cases, they effectively obsolete their FP16 counterparts, offering an irresistible combination of memory savings and speed with negligible performance loss.

Pushing below the 4-bit threshold is where the map becomes less reliable. The potential gains are significant, but so is the volatility. If you venture into 2-bit or 3-bit territory, I urge you to proceed with caution and a rigorous evaluation framework that prioritizes generative capability over simple multiple-choice accuracy. And, most importantly, budget the time and compute for retries. You will likely need them.

I continue to explore these frontiers, including deeper dives into GGUF and other emerging methods. As always, the field moves at a relentless pace. Experimentation and sharing what we learn are our best tools for navigation. If you have your own insights or challenges, feel free to connect with me on X/Twitter; I’m always open to comparing notes from the front lines.

Happy quantizing!