How Quantization Works

How a thirty-gigabyte AI model gets rounded down to fit on a laptop, explained from scratch with no assumed background.

Take a language model with seven billion adjustable numbers in it, which is a fairly ordinary size today. Stored the way it came out of training, each number takes 4 bytes, so the model occupies 28 GB. A high-end consumer graphics card holds 24 GB. It does not fit, and that is before the model needs any working space to actually think in.

Store each number in 1 byte instead and the same model is 7 GB. In half a byte, 3.5 GB. Nothing about the model's design changed; we only changed how its numbers are written down. That is quantization, and what follows covers how it is done, why it works, and where it breaks.

The idea is not new to AI. Every device that records sound or images does the same thing: the real world offers infinitely many values, the storage offers a fixed list, and you round to the nearest entry on the list. What is new is the question of whether a system built from billions of finely tuned numbers can survive that treatment. Mostly it can.

␣The words you will need

Six terms do most of the work in this article. They are worth two minutes now so that the rest reads easily.

weight

One of the model's tuned numbers (one dial). A model "with 7 billion parameters" has 7 billion of these. They are fixed once training ends, which is what makes them the easiest thing to compress.

activation

A number the model calculates while running, on its way from your input to its answer. Weights are the recipe; activations are the food being cooked. Activations differ every time, so they are harder to plan for.

tensor

A block of numbers (a grid, or a stack of grids). A single layer's weights form one tensor. The word means nothing more alarming than "a big array".

layer

One step of the model's work. Numbers come in, get multiplied against a tensor of weights, and numbers come out to pass along. A large model is dozens to hundreds of these in a row.

FP32 / FP16

The high-precision formats. FP stands for floating point, meaning a number with a decimal point that can sit anywhere. FP32 uses 32 bits (4 bytes) per number, FP16 uses 16 bits (2 bytes). Models are trained in these.

INT8 / INT4

The compressed formats. INT stands for integer, meaning whole numbers only, no decimal point. INT8 uses 8 bits (1 byte), giving 256 possible values; INT4 uses 4 bits, giving 16. This is what we round into.

01What you are actually trading

To see what gets lost, it helps to know what those 4 bytes were buying.

A floating-point number is stored the way scientists write very large and very small quantities: as some digits, plus a note about where the decimal point belongs. Written out, 0.0000642 becomes 6.42 × 10⁻⁵. The mantissa is the digits part (6.42), and it controls how precise the number is. The exponent is the −5, and it controls how large or small the number is allowed to be.

FP32 splits its 32 bits between the two: one bit for the sign, 8 bits for the exponent, 23 bits for the mantissa. Those 8 exponent bits are extraordinarily generous, letting a single format cover everything from 0.00000000000001 up to 100000000000000. The 23 mantissa bits then supply many significant digits at every one of those sizes. It is a format designed so that nobody ever has to think about it.

INT8 has none of this machinery. There is no exponent, no floating decimal point, no automatic adjustment. There are 256 evenly spaced values and that is the entire supply. So every decision in quantization reduces to one question: where on the number line do you spend those 256 values?

Two things make the loss survivable. The first is redundancy. A trained model has far more dials than it strictly needs, and training was itself a noisy process, so the model was never relying on any single weight being exact. Rounding is one more small disturbance of a kind it has already been hardened against.

The second matters more. You do not need the internal numbers to be correct, only the final answer: the right next word, the right label. A ranking is far more robust than the numbers producing it, and small errors in the middle rarely reach the top of the list.

The payoff arrives in two forms. The obvious one is arithmetic speed. Integer multiplication is simpler to build in silicon than floating-point multiplication, so a chip can pack in more of it and run roughly twice as many operations per second in INT8 as in FP16.

The less obvious one usually matters more, and it concerns moving data rather than doing math. When a model writes text it produces one word at a time, and for every single word it must pull all of its billions of weights out of memory and into the processor. Reading that much data is slow enough that the processor spends most of its time idle, waiting for the next batch to arrive. Cut each weight from 4 bytes to 1 and there is a quarter as much to read. The model gets faster not because the math sped up but because there was less to carry.

02The two numbers that define a ruler

To build the ruler from Section 00 you need exactly two facts about it. Everything else follows from those.

The first is the scale, written s, which says how much one tick mark is worth. If the ruler covers a span of 3.2 using 256 marks, each mark is worth roughly 0.0125. Scale is the size of the gap you have agreed to lose to rounding.

s = ( r_max − r_min ) / ( q_max − q_min )

So the top of the fraction is how much real ground the ruler covers, and the bottom is how many gaps you have to cover it with. Dividing gives the size of one gap.

The second fact is the zero-point, written z, which says which tick mark sits at real zero. If your numbers run from −0.8 to 2.4, zero is not in the middle (it is a quarter of the way along), so it lands on tick 64 rather than tick 128. The zero-point is just an offset recording where the ruler was laid down.

z = q_min − round( r_min / s )

With scale and zero-point in hand, converting in either direction is a single line of arithmetic. To compress a real number x into a tick number q:

q = clamp( round( x / s ) + z,  q_min,  q_max )

To go back the other way and recover an approximate x:

x̂ = s · ( q − z )

One detail in the setup is not cosmetic. The zero-point is deliberately forced to be a whole number, which guarantees that real 0.0 lands exactly on a tick rather than between two of them. Zero earns this special treatment because models produce so much of it: certain very common operations output zero roughly half the time, and images get padded with zeros around their borders. If zero were represented as 0.006 instead, that tiny error would show up millions of times over and quietly push everything off-centre.

Inside the ruler's span, the worst you can be wrong by is half a tick. That error is small, and just as likely to fall high as low, so it largely cancels out across millions of values. Outside the span, values get pinned to the end and the error can be arbitrarily large. Which brings up the real question (how long should the ruler be?), though one simplification comes first.

03Two ways to lay the ruler down

If you centre the ruler on zero, giving it equal reach in both directions (say −2 to +2), then zero sits at the middle tick, the zero-point becomes 0, and it drops out of the arithmetic entirely. This is symmetric quantization, and the saving is real: with the offset gone, converting is a single multiplication, and a chip performing millions of these per layer notices the difference. Leave the ruler off-centre (say −0.5 to 3.5) and you have asymmetric quantization, which costs a little extra bookkeeping in every calculation.

Which one to use depends on the shape of what you are measuring, and the two kinds of number in a model have different shapes.

Weights come out of training scattered fairly evenly around zero, positive and negative in roughly equal measure. A centred ruler fits them naturally, so almost everyone uses symmetric quantization for weights and pockets the saving.

Activations are often one-sided. A very common operation inside models simply deletes negatives: anything below zero becomes zero, and everything above passes through unchanged. Downstream of that, there are no negative numbers at all. Centring a ruler there would waste half of your 256 ticks on values that can never occur, so the asymmetric version earns its keep by putting the whole ruler where the numbers actually live.

04Choosing the length of the ruler

This is the decision everything else hangs on, and it is a genuine trade-off with no free answer.

Make the ruler short and its 256 ticks pack tightly together, so ordinary numbers get recorded very accurately. But anything past the end gets pinned to the last tick, and large values are flattened into one another. Make the ruler long and nothing gets pinned, but the same 256 ticks are now spread thinly and every ordinary number is recorded crudely.

The obvious rule (measure the smallest and largest values present, then make the ruler exactly that long) turns out to be a trap. A single freak value can stretch the ruler enormously, after which every ordinary number pays for it. You have handed your entire precision budget to the least typical value in the batch.

Calibration is the name for settling this by experiment rather than guesswork. You feed the uncompressed model a few hundred sample inputs (no correct answers needed, just realistic inputs) and watch what numbers come out of each layer. That gives you a picture of the typical range, and you choose a cut-off from it.

There are three common ways to choose. Cut at a percentile, keeping the range that covers 99.9% of values and accepting that the top 0.1% get flattened. Cut where the average error is smallest, by trying many lengths, measuring how far off the rounded values end up, and keeping the winner. Or cut where the least information is destroyed, by trying many lengths and keeping whichever produces a rounded picture that best resembles the original shape of the data. The last of these is what NVIDIA's deployment toolkit does by default.

Weights need none of this. They sit right there in the file and never change, so their exact range is simply read off. Only activations, whose values depend on the input, require sample data to predict.

There is also the question of when the ruler gets sized. Static quantization decides once, during calibration, and bakes the answer in. It is fastest at run time because nothing needs measuring. Dynamic quantization re-measures the range as it goes, for each input, which costs a little speed but never gets caught out by an unusual input. That is why it is common for language models, whose numbers vary a lot depending on what you ask them.

05One ruler, or many?

Nothing requires a whole block of weights to share a single ruler. You can cut the block into pieces and give each piece its own scale and zero-point. This choice is called granularity, and it is quietly one of the biggest levers on final quality.

The reason is that real weight blocks are lopsided. Within one layer, a single row of weights might range over ±0.05 while the row beneath it ranges over ±1.2, a twenty-fold difference. Force them onto a shared ruler sized for the larger row and the smaller row's values all land on two or three ticks, which is very nearly the same as deleting them.

Common practice runs on a ladder. Per-tensor, meaning one ruler for the whole block, is usually fine for activations. Per-channel, meaning one ruler per row, is the universal default for weights, since it costs almost nothing. And once you push down to 4 bits, per-group: chop each row into runs of 64 or 128 values and give every run its own ruler. This is why a 4-bit model file comes out slightly larger than exactly one-eighth of the original, since those extra rulers have to be stored too. It is also the main reason 4 bits works at all. The smaller a ruler's jurisdiction, the less damage any one odd value can do.

06What the machine actually runs

A natural assumption is that a compressed model unpacks each number back to full precision, does the math normally, then repacks, which would make the whole exercise pointless. A properly compiled INT8 model does something better: the numbers stay compressed the entire way through, and the arithmetic happens on the whole numbers directly.

The one complication is that multiplying and adding makes numbers grow. Each layer multiplies thousands of pairs and adds all the results together, and a running total of thousands of products will not fit in the same 8 bits the inputs arrived in. So the total is kept in a larger container of 32 bits, which is cheap because there is only one total per output rather than one per input.

At the end of the layer that oversized total has to be squeezed back down to 8 bits, because the next layer measures things on a different ruler. Converting between two rulers is a single multiplication by a fixed factor:

M = ( s_x · s_w ) / s_y

Because every one of those tick sizes is known in advance, M is computed once at build time and stored as an integer shortcut. This step is called requantization, and it is the last remaining trace of the original decimal numbers anywhere in the running model.

There is a second, simpler approach that now dominates local LLM use: weight-only quantization. Here the weights are stored compressed but expanded back to decimals in the split second before each one is used, and the math itself runs the ordinary way. The arithmetic gains nothing. It is still a large win, because (as in Section 01) the bottleneck was never the arithmetic. It was hauling weights out of memory, and you have just cut that by three-quarters. This is what the Q4 in a downloaded model file usually means. It also conveniently sidesteps the hardest problem in the field, which is the subject of Section 08.

07Round afterwards, or train for it

Everything so far describes post-training quantization (usually abbreviated PTQ): take a finished model, calibrate it, round it, ship it. No retraining, no labelled data, and minutes to hours of work. At 8 bits it typically costs a fraction of a percent of accuracy, which is why it is always the first thing to try.

When that is not good enough, at 4 bits or on small models with no spare capacity to give away, the alternative is to warn the model in advance. This is quantization-aware training (QAT), and the idea is to show the model the rounding while it is still learning, so that it can adapt.

A brief detour on how learning works, since the next paragraph depends on it. The model makes a prediction, you measure how wrong it was, and then you calculate, for every single weight, which direction to nudge it in to make the error smaller. That calculation is the gradient, and repeating the loop millions of times is training.

QAT inserts a rounding step into the model while training continues. The model now experiences the exact errors it will face after deployment, and learning naturally shifts the weights somewhere that tolerates them, away from the awkward midpoints between two ticks where rounding could go either way.

This runs into an obstacle that sounds fatal. To nudge a weight you need to know how the error responds to changing it slightly, but rounding does not respond to slight changes. Nudge 21.4 to 21.41 and it still rounds to 21. As far as the training process can tell, nothing it does matters, so it learns nothing.

The fix is a deliberate lie. During training, the rounding step is told to pretend it passed its input through untouched, so that nudges appear to have their full effect. Rounding still happens in the forward direction where it counts; only the feedback signal ignores it. This is the straight-through estimator, it is not mathematically justifiable, and it works extremely well. One of the more productive fibs in the field.

08Why big language models needed new tricks

Large language models broke the recipe above in an interesting way. Past roughly six billion parameters, something odd appears in their activations: a handful of them, the same handful on every input, carry values a hundred times larger than everything else. This is not random noise but a fixed, structural quirk that shows up once models get big enough.

That is precisely the situation from Section 04 that ruins a shared ruler. A few enormous values force a very long ruler, and the thousands of ordinary values are then recorded so crudely that the model stops working. Nobody predicted this from earlier experience with smaller models. It simply emerged with scale, and the field reorganised around it.

The responses fall into three families.

Treat the outliers separately. The method known as LLM.int8() detects the troublesome channels and keeps just those at full precision, compressing everything else. A small carve-out for a small problem.

Move the problem somewhere it hurts less. SmoothQuant uses the fact that you can shrink one side of a multiplication and grow the other without changing the answer, so it scales the wild activations down and the weights correspondingly up. Weights handle a wide spread far better, so the difficulty gets shifted onto the party that can absorb it.

Do not compress activations at all. This is the weight-only route from Section 06, and it is where most local-model tooling has landed. Two methods dominate. GPTQ compresses weights one column at a time, and after each rounding decision it slightly adjusts the weights not yet processed to compensate, so that errors cancel one another instead of piling up. AWQ starts from the observation that a small fraction of weights matter far more than the rest, identifies them by watching which ones see the largest inputs, and protects those before rounding the others normally.

A fourth idea abandons even spacing altogether. Trained weights are not spread evenly: they cluster near zero and thin out towards the extremes, in the familiar bell-curve shape. So NF4 spaces its 16 available values unevenly to match, packing them tightly where most weights live and sparsely out in the tails. Same storage, better fit, and it is what makes 4-bit fine-tuning on a single consumer GPU practical.

Two further developments are worth knowing about. Newer GPUs support FP8, an 8-bit format that retains a little of floating point's self-adjusting behaviour, making it a gentler compromise that is increasingly used for training as well as for running models. And in long conversations the biggest consumer of memory stops being the weights at all: it becomes the model's running notes on everything said so far, called the KV cache. Compressing that has become just as important as compressing the weights.

09What to actually do

If you are running a language model on your own machine, the short version is to pick a 4-bit weight-only build with a group size of 128, which in practice means a file labelled something like Q4_K_M. It fits the most capable model into your memory for the least quality loss, and for most people it is simply the right default. Step up to 8-bit if you have memory to spare and want the last fraction of quality. Step below 4-bit only if nothing else fits, and expect to notice.

If you are deploying a model you built yourself, start with 8-bit post-training quantization: per-row rulers for the weights, calibrated rulers for the activations, and a few hundred realistic sample inputs. Measure it on your own task before anything else, because quantization does not fail loudly. Nothing crashes, and no error turns up in a log. The model simply gets a little worse in ways that only show up if you go looking.

The photograph remains the right intuition to hold onto. Recording sound and images works because a fixed palette, chosen sensibly, is indistinguishable from infinite precision to the person on the other end. Compressing a neural network is the same bet under stranger conditions: that a model tuned to the very edge of its ability mostly does not need the extra digits either. So far, the bet keeps paying.

∞Further reading