Guide · 20 minutes

Calibration

What calibration changes for a block-scaled format, how much data you need, and the failure it cannot rescue.

Wafer probe station needles contacting a test die under magnification — Probe needles on a test die.

What calibration is for

Calibration means running representative data through the model and using the observed activations to make better quantization decisions. For MXFP4 specifically, it is worth being precise about what it can and cannot change, because the format constrains this more than people expect.

The shared scale is derived from the block's own values. For weights, those values are fixed and known before any data is seen, so a weight-only pass needs no calibration at all to produce valid MXFP4. Calibration enters through the side door: it tells you which layers matter, and it enables error-compensating methods that adjust the weights themselves.

Calibration does not change what MXFP4 can represent. It changes which representable value you choose, and which layers you decide not to quantize.

Three things it actually does

Sensitivity ranking. Push a calibration batch through, measure each layer's output MSE when quantized in isolation, and rank. This gives you a principled exclusion list instead of a superstitious one.
Error compensation. GPTQ-style methods use the calibration Hessian to adjust the not-yet-quantized weights in a layer to compensate for the error already introduced. The output grid is unchanged; the choice of grid point is improved.
Activation range selection. Only relevant if you are quantizing activations, where block scales must be chosen from observed data rather than from a static tensor.

Illustrative effect of calibration strategies on a weight-only MXFP4 pass. These numbers are invented for illustration and should not be cited.
strategy	calib. samples	Δ perplexity	cost
None (round-to-nearest)	0	+0.41	seconds
Sensitivity-ranked exclusions	64	+0.29	minutes
GPTQ-style compensation	128	+0.14	tens of minutes
Compensation + exclusions	128	+0.09	tens of minutes

Choosing calibration data

The received wisdom is that calibration data must match the deployment distribution. That is true for activation quantization and much weaker for weight-only methods, where a few hundred sequences of generic text usually gets you most of the benefit.

Quantity: 64 to 256 sequences is the usual plateau. More rarely helps; fewer than 32 makes the Hessian estimate noisy.
Length: use your actual context length, or at least a substantial fraction of it. Short sequences under-represent long-range attention patterns.
Diversity: matters more than domain match. A narrow calibration set can overfit the compensation to a distribution you do not deploy on.
Contamination: do not calibrate on your evaluation set. It is an easy mistake and it produces numbers that quietly do not generalise.

python calibrate.py --n 128 --seqlen 4096 --source mixedcollected 128 x 4096 tokens from 4 sourceslayer sensitivity (output MSE, quantized in isolation): layers.31.self_attn.o_proj 3.71e-3 <- highest layers.0.self_attn.q_proj 2.90e-3 layers.30.mlp.down_proj 1.44e-3 ... 221 layers below 1e-3

What it will not fix

Calibration cannot make a block hold more than eight magnitudes. If a tensor has a genuine outlier problem — a channel three orders of magnitude above its neighbours — no amount of calibration data changes the fact that the other 31 elements of that block are being quantized on a grid chosen for the outlier.

The fixes for that are structural: a smaller block size, a rotation that spreads the outlier energy, or keeping that tensor in a wider format. Calibration is a refinement on top of a workable configuration, not a rescue for an unworkable one.

Corrections

Found an error, or a result that disagrees? This is a community reference. Corrections with a reproducible test case are the most useful thing you can send us.

Open a correction · Community