Home/Format/Quantization Error

Format · Analysis

Quantization Error

One rounding step, eight magnitudes, and a relative error profile that behaves nothing like INT4.

Where the error comes from

There is exactly one source of error in an MXFP4 tensor: rounding a value to one of eight magnitudes. The scale is a power of two and applies exactly. There is no accumulation of scale error, no drift, and no dependence on the order in which blocks are processed.

Because the E2M1 grid is geometric above 1.0 and linear below it, the relative error is bounded but not uniform. The worst relative error for a normal value sits at the midpoint between adjacent grid points in the widest binade: between 4.0 and 6.0 the spacing is 2.0, giving a worst-case relative error of about 20% for a value at 5.0. Between 1.0 and 1.5 the spacing is 0.5, so the worst case is about 20% again. Values below the smallest subnormal step round to zero outright.

The row that matters most is the first one. Small values relative to the block maximum are the ones that get destroyed — anything below a quarter of the smallest step rounds to zero. In a block containing one large outlier, most of the block ends up in that regime.

Measuring it honestly

Relative Frobenius error on the weight tensor is the cheapest signal and the easiest to over-trust. It tells you whether the encoding worked; it tells you almost nothing about whether the model still works. Two layers with identical Frobenius error can behave completely differently depending on how the error interacts with the activation distribution downstream.

• Relative Frobenius error — a sanity check. Above roughly 0.10 for a linear weight, look at the layer specifically.
• Per-block amax ratio — the ratio of block maximum to block median. High values flag the outlier-dominated blocks that will quantize badly.
• Layer output MSE under real activations — far more predictive than weight error, and cheap if you already have a calibration batch.
• End-to-end task metrics — the only thing that actually decides it. Perplexity moves before accuracy does, and both move before anything a user notices.

# A minimal weight-only MXFP4 pass with a per-layer error report.
for name, p in model.named_parameters():
    if not is_linear_weight(name):
        continue                          # leave norms, embeddings, lm_head alone
    w  = p.detach().float().cpu().numpy()
    wq = roundtrip(w, k=32)
    err = np.linalg.norm(w - wq) / np.linalg.norm(w)
    print(f"{name:48s}  rel_fro={err:.4f}")
    if err > 0.10:
        warn("outlier-heavy layer - consider keeping this one in 8-bit")

Accumulation through a network

Quantization error does not simply add up layer by layer. Residual connections dilute it; normalisation layers partly renormalise it away; attention softmax is relatively tolerant of small perturbations in logits, up to a point. In practice error growth through a deep transformer is closer to sublinear than linear, which is the main reason 4-bit weight quantization works at all.

What breaks that pattern is a layer whose output feeds something sharp. Router logits in a mixture-of-experts model are the classic example: a small perturbation flips an expert selection, and the output changes discontinuously rather than slightly. Layers like that should generally stay in a wider format regardless of what the average error metric says.

How this differs from INT4

INT4 with a per-group scale distributes its error uniformly in absolute terms across the group. MXFP4 distributes it geometrically: large values get large absolute error and small relative error, small values get the reverse. Which is better depends on what the tensor looks like.

For roughly bell-shaped weight distributions, the two are much closer than the format debate suggests, and a well-tuned INT4 scheme with a learned or search-based scale frequently matches or beats naive MXFP4. MXFP4's advantages are structural rather than numerical: a hardware-native path on recent accelerators, an exact power-of-two scale, and no need for a zero-point or asymmetric range handling.