Home/Guides/Quantizing Your First Model

Guide · 30 minutes

Quantizing Your First Model

A weight-only MXFP4 pass from a bf16 checkpoint, with the exclusions, checks and error bands that make the result trustworthy.

Decide the scope first

The single most common mistake is quantizing everything. Start weight-only, leave activations in bf16, and exclude the layers that are known to be fragile. That configuration is where almost all of the memory win lives and almost none of the risk.

• Quantize: the large linear projections — attention Q/K/V/O and the MLP up/gate/down matrices. These are the overwhelming majority of parameters.
• Leave alone: embeddings and the output head, all normalisation parameters, biases, and any router or gating layer.
• Decide later: the first and last transformer block. Both are frequently more sensitive than the middle of the stack.

Measure a baseline you believe

Before touching the weights, run your evaluation twice at bf16 with different seeds and record the spread. If the two runs differ by more than the degradation you would consider acceptable, your harness cannot detect quantization damage and everything after this point is guesswork.

Run the conversion

The pass itself is short. Iterate parameters, skip the excluded ones, round-trip each weight along the reduction axis with block size 32, and record the per-layer error as you go so you have the numbers when something looks wrong later.

# A minimal weight-only MXFP4 pass with a per-layer error report.
for name, p in model.named_parameters():
    if not is_linear_weight(name):
        continue                          # leave norms, embeddings, lm_head alone
    w  = p.detach().float().cpu().numpy()
    wq = roundtrip(w, k=32)
    err = np.linalg.norm(w - wq) / np.linalg.norm(w)
    print(f"{name:48s}  rel_fro={err:.4f}")
    if err > 0.10:
        warn("outlier-heavy layer - consider keeping this one in 8-bit")

Two practical notes. Do the arithmetic in fp32 even if the weights are bf16 — log2 of a bf16 amax can land on the wrong side of a binade boundary and silently shift a whole block. And verify the reduction axis on a layer with non-square weights, because a transposed convention will quantize along the wrong dimension without raising an error.

Verify before you evaluate

Evaluation is slow and its failures are ambiguous. Cheap structural checks first: they catch most conversion bugs in seconds.

• Every scale byte is in range and none is 0xFF unless you intended a NaN block.
• The packed tensor is exactly half the element count in bytes, plus k/32 scale bytes per row.
• Round-tripping a block twice is idempotent — quantizing an already-quantized tensor must be a no-op.
• Per-layer relative Frobenius error is in a plausible band, roughly 0.03 to 0.09 for well-behaved linear weights.
• A single forward pass on a fixed prompt produces finite logits with a sane maximum.

Then what

Run the evaluation. If it lands within your measured noise band, you are done and you have a model roughly a quarter the size. If it does not, go to the troubleshooting guide rather than reaching immediately for calibration — most first-pass failures are configuration errors, not numerical ones.