Home/Guides/Troubleshooting Quality Loss
Guide · 25 minutes
Troubleshooting Quality Loss
An ordered decision tree for a quantized model that came out worse than it should have, including the cases where you should stop.
Work in this order
Quantization bugs look like quantization damage, and they are much more common. Rule out configuration before touching numerics, or you will spend a day tuning a calibration set to compensate for a transposed tensor.
- Confirm the baseline. Re-run bf16 on the same harness, same seed, same prompts.
- Confirm the reduction axis. Blocks must run along the contracted dimension.
- Confirm the exclusion list. Head, embeddings, norms, routers should all be untouched.
- Confirm the round trip is idempotent. Quantizing twice must be a no-op.
- Only now look at per-layer error and calibration.
Symptom to cause
| symptom | most likely cause | check |
|---|---|---|
| Output is NaN or all one token | NaN scale, or wrong axis | scan scale bytes for 0xFF |
| Perplexity fine, generation degenerate | lm_head quantized | verify exclusion list |
| Degrades with longer context | accumulator precision | force fp32 accumulate |
| One task collapses, others fine | a router or gate quantized | exclude gating layers |
| Uniformly ~2x worse everywhere | block axis transposed | re-check weight layout |
| Slower than bf16 | unpack on critical path | profile the kernel |
| Fine on GPU A, broken on GPU B | packing convention mismatch | round-trip a known tensor |
Fixes in order of cost
Once the configuration is genuinely correct and the model is still worse than acceptable, work up this ladder. Each step costs more than the last, in engineering time or in model size.
- Exclude the top few sensitive layers. Usually two to five layers, often the first block, the last block and the output projection. Costs almost nothing in size.
- Add error compensation. A GPTQ-style pass with 128 calibration sequences. Costs tens of minutes and no model size.
- Drop to block 16 on flagged layers. Costs 0.25 bits/element on those layers and probably the native kernel path.
- Apply a rotation. Hadamard or learned rotations flatten outlier structure before quantizing. Costs implementation effort and a small runtime overhead.
- Mixed precision. Keep the worst tensors in MXFP8 or bf16. Costs model size, but the size cost is concentrated where it buys the most.
- Stop. Some models should not be at 4 bits.
Knowing when to stop
There is a real category of model where MXFP4 is the wrong choice and no amount of tuning changes that. Small models are the clearest case: a 1B parameter model has far less redundancy to spend, and the same relative error that a 70B model shrugs off can be visible in output quality.
Others: models already distilled or pruned aggressively, models fine-tuned narrowly enough that the surviving capability is fragile, anything where the router or gating is doing a lot of work, and any deployment where the memory saving is not actually the binding constraint. If you are compute-bound on hardware without a native path, MXFP4 buys you very little and costs you accuracy.
If your bf16 evaluation was never that good, quantization will be blamed for problems it did not cause. A degradation you cannot distinguish from run-to-run noise is not evidence of success either — it is evidence that the harness is not sensitive enough to tell you anything.
Found an error, or a result that disagrees? This is a community reference. Corrections with a reproducible test case are the most useful thing you can send us.