Home/Benchmarks
Reference · Benchmarks
Comparisons and Numbers
Every number on this page is illustrative. They are constructed to show the shape of the tradeoffs, not to report measurements, and they should not be cited as results.
The tables below contain invented, illustrative figures. This is a demonstration reference site; we do not run a benchmark harness and these are not measurements of any real model, kernel or device. Use them to understand which quantities trade against which, and then measure your own.
Storage, exactly
This is the one table that is not illustrative. Bit widths are arithmetic and can be stated exactly for a given block or group size.
| format | element bits | metadata | bits/element | vs bf16 |
|---|---|---|---|---|
| bf16 | 16 | none | 16.00 | 1.00x |
| fp8 E4M3 | 8 | none | 8.00 | 2.00x |
| MXFP8 E4M3 | 8 | E8M0 per 32 | 8.25 | 1.94x |
| INT8 per-channel | 8 | fp16 per row | ~8.01 | 2.00x |
| MXFP4 E2M1 | 4 | E8M0 per 32 | 4.25 | 3.76x |
| INT4 group-128 | 4 | scale + zero per 128 | ~4.25 | 3.76x |
| NF4 group-64 | 4 | fp8 scale per 64, double-quantized | ~4.13 | 3.87x |
Illustrative quality comparison
What follows is a sketch of how these formats typically rank relative to each other on weight-only quantization of a mid-size decoder. The ordering reflects the general pattern reported across the literature; the specific values are invented.
| format | Δ perplexity | Δ accuracy | calibration needed | native HW path |
|---|---|---|---|---|
| bf16 baseline | 0.00 | 0.000 | no | yes |
| MXFP8 E4M3 | +0.01 | -0.001 | no | yes |
| INT8 per-channel | +0.02 | -0.002 | no | yes |
| MXFP4, round-to-nearest | +0.41 | -0.021 | no | recent only |
| MXFP4 + compensation | +0.14 | -0.008 | yes | recent only |
| NF4 group-64 | +0.19 | -0.011 | no | no |
| INT4 group-128 + GPTQ | +0.16 | -0.009 | yes | no |
| INT4 group-128 + AWQ | +0.13 | -0.007 | yes | no |
The pattern worth taking from this, rather than the numbers: at 4 bits, the well-tuned INT4 schemes and MXFP4-with-compensation land in broadly the same band. Naive round-to-nearest MXFP4 is meaningfully worse than any of them. MXFP4's distinguishing advantage is not accuracy — it is that a block-scaled 4-bit format can be consumed natively by hardware, which INT4 group schemes with fp16 scales and zero-points generally cannot.
Illustrative throughput shape
Two regimes behave completely differently. Single-stream decode is memory-bandwidth bound, so storage width dominates. Large-batch prefill is compute bound, so only a native arithmetic path helps.
| configuration | decode, batch 1 | prefill, batch 64 | weight memory |
|---|---|---|---|
| bf16 | 1.00x | 1.00x | 1.00x |
| MXFP4, storage only + bf16 compute | 2.4x | 1.0x | 0.27x |
| MXFP4, emulated compute | 2.6x | 1.0x | 0.27x |
| MXFP4, native block-scaled MMA | 2.8x | 3.1x | 0.27x |
If your workload is single-stream decode, you get most of the benefit from MXFP4 without any special hardware, because you are buying bandwidth rather than arithmetic. If your workload is batched prefill or training, the format is only interesting on hardware with a native path.
When not to use MXFP4
A benchmark page that only shows wins is a marketing page. These are the situations where the honest answer is a different format.
- Small models. Below roughly a few billion parameters there is less redundancy to spend and the same relative error hurts more.
- No native path and compute-bound. You pay accuracy for a bandwidth saving you are not bottlenecked on.
- Numerically sensitive heads. Routers, gates, and anything feeding a hard argmax should stay wider.
- Already-compressed models. Aggressively distilled or pruned checkpoints have had their slack removed already.
- Training in 4 bits without a scheme designed for it. Gradient dynamics are a different problem from weight storage, and naive MXFP4 gradients diverge.
- When 8 bits fits. MXFP8 or INT8 costs twice the memory and is close to lossless. If it fits, take it.
Measured results
Have real numbers? We would rather host measured results with a reproducible method than illustrative ones. If you have a harness and a device, get in touch.