Format · Ecosystem
Kernel Support
What an MXFP4 matmul actually does, why unpack scheduling decides the speedup, and the packing conventions nobody agrees on.
The shape of an MXFP4 GEMM
A weight-only MXFP4 matmul looks like a normal tiled GEMM with an unpack stage inserted. Activations arrive in bf16 or fp16. Weight tiles arrive as packed nibbles plus a scale vector, are expanded in shared memory or registers, and are then fed to the matrix unit. Accumulation is fp32.
The interesting engineering is in hiding the unpack. Each 32-element block needs one scale load and a shift-and-mask per pair of elements, and if that work is not overlapped with the global loads it shows up directly in the critical path. Good kernels unpack one stage ahead; naive ones serialise and end up slower than bf16 despite moving a quarter of the bytes.
# Two 4-bit codes share one byte. Layout is a convention, not a spec mandate -
# check what your kernel expects before you write a checkpoint.
def pack_nibbles(codes):
lo = codes[0::2] & 0x0F
hi = codes[1::2] & 0x0F
return (hi << 4) | lo # element 0 in the low nibble
# Storage cost per block of 32, in bits:
# 32 elements x 4 bits = 128
# 1 shared E8M0 scale x 8 bits = 8
# ------------------------------------
# total = 136 -> 4.25 bits/elementNative versus dequant-in-register
On hardware with a native block-scaled matrix instruction, no expansion happens at all: packed operands and the E8M0 scale go straight into the matrix unit. The kernel becomes simpler and the register pressure drops sharply, which usually matters more than the instruction count.
On everything else, the dequant-in-register approach is the practical answer. It is portable, it still captures the bandwidth win, and it composes with existing autotuned GEMM templates. The catch is that its performance is extremely sensitive to tile shape, because the unpack cost scales with the weight tile and the arithmetic does not.
| kernel | weight bytes read | rel. latency | notes |
|---|---|---|---|
| bf16 reference | 33.6 MiB | 1.00 | bandwidth bound |
| MXFP4, naive unpack | 8.9 MiB | 0.71 | unpack on critical path |
| MXFP4, pipelined unpack | 8.9 MiB | 0.34 | unpack overlapped with loads |
| MXFP4, native MMA | 8.9 MiB | 0.29 | no expansion stage |
Conventions checkpoints disagree about
Every one of these is a real interoperability failure people hit, and none of them is fixed by the numeric specification.
- Nibble order. Whether element 0 lives in the low or high nibble of the shared byte.
- Scale storage. Raw E8M0 bytes, or float32 powers of two, or a log-domain int8 with a different bias.
- Scale layout. A separate tensor of shape
[rows, k/32], or interleaved with the weight blocks for locality. - Transpose. Whether the packed tensor is stored row-major over the output dimension or pre-transposed for the kernel.
- Excluded layers. Which modules were left in higher precision, and whether that is recorded in the checkpoint metadata or only in the conversion script.
Ship a tiny known tensor alongside any MXFP4 converter — a handful of blocks with hand-checked values and their expected dequantized output. It turns an afternoon of confusing accuracy debugging into a one-line assertion.
Found an error, or a result that disagrees? This is a community reference. Corrections with a reproducible test case are the most useful thing you can send us.