Home/Format/Block Scaling

Format · Specification

Block Scaling

One 8-bit power-of-two exponent per 32 elements. How it is chosen, what it recovers, and the outlier problem it cannot solve.

Why a shared exponent at all

A 4-bit float on its own is useless for neural network weights. E2M1 spans 0.5 to 6.0 in magnitude — a dynamic range of about 3.6 binades. Real weight tensors span far more than that, and the distribution is not centred anywhere convenient.

Block floating point solves this by refusing to commit to a global range. Instead of one scale for a tensor, MXFP4 uses one scale for every 32 contiguous elements along the reduction axis. Each block gets to sit in whatever binade suits it. A tensor whose columns differ by six orders of magnitude is no harder to represent than a uniform one, because no block ever has to hold both extremes.

Choosing the scale

The standard rule is amax-based and conservative: take the largest absolute value in the block, find its binade, and set the shared exponent so that value lands at or below the top of the E2M1 range without saturating.

# The shared exponent is chosen so the block maximum lands at the top of the
# E2M1 range without ever saturating. floor() is what guarantees that.
def choose_scale(block):
    amax = np.max(np.abs(block))
    if amax == 0:
        return 127                        # 2^0, an all-zero block
    e = np.floor(np.log2(amax)) - 2   # 2 = max exponent in E2M1
    return int(np.clip(e, -127, 127)) + 127

# A search-based alternative: try a few exponents, keep the lowest MSE.
def choose_scale_mse(block, span=3):
    base = choose_scale(block)
    best, best_e = float("inf"), base
    for e in range(base - span, base + 1):
        err = mse(block, dequantize_block(encode(block, e), e))
        if err < best:
            best, best_e = err, e
    return best_e                        # offline only - too slow for a kernel

Because floor is used rather than rounding, the largest element is guaranteed representable and nothing in the block saturates. The cost is that you sometimes waste up to one binade of the available range: if the block maximum is just above a power of two, the top of the E2M1 grid sits well above anything actually present.

The outlier problem

One large value in a block drags the shared exponent upward, and every other element in that block is then quantized on a coarser grid. With 32 elements, a single outlier degrades 31 neighbours. This is the dominant failure mode of MXFP4 in practice and it is why block size and outlier handling are the two levers that matter most.

Transformer activations are notorious here: certain channels carry consistently large magnitudes, and they persist across tokens and layers rather than appearing randomly. Weight tensors are usually better behaved, which is a large part of why weight-only MXFP4 is common and full activation MXFP4 is not.

• Smaller blocks confine an outlier to fewer neighbours, at the cost of more scale bytes — block 16 costs 4.5 bits/element instead of 4.25.
• Rotation or Hadamard transforms spread outlier energy across a channel group before quantizing, which flattens the per-block amax.
• Mixed precision keeps a small set of identified outlier channels or whole layers in a wider format.
• Reordering groups similar-magnitude channels into the same block, though this usually costs you a permuted matmul.

Block axis matters

Blocks must run along the reduction dimension of the matmul, because that is the axis a dot product consumes. If you quantize along the wrong axis the numbers are still valid MXFP4, but the kernel cannot apply one scale per accumulation chain and you lose the hardware path entirely. When a converted checkpoint is unexpectedly slow, this is the first thing to check.