Home/Guides/Choosing Block Size
Guide · 15 minutes
Choosing Block Size
One scale per k elements. The overhead is exact arithmetic; the benefit depends entirely on how your outliers are distributed.
The arithmetic
Block size is a straight trade between scale overhead and outlier containment. The overhead is exact and easy to compute: one 8-bit scale per k elements adds 8/k bits per element.
| block size k | bits/element | vs bf16 | scales per 4096-row |
|---|---|---|---|
| 8 | 5.00 | 3.20x | 512 |
| 16 | 4.50 | 3.56x | 256 |
| 32 | 4.25 | 3.76x | 128 |
| 64 | 4.125 | 3.88x | 64 |
| 128 | 4.06 | 3.94x | 32 |
Above 32 the returns are tiny — going from 32 to 128 saves 0.19 bits per element, about 4% of the total, while quadrupling the number of neighbours an outlier can damage. Below 32 the cost climbs quickly and, more importantly, you usually leave the hardware-native path.
32 sits at the knee of the overhead curve and matches the block size wired into native block-scaled matrix units. Choosing anything else is choosing to run an emulated kernel. That may still be the right call, but it should be a deliberate one.
The containment argument
The reason to consider a smaller block is entirely about outliers. One large value forces the shared exponent up and coarsens the grid for everything else in its block. With k = 32 that is 31 damaged neighbours; with k = 16 it is 15.
Whether that matters depends on how the outliers are distributed. If they are sparse and randomly placed, halving the block size roughly halves the number of affected elements. If they are systematic — a whole channel that is consistently large — halving the block size does almost nothing, because the outlier is present in every block along that channel anyway. Diagnose which case you are in before paying for smaller blocks.
Rules of thumb
- Start at 32. It is the default for good reasons and it is the only size with a reliable native path.
- Try 16 on specific layers that your sensitivity ranking flagged, not globally. Mixed block sizes across a model are usually fine as long as the runtime supports them.
- Do not go above 64. The remaining storage saving is negligible and the outlier exposure grows linearly.
- If 16 helps a lot, the real problem is outlier structure. A rotation or a mixed-precision exclusion will probably help more, and cost less.
- Check the axis before the size. A wrong reduction axis looks exactly like a block size problem and is far more common.
Found an error, or a result that disagrees? This is a community reference. Corrections with a reproducible test case are the most useful thing you can send us.