No amount of zeros makes it faster

Part three flattened the weights into int8: the values got coarser and the count stayed. There is an opposite move - keep the precision and cut the count, by setting some of them to zero.

Which ones get zeroed

Cut the same fraction and the outcome depends entirely on which. Compare cutting the smallest by magnitude against cutting at random.

fraction zeroed   magnitude (global)   magnitude (per layer)   random
            10%               2.0358                  2.0397   3.7873
            30%               2.2097                  2.3893   4.4976
            50%               3.1009                  3.3498   4.4543
            70%               4.0908                  4.4716   4.5714
            90%               5.0523                  5.0089   4.6292

The baseline is 2.0272. By magnitude it holds to 2.2097 at 30%; at random it is already 3.7873 at 10%.

Looking at the random side more finely shows how fast it goes.

 1%   2.2862
 2%   2.4815
 5%   3.1419
10%   3.7873
20%   4.2955

At 5% it is 3.1419, already worse than part one’s bigram at 2.6501, and by 20% it is 4.2955, up against the untrained 4.6052. Delete one weight in twenty at random and the model is worse than counting the previous character.

Magnitude is different because what it cuts is already near zero. Cutting at random removes large values with the same probability, and one of those shakes the whole next layer.

The table also shows that cutting globally beats cutting per layer: 2.2097 against 2.3893 at 30%. Layers have different distributions of magnitude, and forcing the same fraction on each means cutting large values out of the layers that only have large ones.

To match part three’s loss

Compared directly against quantisation: what sparsity does it take to reach each bit width’s loss?

8 bits (2.0280)   sparsity  1.8%
6 bits (2.0387)   sparsity 10.3%
4 bits (2.3571)   sparsity 35.5%

int8 is a 4x compression, and reaching its loss by pruning allows cutting 1.8%. Deleting 1.8% saves nothing.

At equal space, quantisation wins

Here is pruning’s trap. Zeroing half the weights does not halve the storage, because where the non-zeros are has to be stored too.

bytes per element
float32 dense        4.00
int8 dense           1.00      (part three)
float32 50% sparse   COO 4.00 / bitmask 2.12
float32 75% sparse   COO 2.00 / bitmask 1.12
float32 90% sparse   COO 0.80 / bitmask 0.52

With the common value-and-index layout (COO), cutting 50% saves nothing at all. A one-bit-per-element mask does better and still costs 2.12 bytes at 50%.

Comparing loss at equal compression settles it.

int8            4.00x compression   loss 2.0280
75% sparse      3.56x               loss 4.3935     (bitmask)
50% sparse      1.88x               loss 3.1009

int8 compresses more and loses 2.0280, while pruning at a similar ratio loses 4.3935 - worse than before training. In this model quantisation beats pruning at every compression ratio.

And it is not faster either

The other reason to prune is speed: if half the multiplications are by zero, surely half can be skipped. Measured:

The claim is about multiplication, so narrow it to one multiplication - the shape of the first feed-forward layer, (128 x 128) @ (128 x 512), round-robin over 400 rounds, divided against the dense original inside each round.

                 time (us)   vs dense   quartiles
dense original        73.4      1.000
50% sparse            73.6      1.010   0.957 ~ 1.105
90% sparse            73.2      1.013   0.924 ~ 1.147
99% sparse            74.0      1.005   0.850 ~ 1.142

Zeroing 99% changes nothing. All three quartile ranges contain 1.0, so there is no ground for saying a difference exists. A matrix multiply is still a dense matrix multiply, and multiplying by zero is still multiplying. Skipping would require checking where the zeros are, and that check costs more than the multiplication.

It is part three’s story again, where int8 saved memory and not time, and part four’s, where c = 1.059 made the 8-bit draft dearer than the target. Compression that touches the weights is a memory story, not a time one.

Getting time out of it needs the zeros in a regular pattern - two out of every four, say - so a dedicated instruction can skip them, which is what structured sparsity means. This experiment cut without any such constraint and collects none of that benefit.

So

• Which ones get zeroed is everything. By magnitude, 2.2097 at 30%; at random, 3.7873 at 10%
• Deleting 5% at random gives 3.1419, worse than the bigram’s 2.6501
• Global beats per layer: 2.2097 against 2.3893 at 30%
• Reaching part three’s int8 loss by pruning allows cutting 1.8%
• The zeros’ positions must be stored too, so cutting 50% saves nothing under COO. At equal compression, quantisation wins
• It is not faster. Zeroing 99% gives 1.01x. Compression is a memory story, not a time one

Next time takes up the place part five explicitly left open. It measured padding waste and only named the remedies; this measures one.