Pooling throws away three bits of position

Pooling is a part borrowed from images. A cat three pixels to the left is still a cat, so it is used to build representations that do not wobble under small shifts.

For predicting the next character, position is the entire question. So what pooling does here has to be measured on its own.

What pooling computes

Take the largest value inside a window of w (max pooling) or their average (mean pooling). It has no weights, so it costs no parameters.

With stride w the sequence shortens by a factor of w; with stride 1 the length is unchanged. Either way, padding only on the left keeps it causal.

What it buys: less movement under a shift

Take the first convolutional layer’s output from part four, shift the text by one character, recompute, and compare the downsampled representations.

Above: how much the downsampled representation changes when the text shifts by one character. Plain subsampling becomes effectively a different thing at 1.16, while max pooling falls to 0.22 as the window grows - that is the invariance being bought. To the right, the position information being spent on it. Below: validation loss at the same 171 channels and 639,273 parameters. Between the dotted rules is what the wider reach cost; to the right of them is what pooling itself cost.

window            2       4       8
max pool      0.730   0.403   0.220
mean pool     0.720   0.410   0.231
plain sample  1.168   1.166   1.160

Taking one value every w without pooling gives 1.16 under a single-character shift. A relative norm above one means it is effectively uncorrelated with what it was.

Max pooling brings that to 0.730, 0.403, 0.220. The wider the window, the less it wobbles. That is what pooling buys, and why images use it.

What it spends: log2(w) bits of position

The price is which place inside the window it was. A max-pooled output is one value, so which position won does not survive.

How much is being discarded is countable. The distribution of the winning offset:

window   winning offset                       entropy / max
     2   0.632  0.368                          0.95 / 1.00
     4   0.301  0.231  0.236  0.232            1.99 / 2.00
     8   every offset 0.120 ~ 0.129            3.00 / 3.00

All but perfectly uniform. At window 8 the entropy is 3.00 bits, equal to the maximum to two decimal places. The three bits being thrown away are three bits recoverable from nowhere else. Had there been any regularity in which offset wins, the entropy would have come in under three.

On this task

Whether the trade pays is a training question. Insert stride-1 pooling after each layer of part four’s convolution - stride 1, so the length is unchanged and no parameters are added - and hold everything else fixed.

                        reach  channels  parameters   best val (3 seeds)      median
no pooling                 17       171     639,273   1.8748 1.8709 1.8695  1.8709
max pool 2                 21       171     639,273   2.0047 1.9726 1.9628  1.9726
max pool 4                 29       171     639,273   2.1446 2.1409 2.1060  2.1409
mean pool 4                29       171     639,273   2.1518 2.1458 2.1342  2.1458
control, same reach 29     29       171     639,273   1.8987 1.9033 1.8922  1.8987

Identical channels and identical parameters across all five. Max pooling at 4 takes 1.8709 to 2.1409, worse by 0.27.

Not because the reach widened

Stride-1 pooling still widens the receptive field: a window of w across four layers adds 4(w-1), so max pool 4 reaches 29 rather than 17. Part five found wider to be worse, so some of that 0.27 is the widening.

Hence the control: reach pinned to 29 by dilation alone with no pooling, again at the same channels and parameters.

no pooling (reach 17)          1.8709
control (reach 29, no pool)    1.8987     from the wider reach  +0.0278
max pool 4 (reach 29)          2.1409     from pooling itself   +0.2422

Pooling itself costs 8.7 times what the widening did. The confound is small.

Max and mean are indistinguishable

2.1409 against 2.1458, with seed ranges of 2.1060~2.1446 and 2.1342~2.1518 that overlap. It is not which one you pick but that you pick at all that costs.

Which follows. Both collapse w values into one, and whether the rule is a maximum or an average, the position goes the same way.

What is left

Pooling went in after every layer here. Only at the last, or every other layer, would cost less - unmeasured.

Stride-w downsampling was not trained either. Shortening the sequence means building something to lengthen it again for a per-position prediction, and that makes it a different comparison.

And this conclusion belongs to a task where position is the answer. Classifying a whole passage - is this code or prose - would likely be helped by throwing position away. That task was not built.

So

Pooling collapses a window into one value. No weights, no parameters
What it buys is invariance: a one-character shift moves the representation 1.16 without it and 0.22 with max pooling at window 8
What it spends is position: the winning offset’s entropy at window 8 is 3.00 bits, equal to the maximum - three bits recoverable from nowhere else
On this task it loses, 1.8709 to 2.1409, worse by 0.27
Only +0.0278 of that is the wider reach; +0.2422 is pooling itself. 8.7 times
Max and mean are indistinguishable. The collapsing is the cost, not the rule