Widening the window for free made it worse

Part four showed the convolutional model reaching seventeen characters to its left. The other four see all 128. So widening the window should close the gap.

So widen it.

The window size comes out of an equation

Stacking layers of kernel k with dilation d gives a receptive field of

1 + Σ d_i · (k - 1)

Four layers of kernel 5 gives 1 + 4·4 = 17; dilating them 1, 2, 4, 8 gives 1 + 4·(1+2+4+8) = 61.

Whether the equation is right can be measured. Take the output at the last position, change a character d places back, and run it again. The first d where the output changes by exactly 0.0 is the receptive field.

                channels  parameters   formula   measured
k5 x4                171     639,273        17         17
k5 x8                122     639,370        33         33
k9 x4                128     633,828        33         33
k5 dilated 1-8       171     639,273        61         61
k5 dilated 1-16      153     635,763       125        125

All five agree to the character. This is what part four meant by “not in the graph means no gradient, not a small one”.

Three ways to widen

Add layers, enlarge the kernel, or dilate. The first two cost parameters, so at a fixed budget the channel count has to come down. Dilation costs nothing.

The top of the figure is that difference: dilating leaves the number of lines unchanged while the positions reached spread out.

Wider is worse

                   reach  channels  parameters   best val (3 seeds)        median
k5 x4                 17       171     639,273   1.8748 1.8709 1.8695    1.8709
k5 x8                 33       122     639,370   1.9437 1.9346 1.9326    1.9346
k9 x4                 33       128     633,828   2.0592 2.0652 2.0621    2.0621
k5 dilated 1-8        61       171     639,273   1.9716 1.9795 1.9556    1.9716
k5 dilated 1-16      125       153     635,763   1.9967 2.0020 1.9907    1.9967

The narrowest is the best: 1.8709 at 17 characters against 1.9967 at 125. And none of them come near the GRU’s 1.6449 or the transformer’s 1.7679.

Adding layers or enlarging the kernel has an explanation. The budget is fixed, so channels had to drop from 171 to 122 or 128 - the width of the window was paid for out of the width of the model.

Widening for free is also worse

That explanation does not cover dilation. k5 dilated 1-8 has the same 171 channels, the same 639,273 parameters, the same four layers and the same kernel 5 as k5 x4. The only difference is that its window is 61 rather than 17.

And it comes out at 1.9716 against 1.8709, worse by 0.10, with seed ranges of 1.9556~1.9795 against 1.8695~1.8748 that do not overlap.

Widened for free, and worse for it.

Dilation does not widen, it thins

Why is countable. Count the number of paths from the output back to each input position - more paths means more computation aimed at that position.

                  reach   total paths   positions   share going to the nearest 4
plain x4             17           625     17                             11.2%
dilated 1-8          61           625     61                              1.6%

The path count is 625 either way. Dilation adds no computation. It spreads the same 625 over 61 positions instead of 17.

Position by position it is starker:

back        0    1    2    3    4    5    6
plain       1    4   10   20   35   52   68
dilated     1    1    2    2    4    3    5

Paths to the immediately preceding character drop from 4 to 1, a factor of four. Part one measured the recurrent state remembering about four characters and part two found the gates agreeing. Those four characters are where the signal is, and dilation moves computation away from them.

It does not widen so much as relocate.

The same 33 built two ways is not the same

k5 x8 and k9 x4 both reach 33 and land at 1.9346 and 2.0621, 0.13 apart. Eight layers beats a kernel of nine by a lot.

Enlarging a kernel costs parameters in proportion to k and eats channels for it; stacking layers adds one more nonlinearity per layer. Equal receptive fields, and how you built them still decides the loss.

Correcting a sentence in part seven

Part seven ends with “the CNN sees 17 characters to its left. It loses on sight, not on budget.” The first half is right and the second half is wrong.

Widening the sight to 125 characters gives 1.9967, worse. The CNN is not losing because its view is narrow. A narrow view is what this task wants, and whatever paid for the wider one was taken from somewhere.

That passage in part seven has been corrected.

What is left

Everything here runs to 1200 steps. All five configurations bottom out between 300 and 900, so nothing is cut off, but whether the ordering survives longer training was not checked.

And “wider is worse” belongs to this corpus and this task. Where the front of the sequence genuinely matters - matching brackets, long copying - it would be the reverse. Part eight makes the same point again from the recurrent side.

Only two dilation patterns were tried, 1-2-4-8 and 1-2-4-8-16. Something like 1-1-2-2, which keeps the near positions while spreading a little, is unmeasured.

So

• The receptive field is 1 + Σ d·(k-1), confirmed by changing a character outside it and getting exactly zero change. All five configurations match
• Wider is worse: 1.8709 at 17 characters up to 1.9967 at 125
• Layers and kernels cost channels, so that much is unsurprising. But dilation is free and still takes 1.8709 to 1.9716, with non-overlapping seed ranges
• Dilation adds no paths. It spreads the same 625 over 61 positions instead of 17, dropping the nearest four characters’ share from 11.2% to 1.6%
• Paths to the immediately preceding character go from 4 to 1. Part one’s “memory of about four characters” is exactly what gets starved
• Equal receptive fields built differently differ: 33 by depth is 1.9346, by kernel 2.0621
• Part seven’s “loses on sight, not on budget” is wrong, and has been corrected