Cut the cache and keep the beginning

Part two ended with the cache bigger than the weights: 3.00 MB against 2.43 MB at n=1024, and 4.0 GB at real scale. The longer the context, the further that number climbs.

The simplest answer is to throw entries away: keep the most recent W and delete the rest. The question is how much breaks.

A window alone

On part thirteen’s model, restrict position t to seeing only [t-W+1, t] and measure validation loss.

window W    validation loss
       4             2.7566
       8             2.4575
      16             2.2905
      32             2.1989
      64             2.0770
    full             2.0272

W=16 gives 2.2905. Better than part one’s bigram at 2.6501, but well adrift of the full cache’s 2.0272.

Keeping a few from the beginning

Now add one thing: on top of the window, always keep the first few positions.

on top of W=16    validation loss
keep none                 2.2905
first 1                   2.2068
first 2                   2.1631
first 4                   2.1111
first 8                   2.0905
first 16                  2.1446

The first eight give 2.0905. The full cache is 2.0272, so the gap is 0.063, and the entries stored are 24 against 128 - 19%.

full          128 entries    0.38 MB
W16 + first 8  24 entries    0.07 MB

Note that keeping the first sixteen is worse, at 2.1446. Keeping more does not keep helping.

Is it the beginning, or just four more keys

That question has to be asked. Whether keeping four positions helped because of where they are or because there are four of them needs separating.

on top of W=16          validation loss
keep nothing                    2.2905
first 4 (0-3)                   2.1111
middle 4 (30-33)                2.2049
middle 4 (60-63)                2.2759
just widen the window to 20     2.2655

It is the beginning. The same count taken from the middle gives 2.2759, almost nothing, and widening the window by four gives 2.2655, the same. Sweeping the position looks like this.

start of kept block    validation loss
                  0             2.1111
                  1             2.1203
                  4             2.1258
                 16             2.1493
                 32             2.2100
                 60             2.2759
                 96             2.2971

Earlier is better, and far enough in it is worse than keeping nothing at all (2.2971 against 2.2905).

Attention mass does not explain it

The usual explanation here is the “attention sink”: the earliest positions soak up a large share of attention, so they must not be discarded. Measured, this model has no such thing.

In normal operation, the mass later positions give to position 0:

block 0   0.0035     uniform from those positions would be 0.0108   ratio 0.3
block 1   0.0000                                                     0.0
block 2   0.0000                                                     0.0

It receives less than uniform. Position 0 is not a place this model looks.

Under the window it is the same story.

mass given to those four from later positions
first (0-3)       0.0155
middle (60-63)    0.0336

The middle four take more than twice the mass and help far less. So “keep the positions that receive the most” does not fit what was measured.

Then why does the beginning help? Unknown. The measurement separates position from count, and rules out mass, but what those positions carry is not something this experiment answers. A plausible story could be attached; an unmeasured one will not be written down.

The absence of a sink here does not contradict the reports from large models either. This is a vocabulary of 100 across three blocks, trained 5000 steps on sixty thousand characters. A phenomenon reported at scale not showing up here is unremarkable, and this part’s conclusion stops at the beginning pays, in this model.

So

• Windowing the cache costs loss: 2.2905 at W=16 against the full 2.0272
• Adding the first eight positions gives 2.0905 - 24 entries against 128, 19%, for a gap of 0.063
• Keeping more does not keep helping. The first sixteen go back up to 2.1446
• It is position, not count. The same four from the middle give 2.2759, and widening the window by four gives 2.2655
• Attention mass does not explain it. There is no sink in this model, and the first four receive half the mass the middle four do
• Why it works was not measured. Only that it is position, and that it is not mass

Next time measures a time nobody has measured yet in this series. Everything so far has been the cost of producing characters; before that comes reading the prompt.