Predicting characters with 637,156 parameters

The pieces are all in hand. Now they get wired together and run. The goal is not performance but measurement.

The corpus is the thirteen English posts on this blog. Strip the figure SVGs and that is 75,353 characters over a vocabulary of 100. The blog keeps growing and the corpus with it, so exactly what was trained on is frozen at /data/part13-corpus.txt; every number below comes from that file. Split 90/10, 67,817 characters to train on and 7,536 to validate against. The model is Pre-LN as part twelve concluded, three blocks, d=128, four heads, 128 characters of context, 637,156 parameters.

First, put a scale on the loss

Before looking at a number it has to be clear what the number means. The loss is cross entropy, and a model that knows nothing guesses uniformly. With a vocabulary of 100 that puts the loss at ln(100) = 4.6052.

The check is exact. Zero the final layer’s weight and bias and every logit is equal, so the prediction is perfectly uniform.

head zeroed:  loss 4.605171     ln(100) = 4.605170

A difference of 5.4e-07. At real initialisation it reads 4.7140, slightly higher, which is the amount a random head’s logit spread pushes the prediction off uniform.

That fixes the ceiling; the floor needs one too. Bigrams: count what character follows what, using nothing but the previous character. Counted on the training split and measured on validation, that is 2.6501. A transformer that cannot beat this line has no reason to exist.

uniform guess   4.6052   perplexity 100
bigram          2.6501   perplexity  14.2

Perplexity is exp(loss), and reads as how many ways the model is torn. Uniform is a hundred ways, bigrams 14.2.

Run it

step      train       val
   1     4.5573    4.5573
 200     2.6367    2.6474      <- catches the bigram
1000     2.1031    2.2001
3000     1.2432    1.7510      <- validation minimum
5000     0.8344    2.0272

Two hundred steps catch the bigram. That is what learning to count the previous character costs.

The minimum is 1.7510 at step 3000, a perplexity of 5.8. Down from the bigram’s 14.2, so looking further back genuinely pays.

And part seven repeats itself exactly. Past 3000 steps the training loss keeps falling while validation rises: 0.8344 against 2.0272 at the end. Watch the training curve and it is improving; in reality it is getting worse. With only sixty thousand characters of corpus, 637k parameters are plenty to memorise it.

What it learned

Samples from the same model at several points, first lines only.

step 0      ]TUoF%힣`Y)oYgEk1 fb3cqieQZqsV6{O²h`|F#a {_m:O.5Σ-VE·.

step 200    `1   (힣,  o 1Ee1  b3      as   Oghe        0.
            0he.0.     9     -19alongis.    ug    s

step 1000   `` ovan`, ong vo abe tientas to
            fre is steme- rivaresimere. Sonsing isactinalongis.

step 5000   `16, a=0, 36, 112003  0.3009849
            ```
            The smallest `1.000h`. **`shape(2*x) = 0
            `dL/{w_h` andeds share number `sqrt(d) = sum(0)`

Spaces appear by step 200, English-shaped words by 1000, and by 5000 it is producing markdown - backticks, code fences, ** emphasis, sqrt(d), things resembling dL/dw.

Honestly, that is not English learned but the surface of this corpus learned. The corpus is technical markdown, so the shell of it comes first. A number like 0.272 is a value actually used in part twelve, and is closer to memorised than predicted.

Take off the causal mask

Attention looks everywhere by default. In next-character prediction, the next character is already in the input: the answer at position t is the input at position t+1. Hence the triangular mask that forbids looking ahead.

Run it without.

step     train      val
 300    2.5317   2.5530
 600    0.3393   0.3348
 900    0.0473   0.0506
1200    0.0306   0.0331

A loss of 0.03. Far under the bigram’s 2.65, and far under the 1.75 of the model that trained properly. Validation falls right alongside it, so there is no overfitting signal either. Even part seven’s “judge by validation” fails here.

Reading the answer off the side is simply cheaper. It may have picked up something - positional structure, say - but measured on predicting the next character it is worse than before training. Confirming it takes one line - measure the same model with the mask switched on.

train loss measured with the mask on:   5.9703

Worse than the 4.6052 of a model that never trained. That is what makes this failure frightening: the loss curve descends beautifully, validation follows it down, and the real performance is below guessing.

Take one head out and look

Part ten said heads have room to look at different things but are not forced to. Measure it in this trained model - the average weight each head gives at each relative distance.

block head    d=0     d=1     d=2     d=3     d=4
  0    0    0.017   0.020   0.016   0.016   0.015
  0    1    0.016   0.017   0.018   0.017   0.016
  0    2    0.005   0.879   0.005   0.004   0.005
  0    3    0.019   0.030   0.017   0.017   0.017

Block 0, head 2 gives 0.879 to the immediately preceding character. To itself 0.005, and about 0.005 everywhere else. A nearly pure previous-character head, arrived at by training.

The other three heads in that block sit between 0.015 and 0.030 at every distance. Entropy makes how even that is explicit: spreading uniformly over everything visible from those positions scores 4.551, and the three heads score 4.430, 4.440 and 4.391, or 0.97 of it. Their largest weights are 0.030, 0.029 and 0.040. They are effectively looking nowhere. Head 2 on the same scale is 0.660, a ratio of 0.145. Part ten’s “the structure only makes room” splits like that inside a single block.

In blocks 1 and 2, eleven of the twelve heads peak at distance 1, but broadly, between 0.20 and 0.34. Only block 0’s head 2 is sharp.

So

• The loss has two scales. The ceiling is ln(vocabulary), here 4.6052, reproduced to within 5.4e-07 by zeroing the head. The floor is the bigram’s 2.6501
• Two hundred steps catch the bigram; step 3000 bottoms out at 1.7510. In perplexity, 14.2 down to 5.8
• After that it is part seven again. Training falls to 0.8344 while validation climbs to 2.0272
• Without the causal mask the loss reaches 0.03, and the same model measured with the mask on gives 5.9703 - worse than before training. A pretty curve is not evidence of learning
• Heads can specialise without being forced to. Block 0 head 2 puts 0.879 on the previous character while the other three in its block idle near uniform

That is thirteen parts. What the measuring was for is the series’ own answer, and it was not for making anything faster. It was for finding what was wrong.

In part four the culprit for a dead signal looked like tanh, and its derivative turned out to be 1.0000, the largest it can be. In part six the drift at small batches looked like a bad estimate of the standard deviation, and the output’s standard deviation turned out to be exactly one at every batch size. In part ten a head’s rank ceiling was written up as forbidding a pattern, until measuring showed rank 2 suffices for any permutation and the section had to be rewritten. And in this part a model whose loss curve descends beautifully to 0.03 turned out to be worse than guessing.

Each time a plausible account came first and measurement found it wrong. That is what the drawing is for.