Changing the task inverted the order

This series has said the same thing several times. A recurrent state remembers about four characters (part one); opening the gates makes the loss worse (part eight); widening the convolution’s window makes it worse (part five); reach does not predict performance.

And every time it attached the same caveat - this is a task where the front of the sequence hardly matters. Parts two, five and eight each wrote “on a task that genuinely needed it, this would be the reverse” and moved on. Three deferrals is enough.

A task where the front is required

Draw 60 characters at random from an alphabet of 20, put in a separator, then repeat the same 60. Only the second copy is scored.

What has to be predicted at position 60 is the character at position 0 - a distance of exactly 60 - and position 60+j needs position j, so the distance stays 60 throughout. The string is random, so without looking back there is nothing to do but guess, and guessing is ln(20) = 2.9957.

The models, the budget and the optimiser are the same as the earlier parts. Only the data changed.

Three of them cannot do it at all

             best val (3 seeds)               median
transformer  0.0002  0.0002  0.0002       0.0002
GRU          1.8907  1.9005  1.8966       1.8966
LSTM         2.9967  2.9974  2.9966       2.9967
CNN          2.9974  2.9975  2.9974       2.9974
RNN          2.9980  2.9982  2.9975       2.9980

RNN, LSTM and CNN do not leave 2.9957 to three decimal places. They learned nothing.

For the CNN that is expected. Part five confirmed its receptive field at 17 characters with everything outside it absent from the computation graph. Sixty back may as well not exist. There is a configuration reaching 125, and it was not run here - that is noted below.

The two recurrent models can structurally see it. They still cannot do it. Part one measured a single changed character losing half its trace within four and under 1% by thirty-two. Reaching back 60 with that is not possible.

The transformer solves it outright

0.0002, the same for all three seeds. A loss of 0.0002 means all sixty characters right - 15,000 times better than chance.

Attention wires positions directly to each other. That is what part eight measured: at 127 back its gradient was 1.19e-02, 39,145 times the GRU’s at the same distance. There, that reach was useless. Here it is the whole task.

Part eight’s “reach does not predict performance” inverts here - or more precisely, the condition under which it inverts is now known.

The GRU does about half of it

1.8966, well under chance and well over the transformer. Position by position, the first copied character is easiest at 1.237 and it worsens toward 1.77 further along.

The first being easiest was strange enough to hypothesise about: the first input writes into an empty state with nothing to compete with, so perhaps it leaves a larger imprint. Measured, on an untrained GRU, changing position j and seeing how much the state at step 60 moves:

changed at   j=0      j=20     j=40     j=58     j=59
state moves  0.0000   0.0000   0.0001   0.5809   1.1070

Everything through j = 40 is 0. There is no first-impression effect. The hypothesis was wrong and the reason remains unknown.

Gates move when the task asks

Since the GRU manages something, the place to look is where its gates went. Pulling the update gate out exactly as part three did:

                          z mean   unit median   median half-life   10+ chars
trained on the copy task   0.921         0.914          7.75 chars   131 of 381
trained on characters (3)  0.405         0.398          0.75 chars            0

Same architecture, same width, same optimiser, and the update gate has gone from 0.405 to 0.921. The median half-life goes from 0.75 characters to 7.75, a factor of ten, and the “holds ten or more” units that part three could not find even one of now number 131.

Parts two and three were not wrong. Part two found the forget gate barely off its 0.5 initialisation; part three found no unit holding a long memory. Both were right, and both because character prediction never asked. Ask, and they move.

The earlier parts are not overturned

What inverts here is not the earlier numbers but the conditions they were attached to.

• Part one’s “about four characters” still holds. It is exactly why the RNN and LSTM fail here
• Part two’s “the gates barely moved” still holds. Change the task and they move
• Part five’s “widening makes it worse” still holds. There was no reason to widen
• Part eight’s “reach does not predict performance” still holds. Here reach is all there is

That the three caveats were right is this part’s result.

What is left

The CNN was not run at a receptive field of 125. Part five’s configuration exists, so it could be, and whether a window past 60 lets it copy is unmeasured. It probably would - unmeasured is unmeasured.

Only one copy length was used, 60. Dropping it to 4, 8, 16 and watching where recurrence starts succeeding would connect directly to part one’s “four characters”, and was not done.

And the transformer’s 0.0002 comes from 3000 steps. Copying has a single rule and nothing to memorise, so unlike the earlier parts no overfitting appears. It is a different kind of easy from real text.

So

• Built a task where the front of the sequence is required: 60 random characters rewritten after a separator, chance ln(20) = 2.9957
• RNN 2.9980, LSTM 2.9967, CNN 2.9974 - none of the three move off chance. Part one’s four-character memory and part five’s 17-character window are what is being paid for
• The transformer solves it at 0.0002, 15,000 times better than chance
• Only the GRU sits between, at 1.8966. Its first copied character is easiest at 1.237 and why is unexplained - the first-impression hypothesis was measured and rejected
• The GRU’s update gate moves from 0.405 to 0.921 and its median half-life from 0.75 characters to 7.75. Part three found zero units holding ten or more; here there are 131
• The earlier numbers all stand. What inverted is the condition attached to them, and three of those parts had already written it down