Sending the gradient further made the loss worse

Part four of the first series watched gradients disappear through twenty stacked layers. In a recurrent network, time takes the place of depth. An RNN reading 128 characters is a 128-layer network passing through the same weights 128 times, so what part four measured against depth gets measured here against length.

How far back does it reach

Send the loss from one position - the last - backward, and measure the size of the gradient arriving at each position’s input embedding. Set the last position to 1 and what survives at each distance reads straight off.

Taking the five architectures from part seven, each trained to its own minimum:

             17 back    64 back   127 back
transformer  6.76e-02   1.32e-02   1.19e-02
RNN          2.98e-02   2.09e-04   8.24e-07
LSTM         3.65e-02   2.28e-04   4.78e-07
GRU          2.67e-02   1.41e-04   3.04e-07
CNN          0.00e+00   0.00e+00   0.00e+00

The transformer stops falling somewhere past 64. At 127 back it is still at 1.19e-02, which is 39,145 times the GRU’s 3.04e-07 at the same distance. Attention wires positions to each other directly, so distance does not lengthen the path.

The convolution is exactly zero at 17

The CNN’s entries are 0.00e+00. Not small - zero.

Part seven put its receptive field at 4 x 4 + 1 = 17 characters for four layers of kernel 5. That means positions further back than 17 are not in the computation graph at all, and what is not in the graph has no gradient rather than a small one. Part seven’s sentence is confirmed exactly here.

Gates do not extend the reach

The textbook says LSTM and GRU solve vanishing gradients. It is not visible in that table. At 127 back the ungated RNN is at 8.24e-07, the LSTM at 4.78e-07 and the GRU at 3.04e-07 - the gated ones are smaller.

A trained model may simply have decided to forget, so measure again before training. The recurrent models have no context limit, so they can also be unrolled to 512.

                  half    1/100   1/10000
RNN                  2        8        15
LSTM                 2        9        19
GRU                  2        9        19

Three seeds agree to the character. Unrolling to 512 changes nothing. What gating buys is the 1/10000 reach going from 15 characters to 19. Four characters.

The cause is one bias

An LSTM carries its cell state as c_t = f_t · c_{t-1} + ..., so a gradient travelling backward is multiplied by f_t at every step. What sets the reach is the forget gate.

PyTorch starts every bias near 0, forget gate included, so sigmoid(0) = 0.5. Half the signal dies every step. 0.5^127 is 5.88e-39, below float32’s smallest normal number of 1.18e-38, which is why an untrained model’s gradient at 127 back underflows to exactly 0 - the same as the ungated RNN.

Changing only that bias to b and measuring again:

   b   sigmoid(b)   half   1/100     1/10000
   0        0.500      2       9          19
   1        0.731      3      23          51
   2        0.881      5     117    never dies
   3        0.953      1  never dies never dies

At b = 2 the gradient never falls below 1/10000 within the 128-character context. At b = 3 the gradient 127 back is 1.80 times the last position’s - larger than its source, the other side of vanishing.

Gating does not solve the problem; gating held open solves it. The old advice to initialise the forget-gate bias at 1 or 2 is this table.

And then the loss gets worse

That would be a tidy story, except for what comes next. Changing only the bias and training again under the same protocol:

   b   1/10000 reach   best val (3 seeds)              median
   0        19 chars   1.6771  1.6739  1.6787       1.6771
   1        51 chars   1.7204  1.6983  1.7258       1.7204
   2      never dies   1.7546  1.7350  1.7463       1.7463

Monotonically worse. The three seeds at b = 0 span 1.6739~1.6787 and at b = 2 span 1.7350~1.7546, ranges that do not overlap. The further the gradient travels, the higher the loss.

Which makes sense. The task is next-character prediction over 128 characters. A character a hundred back rarely decides the one now. Holding the forget gate open makes the hidden state keep carrying old material, and material with no use is noise. Forgetting is a feature.

Part seven already showed the same shape: the transformer sends gradient forty thousand times further than the GRU and loses to it by 11.6%. Reach does not predict performance.

What is left

This conclusion is attached to 128 characters of context and next-character prediction. On a task where the front of the sequence genuinely matters - matching brackets, long dependencies, copying - reach would be performance. No such task was built here.

The cell-state path was not isolated either. What is measured is the gradient arriving at the input embedding, which mixes the cell route with the gate route. Separating out the identity path the textbook talks about might look different.

And whether b = 2 hurt because of the reach or because the state saturates early in training is not distinguished here. Telling those apart means pulling the gate values out directly, which is the next part.

So

• Measuring what the loss at the last position sends backward, the three recurrent models nearly coincide: 8.24e-07 / 4.78e-07 / 3.04e-07 at 127
• The transformer stops falling past 64 and is 39,145 times the GRU at 127 back
• The CNN is exactly zero at 17. Not in the graph means no gradient, not a small one
• At init, gating buys 15 -> 19 characters of 1/10000 reach. Four characters
• The cause is a forget-gate bias of 0, making sigmoid(0) = 0.5 and killing half per step. 0.5^127 underflows in float32
• Opening the bias to 2 stops it dying inside 128 characters; at 3 the gradient 127 back is 1.80 times its source
• But the loss goes 1.6771 -> 1.7204 -> 1.7463, monotonically worse. On this task, forgetting is a feature