Pulling the gates out, the forget gate had barely moved

Part one changed one character and watched the difference in hidden state halve within four. The gated LSTM and GRU decayed at nearly the same rate as the ungated RNN. Seeing why a gated cell behaves like an ungated one means pulling the gates out and looking.

One LSTM step

An RNN replaces its state wholesale. An LSTM splits the state in two: the cell state c is only edited a little at a time, and the hidden state h takes out of it whatever is needed. Gates decide how much of each.

z = W_ih @ x + b_ih + W_hh @ h + b_hh     # four chunks at once
i = sigmoid(z[0:H])      # input gate - how much of the new goes in
f = sigmoid(z[H:2H])     # forget gate - how much of the old stays
g = tanh(z[2H:3H])       # candidate - the content to put in
o = sigmoid(z[3H:4H])    # output gate - how much of the cell shows

c = f * c + i * g
h = o * tanh(c)

i, f and o come through sigmoid, so they live between 0 and 1: shut at 0, open at 1.

The line that matters is c = f * c + i * g. A gradient going backward is multiplied by f at each step, so how far back the cell remembers is set by f - in theory at least.

nn.LSTM returns only h and keeps the gates to itself, so the lines above were run by hand from the trained weights. They match PyTorch’s h to at most 6.85e-07. Everything below is read out that way.

What the gates actually are

Taking the same 800-step LSTM part one used, feeding 32 validation lines of 128 characters and collecting all 325 units’ gates gives 1,331,200 values.

              mean     5%     25%     50%     75%     95%
input i      0.661  0.294   0.515   0.684   0.828   0.946
forget f     0.495  0.081   0.258   0.491   0.730   0.922
output o     0.587  0.178   0.392   0.597   0.798   0.947

The forget gate averages 0.495. PyTorch starts biases at zero, so an untrained forget gate is sigmoid(0) = 0.5. After 800 steps of training it has barely left.

No unit is holding a long memory

An average of 0.5 would still be interesting if some units sat at 0.95, holding on for a long time, while others sat at 0.1 and dropped everything - a division of labour across timescales, which is the usual telling. Unit by unit:

mean forget gate across the 325 units
  min 0.307    5% 0.377    median 0.494    95% 0.622    max 0.793
the half-life in characters that implies
  min 0.59     median 0.98                 max 2.99

All 325 fall between 0.31 and 0.79. As half-lives, the most retentive unit holds for 3 characters, and 172 of them - more than half - sit below the 0.5 they started at. Not one holds for ten.

There is no division of labour across timescales. They all forget quickly together.

The forget gate is not the whole rate

Except that part one measured the state difference halving in four characters, four times the 0.98 the median unit’s forget gate implies.

Looking at c = f * c + i * g again shows why. A perturbation does not only shrink by f. A different h makes the next step’s i and g different too, and that difference goes back into the cell. There is a return path alongside the forgetting one.

Separating them: take the cell difference just after the perturbation and roll it forward multiplying only by f, against actually running both texts to the end.

    after   measured   forget gate alone
        1      0.711               0.574
        4      0.337               0.124
        8      0.144               0.022
       16      0.041               0.003
       32     0.0069            6.48e-04

Half-life 4 characters measured, 3 with the forget gate alone. By thirty-two characters the measured difference is 11 times what the gate would have left.

The forget gate sets only part of the rate. The rest is what re-enters through the input side every step, and it drags the memory out longer than the gate says.

The gates respond to the character

Gate values are not fixed numbers; they are computed from x and h at every step, so they should move with the text. Splitting the 4,096 positions by character and averaging the forget gate:

space       0.4623   (865 positions)
newline     0.4597   (105 positions)
everything else 0.5057   (3,126 positions)

About 0.05 more closed at spaces and newlines - discarding slightly more where a word ends. Not a large difference, but the direction is unmistakable: clear out at the boundary.

What is left

These numbers come from a model trained 800 steps, which part seven shows is the bottom of its validation loss. Where the gates go if it keeps training was not looked at.

The GRU was not opened either. It has two gates and no separate cell state, and part one found it forgetting at almost the LSTM’s rate. That is the next part.

And “no unit holds a long memory” is attached to this corpus and this task. Part eight opens the forget bias and the reach goes to the whole context while the loss gets worse. On a task that needed long memory, training would have taken the gates somewhere else.

So

• One LSTM step is c = f·c + i·g and h = o·tanh(c), with i, f and o all doors between 0 and 1
• nn.LSTM does not hand over the gates, so the step was run by hand from the weights - matching PyTorch to 6.85e-07
• The forget gate averages 0.495, essentially the sigmoid(0) = 0.5 it started at
• All 325 units’ mean forget gates fall inside 0.307~0.793. As half-lives that tops out at 3 characters with none past ten. No division of timescales
• Yet the state itself has a half-life of 4, and at thirty-two characters holds 11 times what the forget gate alone would leave - the difference re-enters through the input side
• The gates move with the text: 0.4623 at spaces, 0.4597 at newlines, 0.5057 elsewhere. Slightly more discarded at boundaries