writing 한국어로 읽기
재귀와 합성곱 part 3 of 13

The one with a gate removed won

guide / / 6 sections

Part two found the LSTM guarding a cell state with three gates. A GRU does the same job with two and keeps no separate cell state at all. Something was dropped, and yet part one found it forgetting at almost the same rate and part seven found its loss lower.

One GRU step

gi = W_ih @ x + b_ih       # three chunks
gh = W_hh @ h + b_hh

r = sigmoid(gi_r + gh_r)          # reset gate
z = sigmoid(gi_z + gh_z)          # update gate
n = tanh(gi_n + r * gh_n)         # candidate; r multiplies only the old state

h = z * h + (1 - z) * n

The last line stands where the LSTM’s c = f * c + i * g stands, in a different shape. The LSTM decides how much to keep, f, and how much to add, i, separately; the GRU decides both with z alone - keep z, add the remaining 1 - z.

r has no LSTM counterpart. It decides how much of the old state goes into building the candidate. So a GRU is not an LSTM with a gate taken out; it is a different arrangement.

As in part two, the step was run by hand from the trained weights and matches nn.GRU to at most 7.90e-07. Everything below is read from that.

              mean     5%      50%     95%
update z     0.405  0.032   0.370   0.883
reset r      0.629  0.120   0.690   0.971

The LSTM does use them separately

The most plausible reason merging would be safe is that the LSTM is already setting i to roughly 1 - f, leaving the extra freedom idle. Then a GRU loses nothing by tying them.

Measured, that is not the case.

input gate against 1 - forget coupled: on this line 0 0 0.5 0.5 1 1 input gate i 1 - forget gate f half-life per unit 0.5 1 2 3 LSTM f GRU z units, sorted
Left: where the LSTM's input gate i and one-minus-forget-gate land, 1.33 million values binned into 24 x 24. Coupled the way a GRU couples them, the mass would gather on the dashed line; instead it fills the square. Right: the per-unit half-life implied by the LSTM's forget gate and the GRU's update gate. The GRU's is shorter.
correlation of i with (1 - f), all 1,331,200 values      +0.071
per unit   min -0.629   median +0.052   max +0.768
units above 0.5                                       13 of 325
mean of i + f (1.0 if coupled)                            1.156

In the left panel the mass does not gather on the dashed line; it fills the square. Comparing the two-dimensional distribution against the product of its marginals, the largest departure is 0.175% of a cell - all but perfectly independent.

The LSTM is not leaving the freedom idle. It uses it.

And still loses

Using it, and still part seven has the GRU at 1.6449 against the LSTM’s 1.6776. The remaining explanation is width: three gates instead of four means the same budget buys a larger state.

LSTM  4 gates   width 325   4 x 325² = 422,500
GRU   3 gates   width 381   3 x 381² = 435,483

At the same 637k budget the GRU is 17.2% wider. So the GRU was pinned to the LSTM’s width of 325 and retrained. The state matches; the parameter count drops below the LSTM’s.

                 parameters   best val (3 seeds)                median
GRU   width 381     635,835   1.6449  1.6503  1.6276      1.6449
GRU   width 325     489,675   1.6498  1.6624  1.6611      1.6611
LSTM  width 325     637,550   1.6776  1.6737  1.6785      1.6776

Matched on width the GRU still wins - on 23.2% fewer parameters. The seed ranges, 1.6498~1.6624 against 1.6737~1.6785, do not overlap.

The advantage splits almost exactly in half.

LSTM  width 325   1.6776
GRU   width 325   1.6611     from the structure   0.0165
GRU   width 381   1.6449     from the width       0.0162

Half comes from how the gates are arranged, half from having one gate fewer to pay for and being able to spend it on width.

The GRU forgets faster

That is the right panel. Turning each unit’s gate into a half-life:

           min  median    max
LSTM f    0.59    0.98   2.99
GRU  z    0.45    0.75   1.89

Shorter everywhere. Part two found no LSTM unit holding past ten characters; no GRU unit holds past two.

The one that forgets faster does better. Same direction as part eight, where opening the forget gate extended the reach and worsened the loss. Holding on is not an advantage on this task.

What is left

What the reset gate r does was not examined. It averages 0.629, half open, and pinning it to 1 to see what breaks would put a number on it. Not measured.

And “half structure, half width” is measured at the single width of 325. Wider or narrower, the split could move.

Every model here is also one layer. How the trade between gate count and width behaves once recurrence is stacked is not covered in this series.

So

  • A GRU step is h = z·h + (1-z)·n. What the LSTM sets separately with f and i, the GRU sets with z alone
  • The reset gate r has no LSTM counterpart. A GRU is a different arrangement, not an LSTM minus a gate
  • Run by hand, it matches nn.GRU to 7.90e-07
  • The LSTM does use i and 1-f separately: correlation +0.071, mean i + f of 1.156, and a largest departure from independence of 0.175% of a cell
  • It still loses. Matched at width 325 the GRU is at 1.6611 against 1.6776 on 23.2% fewer parameters, with non-overlapping seed ranges
  • The advantage is half and half: 0.0165 from the structure, 0.0162 from the width
  • Per-unit half-lives run 0.45~1.89 for the GRU against 0.59~2.99 for the LSTM. The faster forgetter does better

Comments