At the same budget the GRU wins - for 700 steps

The six parts before this opened up one recurrent step, the LSTM’s gates, the GRU, the convolution kernel, the receptive field and pooling. What each of them computes has been seen; which of them is better has not.

So put them together. One condition: everything has to be measured against the same thing. Match the parameter budget to the transformer from part thirteen of the first series, and keep the corpus and the optimiser identical.

Matching the budget

Part thirteen’s model has 637,156 parameters. Same corpus, same 128-character context, same batch of 32, same AdamW at 3e-4, and each architecture’s width tuned to hit that number.

              width    parameters   vs transformer
RNN             683       637,845          100.1%
LSTM            325       637,550          100.1%
GRU             381       635,835           99.8%
CNN     171 channels      639,273          100.3%
transformer                637,156          100.0%

All within 0.3%. The embedding and the output layer are identical across all five; only the middle changes. The CNN is four causal convolutions with kernel 5, so its receptive field is 4 x 4 + 1 = 17 characters - that comes back later.

The same budget does not mean the same use of it. The RNN pours 466k of it into a single 683 x 683 recurrent matrix while the transformer spreads it over three blocks. That is part of the architecture too.

Comparing at a fixed step count is wrong

The first attempt ran all five for 4000 steps and compared the last value:

RNN 3.4327   LSTM 3.2872   GRU 3.3432   CNN 3.6865   transformer 1.8300

A rout for the transformer. Except that at step 800 of those same runs, the LSTM is at 1.6817 and the GRU at 1.6550. The corpus is only 75,353 characters, so all five memorise it, and they reach the bottom at different times. Comparing at a fixed step count means measuring someone else past their minimum.

So the protocol changed: validate every 100 steps up to 1500 and every 250 after that, and take each architecture’s lowest validation loss as its score. Three seeds each.

All five on one plot

              parameters   best val   3 seeds              at step   perplexity
GRU              635,835     1.6449   1.6276~1.6503            800        5.180
LSTM             637,550     1.6776   1.6737~1.6785            800        5.353
transformer      637,156     1.7679   1.7662~1.7694           2750        5.859
RNN              637,845     1.8355   1.8354~1.8362            800        6.268
CNN              639,273     1.8709   1.8695~1.8748            300        6.494

The seed spread is tiny. The RNN’s three land between 1.8354 and 1.8362, the transformer’s between 1.7662 and 1.7694. Seeds do not reorder this.

The GRU wins

1.6449 against 1.7679. In perplexity, 5.180 against 5.859 - 11.6% better. The LSTM leads too at 1.6776. At 637k parameters learning 75 thousand characters, attention is not the best answer available.

Which is not surprising. Attention earns its keep where the context is long and the data is plentiful; here the context is 128 characters and the data is a chapter of a novel. At that size, carrying one hidden state is the more efficient arrangement.

The RNN losing at 1.8355 is the other side of the same story. Without gates there is no deciding what to keep and what to drop. The margin from the RNN to the LSTM and GRU (0.16 to 0.19) is wider than the margin to the transformer. Gating makes a bigger difference here than attention does.

For 700 steps

But that table collects everybody’s best moment. The curves say something else.

              best    at 4000     ratio
GRU         1.6449     3.4170      2.08
LSTM        1.6776     3.3660      2.01
RNN         1.8355     3.5293      1.92
CNN         1.8709     3.7161      1.99
transformer 1.7679     1.8424      1.04

Only the transformer holds together. Past its minimum and out to step 4000 it is at 1.04 times its best, while the others double.

Counting the steps where the GRU is under the transformer’s best of 1.7679 gives step 500 to step 1200. All three seeds give exactly that interval. The LSTM gives 500~1100. Outside it, the transformer wins.

The other way round, the transformer stays under 1.1 times its own best (1.945) from step 1500 to step 4000 and is still inside at 4000. The GRU on the same test gets 300~1500.

So read it like this. The GRU’s peak is higher, but it sits in a 700-step window, and the transformer’s seat stays open. If you know exactly when to stop, use the GRU. If you do not, use the transformer.

The convolution sees 17 characters

Four layers of kernel 5 reach 17 characters to the left. The other four see all 128.

That was first written up as the reason it comes last. Part five widens the window to 125 and gets 1.9967 - worse - and even widening by dilation alone, without adding a single parameter, takes 1.8709 to 1.9716. It is not losing because its view is narrow.

It is also first to the bottom, at 300 steps. It has the least to learn, so it finishes learning first and starts memorising first.

How to widen a receptive field, and whether widening it closes the gap, gets measured later in this series.

Why this differs from part thirteen

Part thirteen reported the same transformer’s best validation as 1.7510. Here the three seeds are 1.7662, 1.7679 and 1.7694. Part thirteen’s number is outside that range.

The cause turned up. Part thirteen’s training script samples text partway through, and that function begins with torch.manual_seed(0). That reseeds the global generator, so from there the batch order repeats from the beginning.

no reseed midway   136044 248239 714933  93760 848963 848379
reseed on the 3rd  136044 248239 714933 136044 248239 714933

Part thirteen’s model took that reseed at steps 200 and 1000, so it saw a different data order than this run did. There is no reason the two numbers would agree. What part thirteen published is what that run actually produced, and that checkpoint still gives that loss, so the earlier series stand. This part just uses numbers from one protocol run across all five.

What is left

What this gives is a ranking for this corpus at this size. 75 thousand characters is small. Scaling the data would be expected to reorder it, but expectation is not measurement.

Depth was not touched either. Every recurrent model here is one layer and the transformer is three. The budget was matched on width alone, so what happens when recurrence gets a second layer is unknown.

And training time was not measured. This first said one GRU run took more than twice as long as the transformer - a number taken by timing five configurations one after another, which makes it unusable. Part nine measures it round-robin and gets 1.21 times per step, and answers whether comparing at equal step counts was fair: it was, and it leaned the transformer’s way.

So

• Matched to 637,156 parameters, all five architectures land within 0.3%
• All five overfit. Comparing at a fixed step count measures someone else past their minimum
• By lowest validation: GRU 1.6449 < LSTM 1.6776 < transformer 1.7679 < RNN 1.8355 < CNN 1.8709
• The GRU is 11.6% better than the transformer in perplexity. The ungated RNN loses - gating makes a bigger difference here than attention
• But the GRU only leads from step 500 to 1200, and by 4000 it is 2.08 times its best. The transformer is at 1.04
• The CNN sees 17 characters to its left. But part five widens that window and it gets worse - a narrow view is not why it loses