Normalisation erases the initialisation

Part four concluded that the initialisation is a precondition. Too small and the signal is 1e-16 after twenty layers; too large and the gradients differ by 1e9 across the depth. Only Xavier avoided both.

Normalisation removes the precondition entirely. Add one line per layer and any initialisation arrives at the same place. Start with how same.

One line

Right after multiplying by the weights and before the activation, subtract the mean and divide by the standard deviation.

z = h @ W
z = (z - z.mean(0)) / (z.std(0) + 1e-5)   # batch normalisation
h = np.tanh(z)

mean(0) is the mean along the batch: the values one neuron produced across the 512 samples in the batch, and their mean and standard deviation. Layer normalisation, later, differs only here.

Put that single line into part four’s twenty-layer stack. One difference: the three initialisations here are redrawn to share the same random numbers, so that only the constant factor separates them - which is what makes normalisation collapsing them visible. That is why unnormalised Xavier reads 1.543e-01 here rather than the 1.62e-01 of part four’s table.

initialisation   no normalisation (L20)    batch norm (L20)
0.01                      1.135e-16             0.6310
1.0                       9.742e-01             0.6310
Xavier                    1.543e-01             0.6310

Three identical values to four decimals. Not a coincidence - it follows from the definition.

First, what exactly is set to one. What gets normalised is z, before the activation. The layer’s output tanh(z) is not 1 but 0.6310, which is the number in the table. Every layer returns z to the same distribution, so the same value comes out each time.

The reason all three agree is simpler still. Batch normalisation erases a positive scalar factor completely: BN(cz) = BN(z). Not exactly, strictly speaking - the eps in the code above sits in the denominator as a constant, so feeding it z and 100z leaves a difference of 4e-5. Against an activation of 0.6310 that is 0.007%, too small to show up in the table; drop the eps and the difference is 5e-15, pure floating-point noise. The initialisations 0.01, 1.0 and Xavier are the same random numbers times different constants, so the moment normalisation is added the three are the same network. What part four called the base of the multiplication is reset at every layer - and this argument does not carry to initialisations that are not scalar multiples of each other, such as orthogonal initialisation.

That is normalisation’s real value. Before any speedup, the initialisation stops being a hyperparameter. Building twenty layers no longer involves fussing over scale.

Price 1: it depends on the batch

Not free. The mean and standard deviation are estimated from the batch. Measure the layer-20 activation while varying only the batch size:

batch      batch norm     layer norm
4            0.6756         0.6261
8            0.6507         0.6339
32           0.6346         0.6272
256          0.6317         0.6287
512          0.6310         0.6282

Batch normalisation drifts upward as the batch shrinks. Layer normalisation is flat, independent of the batch.

It is easy to explain this as “a small batch underestimates the standard deviation, so dividing by it inflates the result”. That is wrong. Normalisation divides a batch by that batch’s own standard deviation, so the output has standard deviation exactly 1 at any batch size. Measured: 0.999983 at batch 4 and 0.999990 at batch 512, no difference. However far off the estimate is, dividing by it returns 1.

The real cause is shape. Standardise n samples against themselves and no value can exceed sqrt(n-1). At batch 4 that bound is 1.732; at batch 512 it is 22.6. The tails are clipped and the kurtosis changes from 2.99 at batch 512 to 1.80 at batch 4 - not approximately but exactly 3(n-1)/(n+1), falling away from a normal distribution’s 3 as the batch shrinks. tanh is a function that squashes large values - and at a small batch there are no large values to squash. So less is squashed and the standard deviation comes out higher.

Two pieces of evidence. Swap the activation for a linear one and the batch dependence disappears completely: 1.0000 at both batch 4 and 512. And feeding exactly N(0,1) through normalisation plus tanh for a single layer already shows the whole gap, 0.6768 against 0.6281. It is not something twenty layers built up.

The estimator itself is still worth a look. The sample standard deviation is off in two ways.

batch   bias (how low on average)   spread (how much it moves per draw)
2              -0.4363                        0.4253
4              -0.2020                        0.3366
8              -0.0979                        0.2449
32             -0.0237                        0.1244
256            -0.0030                        0.0440

The spread matches 1/sqrt(2*batch): predicted 0.1250 at batch 32, measured 0.1244. The bias follows -3/(4*batch). Both are leading-order approximations for a large batch, so the first row, batch 2, is off by 14 to 17 percent; from batch 8 they are within a few percent.

The bias splits into two pieces. NumPy’s .std() divides by n rather than n-1, which is -1/(2n), and measuring the square root rather than the variance adds -1/(4n) through Jensen’s inequality. Switching to ddof=1 still leaves 3.5 percent at batch 8. Frameworks’ BatchNorm uses the biased variance too, so this is not a NumPy quirk.

What that estimation error does during training is not to change the scale but to shake the value from batch to batch. The regularising effect in the next section comes from there.

Price 2: training and inference diverge

Using batch statistics means the same sample produces a different output depending on which samples share its batch. During training that wobble acts as a kind of regulariser and can help.

Inference is the problem. A single sample has no batch. So batch normalisation accumulates running means and variances during training and uses those at inference. The training path and the inference path compute different things, and the gap shows when the batch is small or the two distributions differ.

That is where layer normalisation comes from. It normalises along the feature direction within one sample instead of along the batch.

z = (z - z.mean(1, keepdims=True)) / (z.std(1, keepdims=True) + 1e-5)

One axis changed and the properties change with it. It never looks at other samples, so it is independent of batch size, training and inference compute the same thing, and it works on data whose samples have different lengths. That is why transformers use it.

What remains

Normalisation does not replace initialisation. The three initialisations above arrived at the same place, but that is the scale being equal; the directions the weights carry are still whatever the initialisation drew. And the normalisation layer has scale and shift parameters of its own, which have to be initialised too.

So

• Normalisation resets the base of the multiplication at every layer. Three initialisations landing on 0.6310 after twenty layers is the evidence
• What it buys, before any speedup, is the disappearance of sensitivity to the initialisation
• Batch norm’s output has variance 1 at any batch size. The rise at a small batch is not estimator bias but self-standardisation clipping the tails at sqrt(batch-1) - swap in a linear activation and the difference vanishes
• The estimation error shows up as wobble between batches, not as scale. Bias -3/(4*batch), spread 1/sqrt(2*batch), both large-batch approximations
• Layer norm never looks at the batch, so it avoids that. It normalises along a different axis, which makes it a different thing, not a strictly better one

The next part is about the difference between training well and predicting well. It drives the training loss to zero and measures what happens on data the model has not seen.