writing 한국어로 읽기
재귀와 합성곱 part 4 of 13

What the first-layer kernels learned was line endings

guide / / 6 sections

Parts one to three all carried a state around. A convolution carries nothing. It looks at a few characters near each position, answers, and that is the whole of it.

What one kernel computes

A kernel is a window of a few weights. At position t it multiplies the characters inside the window by those weights and adds them up, then moves along one and does it again.

one kernel slides across positions t h e · c a t · s a t same kernel every time output what channel 12 learned to do on real text -1 +0 +1 n k s · a n d · ` m + 1 ` p i e c e s . H e r e · i s · ` s i n ( 3 x ) ` newline ¶
Above: a kernel of five reads five characters and slides one position at a time. The same kernel at all three, and the window only reaches left, so nothing ahead is visible. Below: what the trained first-layer channel 12 does on real text. The vertical rules are newlines, where it jumps to about 1.2 against an average of 0.02 everywhere else.
xp = F.pad(x, (k-1, 0))            # pad only on the left, by k-1
for t in range(L):
    win = xp[:, :, t:t+k]          # (batch, channels, k)
    out[:, :, t] = (win * W).sum() + b

Running those lines with the trained weights matches nn.Conv1d to at most 1.79e-06. The numbers below come from there.

The three windows at the top of the figure are the same kernel. That is the same idea as recurrence reusing one matrix at every step, pointed differently. Recurrence shares along the time axis and has to go in order; convolution shares along the position axis and can compute every position at once.

Length grows, the parameter count does not

The first convolution’s weights are (171, 171, 5), or 146,205 numbers. Whether the context is 128 characters or 1024, that does not change, because a kernel only ever looks near one position.

Doing the same job with a dense layer connecting every position to every position would take 128 x 171 = 21,888 inputs wired to as many outputs: 479,084,544 weights. 3,277 times as many.

Over the whole model:

tok.weight        (100, 128)      12,800
inp.weight     (171, 128, 1)      21,888
convs.0~3   (171, 171, 5) x 4    585,504    91.6%
head.weight       (100, 171)      17,100
biases and norms                   1,981
                                 639,273

The four kernels are 91.6% of it - the same shape as part one’s RNN spending 73% on carrying state to state.

Causality is left-only padding

For next-character prediction, position t must not see past t, or it is reading the answer.

A convolution usually pads both sides so the window reaches left and right. Here it is padded only on the left, by k-1, so position t’s window runs from t-4 to t and nothing ahead gets in. That is the single F.pad(x, (k-1, 0)) in the code above.

What part thirteen of the first series did with an attention mask, a convolution does with where it puts the padding.

What the kernels respond to

Scanning 7,000 characters of validation text for the five characters each first-layer channel responds to most turns up readable ones.

channel  12   'd^2`\n'  'arts\n'  'why.\n'  '887`\n'
channel 134   '# So\n'  ' the\n'  'd^2`\n'  'or a\n'
channel  20   'ver.\n'  'sum.\n'  'ces.\n'  'full\n'
channel 169   'um.\n\n'  '0`.\n\n'  'ad.\n\n'  '0885\n'
channel  59   ' 0.10'  ' 0.80'  '/0.80'  '`0.80'
channel 105   '`m+1`'  ' = 0`'  ' = 8`'  '0.02`'
channel  63   'bias,'  'ches,'  'ides,'  'odels'
channel  51   ' add '  'y `Σ '  ' 90% '  'eads '

Three of them catch line endings. 169 catches two newlines together - a blank line. 59 catches decimal numbers, 105 backticks, 63 commas and plural s, 51 things ending in a space.

The lower half of the figure is channel 12 run over real text. It jumps to 1.23 and 1.16 at the newlines against an average of 0.02 everywhere else.

That is what a first layer does. It picks out whatever is visible in a five-character window - has the line ended, is this a number, is this code - and the layers above work from those.

What is left

This is the first layer. Going up, the window widens - nine characters at layer two, seventeen at layer four - and what the upper channels catch was not examined. Part five is about that widening.

This model also bottomed out at 300 steps, the fastest of the five in part seven’s table, and what the channels turn into with longer training is unknown.

Only kernel size 5 was used. What changes at 3 or 7 is unmeasured.

So

  • A convolution carries no state. It multiplies a five-character window and adds
  • Run by hand it matches nn.Conv1d to 1.79e-06
  • All three windows are the same kernel. Recurrence shares weights along time, a convolution shares them along position
  • The first kernel is (171, 171, 5), 146,205 numbers, independent of context length. A dense layer doing the same would need 479,084,544 - 3,277 times more
  • 91.6% of the model’s parameters are the four kernels
  • Causality comes from padding only the left by k-1. What attention does with a mask, a convolution does with where the padding goes
  • First-layer channels catch newlines, blank lines, decimals, backticks and commas. Channel 12 reaches 1.23 at a newline against 0.02 elsewhere

Comments