A tensor is one-dimensional

How long does it take to turn a (3, 4) array into a (4, 3) one? Twelve values to move, so about twelve operations. In fact nothing moves at all. To see why, look at how the array lies in memory.

There is no such thing as a multidimensional array

Memory is one-dimensional. Addresses run 0, 1, 2 in a single line, and there is no way to lay a second dimension on top of that. A NumPy array is no exception. The values sit in one row, and “two-dimensional” is nothing but a rule for what step size to read them at.

That step is called the stride.

import numpy as np

a = np.arange(12, dtype=np.int32)
b = a.reshape(3, 4)

print(a.strides)          # (4,)
print(b.shape, b.strides) # (3, 4) (16, 4)
print(np.shares_memory(a, b))

(4,)
(3, 4) (16, 4)
True

An int32 is 4 bytes. Strides of (16, 4) say: to move down one row skip 16 bytes, to move across one column skip 4. Sixteen is 4 bytes times 4 columns. That is the whole of it. reshape touched no values. It wrote down two different numbers, which is why shares_memory is True.

Transposing swaps two numbers

c = b.T
print(c.shape, c.strides)          # (4, 3) (4, 16)
print(np.shares_memory(a, c))      # True

(4, 3) (4, 16)
True

(16, 4) became (4, 16). That is all. Leave the values alone and write down “4 to move along a row, 16 to move along a column”, and the same memory reads as the transposed matrix. Transposing a million-by-million matrix costs exactly the same.

Address arithmetic is one line.

offset = index[0]*stride[0] + index[1]*stride[1] + ...

Check it. The offset of b[2, 1] is 2x16 + 1x4 = 36 bytes, which divided by 4 is slot 9 of the buffer.

print(b[2, 1])                                    # 9
print(2 * b.strides[0] + 1 * b.strides[1])        # 36

9
36

Indexing is two multiplications and an addition. Slicing is the same: b[:, 1:3] moves the starting offset and inherits the strides untouched.

When it is not free

So far everything looks free. It is not. The moment something is asked for that strides cannot express, a copy happens.

c = b.T                       # (4, 3), strides (4, 16)
print(np.shares_memory(a, c.reshape(12)))
print(np.shares_memory(a, c.ravel()))

False
False

Flattening c would have to produce 0, 4, 8, 1, 5, 9, ..., and there is no single constant step that walks that order. So NumPy copies into a fresh buffer, and that is the one operation here that genuinely costs O(n).

This is what that message in a deep-learning framework is about. In the same situation PyTorch refuses with view size is not compatible with input tensor's size and stride. view promises to do only what a stride change can do; reshape will fall back to copying. The two names exist because of performance.

The difference between ravel and flatten is the same line: ravel returns a view where it can, flatten always copies. Above, c is non-contiguous, so both copied.

Broadcasting is a stride of zero

One more. Stretching a (2, 3) array to (4, 2, 3) looks like it should cost four times the memory.

e = np.arange(6, dtype=np.int32).reshape(2, 3)
print(np.broadcast_to(e, (4, 2, 3)).strides)

(0, 12, 4)

The first axis has stride 0. “Moving one step along that axis leaves the address unchanged”, so the same six values are read four times. No copy. Broadcasting is fast not because of a clever algorithm but because someone wrote down a zero.

So

An array is not a grid of values. It is one buffer plus a rule for reading it. Hold that picture and the following are all the same story.

• why reshape, transpose and slicing return instantly
• why some operations refuse, complaining about contiguity
• why one call to .contiguous() makes what follows faster
• why broadcasting costs no memory

The next part looks at loss rather than values. It draws the loss surface of a two-parameter problem and follows exactly what steps gradient descent takes across it.