Layers are matrix multiplications

In the artificial neuron you met a single neuron: multiply each input by a weightWeightA number that sets how much one input matters to a neuron, and in which direction. Positive weights push the output up, negative weights push it down.Taught in The artificial neuronOpen in glossary, add them up, add a biasBias (neuron)A number a neuron adds to its weighted sum before the activation function. It shifts how easily the neuron switches on, regardless of the inputs.Taught in The artificial neuronOpen in glossary, then apply an activation function. Real networks have thousands to billions of these. Nobody computes them one at a time. This lesson shows the bookkeeping trick that every neural network library uses, and once you see it, the shapes in any architecture diagram start to make sense.

One neuron is a dot product

Take a neuron with three inputs $x_1, x_2, x_3$. Before its activation, it computes

Collect the inputs into a vector $\mathbf{x} = (x_1, x_2, x_3)$ and the weights into $\mathbf{w} = (w_1, w_2, w_3)$. The sum of products is the dot product $\mathbf{w} \cdot \mathbf{x}$, so $z = \mathbf{w} \cdot \mathbf{x} + b$. A large positive $z$ means the input lines up well with the neuron’s weights.

A layer stacks those dot products

A layerLayerA group of neurons that all read the same inputs. A fully connected layer computes Wx + b for a weight matrix W and bias vector b, then applies an activation function.Open in glossary is a set of neurons that all read the same inputs. Give each neuron its own row in a matrix and the whole layer becomes one matrix multiplicationMatrix multiplicationCombining a matrix with a vector or another matrix by taking dot products of rows with columns. An (m x n) matrix times an n-vector gives an m-vector.Open in glossary:

Here $n$ is the number of inputs and $m$ the number of neurons. Row $i$ of $W$ holds neuron $i$‘s weights, and entry $z_i$ is that row’s dot product with $\mathbf{x}$, plus bias $b_i$. The activation function is then applied to each $z_i$ separately.

import numpy as np

# The demo's starting layer: 4 neurons, each reading the same 3 inputs.
W = np.array([[ 0.7, -1.4, -0.1],    # row i holds neuron i's weights
              [-1.3,  0.8, -0.1],
              [ 0.0, -1.0, -0.7],
              [ 0.2, -0.9, -1.2]])
b = np.array([0.2, 0.0, 0.3, 0.4])
x = np.array([1.0, 0.5, -1.0])

z = W @ x + b                  # one dot product per row, plus that row's bias
print("shapes:", W.shape, "times", x.shape, "->", z.shape)
print("z =", z.round(2))

# Neuron 1 by hand: the first row's dot product with x.
print("z1 =", round(0.7 * 1.0 + (-1.4) * 0.5 + (-0.1) * (-1.0) + 0.2, 2))

# A batch: stack inputs as rows of X and compute X W^T + b in one go.
X = np.array([[1.0, 0.5, -1.0],
              [0.0, 0.0,  0.0]])   # an all-zero input returns b
print((X @ W.T + b).round(2))
print("parameters:", W.size + b.size)

shapes: (4, 3) times (3,) -> (4,)
z = [ 0.3  -0.8   0.5   1.35]
z1 = 0.3
[[ 0.3  -0.8   0.5   1.35]
 [ 0.2   0.    0.3   0.4 ]]
parameters: 16

Shapes are the grammar of networks

The rule for a matrix times a vector is that the inner sizes must match: an $(m \times n)$ matrix accepts only $n$ inputs and always returns $m$ outputs. Chaining layers means chaining shapes. A network that reads a 28 × 28 image (784 pixels) through layers of 64, 32, and 10 neurons uses

and each layer’s output length is the next layer’s input length. That exact network appears later in this track, in Reading handwriting, where it reads digits you draw.

Counting parameters

Every weight and every bias is a parameterParameterA number inside a model that training adjusts, such as the slope of a line or a weight in a neural network. Large language models have billions of them.Taught in The learning loopOpen in glossary that training adjusts. A layer from $n$ inputs to $m$ neurons has $m \times n$ weights and $m$ biases. For the handwriting network above:

Almost all of the parameters sit in the first layer, because it connects every pixel to every neuron. Large language models follow the same arithmetic, just with matrices thousands of rows wide, which is how they reach billions of parameters.

Whole batches at once

Training rarely feeds one example at a time. Stack $B$ input vectors as the rows of a matrix $X \in \mathbb{R}^{B \times n}$, and the layer handles all of them in one multiplication:

where the bias is added to every row. This is the form libraries actually use. PyTorch’s nn.Linear, for example, stores its weight as an (out × in) matrix and computes $XW^{\top} + \mathbf{b}$.