Neural Networks as Matrix Operations

Linear Algebra for Machine Learning · Axiom Academy

Neural Networks as Matrix Operations Understanding the four core steps of a forward pass By the end of this lesson, you'll understand: How input data is represented as vectors How weights are organized into matrices The mathematical formulation y = Wx + b How layers compose to create deep networks Every piece of input data is represented as a column vector x ∈ ℝⁿ, where n is the number of features. For a single sample with 3 features (e.g., pixel values, sensor readings): For practical machine learning, we often process multiple samples simultaneously. Each sample becomes a row in a matrix X (batch_size × n_features). The learned parameters of a layer are organized into a weight matrix W ∈ ℝᵐˣⁿ, where: m = number of neurons in this layer (output dimension) Each row of W represents the weights for one neuron. Each column represents which weights all neurons receive from one input feature. Linear Transformation with Bias The forward pass computes: y = Wx + b Wx + b computes a weighted sum of inputs, plus a bias offset The result y is called the pre-activation or logit. Each element of y is a linear combination of all inputs. Stacking Layers Creates Deep Networks A single layer transforms input to an intermediate representation. To create depth, we apply activation and then feed to the next layer: This composition allows networks to learn complex non-linear functions by alternating linear transformations with non-linear activations.

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.