Backpropagation Fundamentals

Linear Algebra for Machine Learning · Axiom Academy

Backpropagation Through the Chain Rule How gradients flow backward through the network We need to measure how wrong our predictions are. For regression (continuous output): For classification, we typically use cross-entropy loss instead. Step 2: Gradient of Output Layer We start from the end: how does the loss change with respect to the final pre-activation? Step 3: Propagating Backward (Chain Rule) To find how L changes with W, we use: We update weights in the opposite direction of the gradient: Complete Backprop Example (3-layer network) Chain rule on matrices: ∂L/∂W = (∂L/∂z) × (h^T) is pure linear algebra Backprop = organized chain rule: Apply chain rule layer by layer, backward through the network Gradients flow backward: Information about loss propagates back through each layer Automatic differentiation: Modern frameworks (PyTorch, TensorFlow) compute this automatically Vanishing/exploding gradients: If products of gradients become very small or large, training becomes difficult Unit 8: Linear Algebra in Neural Networks | Module 7: Backpropagation

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.