Gradient Descent Optimization

Linear Algebra for Machine Learning · Axiom Academy

Iterative weight updates to minimize loss The Gradient Descent Update Rule Imagine standing on a hill (loss is height). The gradient points uphill. To reach the valley (minimum loss), we step downhill (opposite gradient direction). Worked Example: Single Weight Update SGD (Stochastic GD): Update on one sample at a time (noisy but efficient) Mini-batch GD: Update on batch of samples (balance between speed and stability) Adam: Adaptive learning rates per parameter (modern default) RMSprop: Adaptive learning rate with momentum Momentum: Accumulate gradient direction for acceleration Unit 8: Linear Algebra in Neural Networks | Module 10: Gradient Descent

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.