Optimization Landscape & Hessian

Linear Algebra for Machine Learning · Axiom Academy

Loss Landscape and Optimization Geometry How matrix properties shape the optimization landscape The Hessian is the matrix of second derivatives: Large positive eigenvalue: Steep curvature in that direction Small positive eigenvalue: Flat, direction not important for optimization Mixed signs: Saddle point (some directions up, some down) The convergence speed depends on the condition number of the Hessian: Vanishing and Exploding Gradients Careful weight initialization: Xavier, He initialization Batch normalization: Keeps layer outputs standardized Residual connections: Allow gradients to bypass layers Gradient clipping: Prevent explosions during training Hessian eigenvalues determine curvature: They shape how gradient descent moves Condition number matters: Large κ means slow, oscillatory convergence Deep networks have gradient flow issues: Products of Jacobians can vanish/explode Loss landscapes are complex: Many saddle points, but few local minima in high dimensions Modern techniques address this: Normalization, residual connections, careful initialization Unit 8: Linear Algebra in Neural Networks | Module 14: Optimization Landscape

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.