Unconstrained Optimization Summary

Optimization · Axiom Academy

SUMMARY Unconstrained Optimization Let's review the key concepts, algorithms, and strategies for finding optimal solutions without constraints. Definition: Finding the minimum or maximum of a function f(x) with no restrictions on x Goal: Locate critical points where the function stops improving Why It Matters: Foundation for machine learning (gradient descent), engineering design, portfolio optimization, and scientific computing Real Applications: Training neural networks, curve fitting, least squares regression, and resource allocation First-Order (Necessary): At a local minimum, the gradient must be zero: ∇f(x*) = 0 Second-Order (Sufficient): The Hessian must be positive definite to confirm it's a minimum, not a maximum or saddle point Geometric Intuition: Gradient = 0 means "no downhill direction exists," Hessian positive definite means "curvature bends upward in all directions" Practical Check: Solve gradient equations, then verify eigenvalues of Hessian are all positive Gradient Descent: Iteratively move in the negative gradient direction. Simple to implement, linear convergence, works well for large-scale problems Newton's Method: Uses second-order (Hessian) information for quadratic convergence. Fast near solution but expensive per iteration Quasi-Newton (BFGS): Approximates Hessian to balance speed and cost. Superlinear convergence with moderate memory requirements

This is the written version of the interactive lesson above. See the full Optimization course.