Neural Network Formulas & Reference

Linear Algebra for Machine Learning · Axiom Academy

Complete Neural Network Formulas All key equations for reference z ∈ ℝ^(m × 1): pre-activation output X ∈ ℝ^(batch_size × n): batch of inputs Z ∈ ℝ^(batch_size × m): batch of outputs Where σ is an activation function Mean Squared Error (Regression) Cross-Entropy (Classification) ∂L/∂z: gradient w.r.t. pre-activation ⊙ denotes element-wise multiplication For f: ℝ^n → ℝ^m, Jacobian is m × n Gradient Descent with Momentum β: momentum coefficient (typically 0.9) Q: query matrix (seq_len × d_k) V: value matrix (seq_len × d_v) Applies filter w to input x at position [i,j] Converts convolution to standard matrix multiplication Maintains variance through ReLU layers Xavier/Glorot (for Sigmoid/Tanh) Maintains variance through layers Unit 8: Linear Algebra in Neural Networks | Module 18: Formula Sheet

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.