Activation Functions & Nonlinearity

Linear Algebra for Machine Learning · Axiom Academy

How element-wise non-linearity enables learning complex patterns Without activation functions, stacking layers would just be repeated matrix multiplication—which collapses to a single linear transformation! GELU (Gaussian Error Linear Unit) Applying Different Activations to Same Input Activation functions are element-wise: Applied independently to each value in the pre-activation vector They break linearity: Stacking linear layers only works because of activations Choice matters: ReLU is standard for hidden layers, sigmoid for binary output, tanh for RNNs Gradient behavior: Sigmoid/Tanh saturate (gradient → 0), ReLU has sparse gradients (many zeros) Modern default: ReLU and variants for hidden layers, specific activation for output layer based on task Unit 8: Linear Algebra in Neural Networks | Module 6: Activation Functions

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.