Weight Matrices & Parameters

Linear Algebra for Machine Learning · Axiom Academy

Weight Matrices in Neural Networks Structure, meaning, and initialization strategies Step 1: Understanding Dimensions A weight matrix W connecting layer l to layer l+1 has shape (n_l+1, n_l): n_l: number of neurons/features in layer l (input) n_l+1: number of neurons in layer l+1 (output) To multiply: y = Wx requires W to be (m × n) and x to be (n × 1), producing y of shape (m × 1). Each entry W_ij in the weight matrix represents: i: which neuron in the next layer j: which input feature from the current layer Value: strength of connection (weight) from input j to neuron i Large positive values = strong excitation. Large negative values = strong inhibition. Values near zero = weak influence. Step 3: Initialization Strategies How you initialize weights dramatically affects training speed and convergence. Poor initialization can lead to vanishing/exploding gradients or dead neurons. Why Different Initializations? Goal: Keep activations and gradients in reasonable ranges throughout training. Too small: gradients vanish, network won't learn Too large: activations explode, training becomes unstable Right scale: variance is preserved through layers (proper signal flow) For each layer with input dim n_in and output dim n_out: Memory: Each parameter takes space (usually 4 bytes for float32) Computation: More parameters = more multiply-accumulate (MAC) operations Generalization: More parameters require more training data to avoid overfitting

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.